REVIEWING ARTICLE AND CHECKLIST OF REVIEW CRITERIA

Review articles: purpose, process, and structure

By Robert W. Palmatier,  Mark B. Houston &  John Hulland 

Source: Journal of the Academy of Marketing Science volume 46, pages1–5 (2018)

Many research disciplines feature high-impact journals that are dedicated outlets for review
papers (or review–conceptual combinations) (e.g., Academy of Management Review, Psychology
Bulletin, Medicinal Research Reviews). The rationale for such outlets is the premise that research
integration and synthesis provides an important, and possibly even a required, step in the
scientific process. Review papers tend to include both quantitative (i.e., meta-analytic, systematic
reviews) and narrative or more qualitative components; together, they provide platforms for new
conceptual frameworks, reveal inconsistencies in the extant body of research, synthesize diverse
results, and generally give other scholars a “state-of-the-art” snapshot of a domain, often written
by topic experts (Bem 1995). Many premier marketing journals publish meta-analytic review
papers too, though authors often must overcome reviewers’ concerns that their contributions are
limited due to the absence of “new data.” Furthermore, relatively few non-meta-analysis review
papers appear in marketing journals, probably due to researchers’ perceptions that such papers
have limited publication opportunities or their beliefs that the field lacks a research tradition or
“respect” for such papers. In many cases, an editor must provide strong support to help such
review papers navigate the review process. Yet, once published, such papers tend to be widely
cited, suggesting that members of the field find them useful (see Bettencourt and Houston 2001).

In this editorial, we seek to address three topics relevant to review papers. First, we outline a case
for their importance to the scientific process, by describing the purpose of review papers.
Second, we detail the review paper editorial initiative conducted over the past two years by
the Journal of the Academy of Marketing Science (JAMS), focused on increasing the prevalence
of review papers. Third, we describe a process and structure for systematic (i.e., non-meta-
analytic) review papers, referring to Grewal et al. (2018) insights into parallel meta-analytic
(effects estimation) review papers. (For some strong recent examples of marketing-related meta-
analyses, see Knoll and Matthes 2017; Verma et al. 2016).

Purpose of review papers

In their most general form, review papers “are critical evaluations of material that has already
been published,” some that include quantitative effects estimation (i.e., meta-analyses) and some
that do not (i.e., systematic reviews) (Bem 1995, p. 172). They carefully identify and synthesize
relevant literature to evaluate a specific research question, substantive domain, theoretical
approach, or methodology and thereby provide readers with a state-of-the-art understanding of
the research topic. Many of these benefits are highlighted in Hanssens’ (2018) paper titled “The
Value of Empirical Generalizations in Marketing,” published in this same issue of JAMS.

The purpose of and contributions associated with review papers can vary depending on their
specific type and research question, but in general, they aim to

 Resolve definitional ambiguities and outline the scope of the topic.

 Provide an integrated, synthesized overview of the current state of knowledge.
 Identify inconsistencies in prior results and potential explanations (e.g., moderators,
mediators, measures, approaches).
 Evaluate existing methodological approaches and unique insights.
 Develop conceptual frameworks to reconcile and extend past research.
 Describe research insights, existing gaps, and future research directions.
Not every review paper can offer all of these benefits, but this list represents their key
contributions. To provide a sufficient contribution, a review paper needs to achieve three key
standards. First, the research domain needs to be well suited for a review paper, such that a
sufficient body of past research exists to make the integration and synthesis valuable—especially
if extant research reveals theoretical inconsistences or heterogeneity in its effects. Second, the
review paper must be well executed, with an appropriate literature collection and analysis
techniques, sufficient breadth and depth of literature coverage, and a compelling writing style.
Third, the manuscript must offer significant new insights based on its systematic comparison of
multiple studies, rather than simply a “book report” that describes past research. This third, most
critical standard is often the most difficult, especially for authors who have not “lived” with the
research domain for many years, because achieving it requires drawing some non-obvious
connections and insights from multiple studies and their many different aspects (e.g., context,
method, measures). Typically, after the “review” portion of the paper has been completed, the
authors must spend many more months identifying the connections to uncover incremental
insights, each of which takes time to detail and explicate.

The increasing methodological rigor and technical sophistication of many marketing studies also
means that they often focus on smaller problems with fewer constructs. By synthesizing these
piecemeal findings, reconciling conflicting evidence, and drawing a “big picture,” meta-analyses
and systematic review papers become indispensable to our comprehensive understanding of a
phenomenon, among both academic and practitioner communities. Thus, good review papers
provide a solid platform for future research, in the reviewed domain but also in other areas, in
that researchers can use a good review paper to learn about and extend key insights to new areas.

This domain extension, outside of the core area being reviewed, is one of the key benefits of
review papers that often gets overlooked. Yet it also is becoming ever more important with the
expanding breadth of marketing (e.g., econometric modeling, finance, strategic management,
applied psychology, sociology) and the increasing velocity in the accumulation of marketing
knowledge (e.g., digital marketing, social media, big data). Against this backdrop, systematic
review papers and meta-analyses help academics and interested managers keep track of research
findings that fall outside their main area of specialization.

JAMS’ review paper editorial initiative

With a strong belief in the importance of review papers, the editorial team of JAMS has
purposely sought out leading scholars to provide substantive review papers, both meta-analysis
and systematic, for publication in JAMS. Many of the scholars approached have voiced concerns
about the risk of such endeavors, due to the lack of alternative outlets for these types of papers.
Therefore, we have instituted a unique process, in which the authors develop a detailed outline of
their paper, key tables and figures, and a description of their literature review process. On the
basis of this outline, we grant assurances that the contribution hurdle will not be an issue for
publication in JAMS, as long as the authors execute the proposed outline as written. Each paper
still goes through the normal review process and must meet all publication quality standards, of
course. In many cases, an Area Editor takes an active role to help ensure that each paper provides
sufficient insights, as required for a high-quality review paper. This process gives the author
team confidence to invest effort in the process. An analysis of the marketing journals in
the Financial Times (FT 50) journal list for the past five years (2012–2016) shows that JAMS has
become the most common outlet for these papers, publishing 31% of all review papers that
appeared in the top six marketing journals.

As a next step in positioning JAMS as a receptive marketing outlet for review papers, we are
conducting a Thought Leaders Conference on Generalizations in Marketing: Systematic Reviews
and Meta-Analyses, with a corresponding special issue (see www.springer.com/jams). We will
continue our process of seeking out review papers as an editorial strategy in areas that could be
advanced by the integration and synthesis of extant research. We expect that, ultimately, such
efforts will become unnecessary, as authors initiate review papers on topics of their own
choosing to submit them to JAMS. In the past two years, JAMS already has increased the number
of papers it publishes annually, from just over 40 to around 60 papers per year; this growth has
provided “space” for 8–10 review papers per year, reflecting our editorial target.

Consistent with JAMS’ overall focus on managerially relevant and strategy-focused topics, all
review papers should reflect this emphasis. For example, the domains, theories, and methods
reviewed need to have some application to past or emerging managerial research. A good rule of
thumb is that the substantive domain, theory, or method should attract the attention of readers
of JAMS.

The efforts of multiple editors and Area Editors in turn have generated a body of review papers
that can serve as useful examples of the different types and approaches that JAMS has published.

Domain-based review papers

Domain-based review papers review, synthetize, and extend a body of literature in the same
substantive domain. For example, in “The Role of Privacy in Marketing” (Martin and
Murphy 2017), the authors identify and define various privacy-related constructs that have
appeared in recent literature. Then they examine the different theoretical perspectives brought to
bear on privacy topics related to consumers and organizations, including ethical and legal
perspectives. These foundations lead in to their systematic review of privacy-related articles over
a clearly defined date range, from which they extract key insights from each study. This exercise
of synthesizing diverse perspectives allows these authors to describe state-of-the-art knowledge
regarding privacy in marketing and identify useful paths for research. Similarly, a new paper by
Cleeren et al. (2017), “Marketing Research on Product-Harm Crises: A Review, Managerial
Implications, and an Agenda for Future Research,” provides a rich systematic review,
synthesizes extant research, and points the way forward for scholars who are interested in issues
related to defective or dangerous market offerings.

Theory-based review papers

Theory-based review papers review, synthetize, and extend a body of literature that uses the
same underlying theory. For example, Rindfleisch and Heide’s (1997) classic review of research
in marketing using transaction cost economics has been cited more than 2200 times, with a
significant impact on applications of the theory to the discipline in the past 20 years. A recent
paper in JAMS with similar intent, which could serve as a helpful model, focuses on “Resource-
Based Theory in Marketing” (Kozlenkova et al. 2014). The article dives deeply into a description
of the theory and its underlying assumptions, then organizes a systematic review of relevant
literature according to various perspectives through which the theory has been applied in
marketing. The authors conclude by identifying topical domains in marketing that might benefit
from additional applications of the theory (e.g., marketing exchange), as well as related theories
that could be integrated meaningfully with insights from the resource-based theory.

Method-based review papers

Method-based review papers review, synthetize, and extend a body of literature that uses the
same underlying method. For example, in “Event Study Methodology in the Marketing
Literature: An Overview” (Sorescu et al. 2017), the authors identify published studies in
marketing that use an event study methodology. After a brief review of the theoretical
foundations of event studies, they describe in detail the key design considerations associated with
this method. The article then provides a roadmap for conducting event studies and compares this
approach with a stock market returns analysis. The authors finish with a summary of the
strengths and weaknesses of the event study method, which in turn suggests three main areas for
further research. Similarly, “Discriminant Validity Testing in Marketing: An Analysis, Causes
for Concern, and Proposed Remedies” (Voorhies et al. 2016) systematically reviews existing
approaches for assessing discriminant validity in marketing contexts, then uses Monte Carlo
simulation to determine which tests are most effective.

Our long-term editorial strategy is to make sure JAMS becomes and remains a well-recognized

outlet for both meta-analysis and systematic managerial review papers in marketing. Ideally,
review papers would come to represent 10%–20% of the papers published by the journal.

Process and structure for review papers

In this section, we review the process and typical structure of a systematic review paper, which
lacks any long or established tradition in marketing research. The article by Grewal et al. (2018)
provides a summary of effects-focused review papers (i.e., meta-analyses), so we do not discuss
them in detail here.

Systematic literature review process

Some review papers submitted to journals take a “narrative” approach. They discuss current
knowledge about a research domain, yet they often are flawed, in that they lack criteria for article
inclusion (or, more accurately, article exclusion), fail to discuss the methodology used to
evaluate included articles, and avoid critical assessment of the field (Barczak 2017). Such
reviews tend to be purely descriptive, with little lasting impact.

In contrast, a systematic literature review aims to “comprehensively locate and synthesize

research that bears on a particular question, using organized, transparent, and replicable
procedures at each step in the process” (Littell et al. 2008, p. 1). Littell et al. describe six key
steps in the systematic review process. The extent to which each step is emphasized varies by
paper, but all are important components of the review.

1. Topic formulation. The author sets out clear objectives for the review and articulates
the specific research questions or hypotheses that will be investigated.

2. Study design. The author specifies relevant problems, populations, constructs, and
settings of interest. The aim is to define explicit criteria that can be used to assess
whether any particular study should be included in or excluded from the review.
Furthermore, it is important to develop a protocol in advance that describes the
procedures and methods to be used to evaluate published work.

3. Sampling. The aim in this third step is to identify all potentially relevant studies,
including both published and unpublished research. To this end, the author must first
define the sampling unit to be used in the review (e.g., individual, strategic business unit)
and then develop an appropriate sampling plan.

4. Data collection. By retrieving the potentially relevant studies identified in the third
step, the author can determine whether each study meets the eligibility requirements set
out in the second step. For studies deemed acceptable, the data are extracted from each
study and entered into standardized templates. These templates should be based on the
protocols established in step 2.

5. Data analysis. The degree and nature of the analyses used to describe and examine the
collected data vary widely by review. Purely descriptive analysis is useful as a starting
point but rarely is sufficient on its own. The examination of trends, clusters of ideas, and
multivariate relationships among constructs helps flesh out a deeper understanding of the
domain. For example, both Hult (2015) and Huber et al. (2014) use bibliometric
approaches (e.g., examine citation data using multidimensional scaling and cluster
analysis techniques) to identify emerging versus declining themes in the broad field of
marketing.
 6. Reporting. Three key aspects of this final step are common across systematic reviews.
First, the results from the fifth step need to be presented, clearly and compellingly, using
narratives, tables, and figures. Second, core results that emerge from the review must be
interpreted and discussed by the author. These revelatory insights should reflect a deeper
understanding of the topic being investigated, not simply a regurgitation of well-
established knowledge. Third, the author needs to describe the implications of these
unique insights for both future research and managerial practice.

A new paper by Watson et al. (2017), “Harnessing Difference: A Capability-Based Framework

for Stakeholder Engagement in Environmental Innovation,” provides a good example of a
systematic review, starting with a cohesive conceptual framework that helps establish the
boundaries of the review while also identifying core constructs and their relationships. The
article then explicitly describes the procedures used to search for potentially relevant papers and
clearly sets out criteria for study inclusion or exclusion. Next, a detailed discussion of core
elements in the framework weaves published research findings into the exposition. The paper
ends with a presentation of key implications and suggestions for the next steps. Similarly,
“Marketing Survey Research Best Practices: Evidence and Recommendations from a Review
of JAMS Articles” (Hulland et al. 2017) systematically reviews published marketing studies that
use survey techniques, describes recent trends, and suggests best practices. In their review,
Hulland et al. examine the entire population of survey papers published in JAMS over a ten-year
span, relying on an extensive standardized data template to facilitate their subsequent data
analysis.

Structure of systematic review papers

There is no cookie-cutter recipe for the exact structure of a useful systematic review paper; the
final structure depends on the authors’ insights and intended points of emphasis. However,
several key components are likely integral to a paper’s ability to contribute.

Depth and rigor

Systematic review papers must avoid falling in to two potential “ditches.” The first ditch
threatens when the paper fails to demonstrate that a systematic approach was used for selecting
articles for inclusion and capturing their insights. If a reader gets the impression that the author
has cherry-picked only articles that fit some preset notion or failed to be thorough enough,
without including articles that make significant contributions to the field, the paper will be
consigned to the proverbial side of the road when it comes to the discipline’s attention.

Authors that fall into the other ditch present a thorough, complete overview that offers only a
mind-numbing recitation, without evident organization, synthesis, or critical evaluation.
Although comprehensive, such a paper is more of an index than a useful review. The reviewed
articles must be grouped in a meaningful way to guide the reader toward a better understanding
of the focal phenomenon and provide a foundation for insights about future research directions.
Some scholars organize research by scholarly perspectives (e.g., the psychology of privacy, the
economics of privacy; Martin and Murphy 2017); others classify the chosen articles by objective
research aspects (e.g., empirical setting, research design, conceptual frameworks; Cleeren et
al. 2017). The method of organization chosen must allow the author to capture the complexity of
the underlying phenomenon (e.g., including temporal or evolutionary aspects, if relevant).

Replicability

Processes for the identification and inclusion of research articles should be described in
sufficient detail, such that an interested reader could replicate the procedure. The procedures
used to analyze chosen articles and extract their empirical findings and/or key takeaways should
be described with similar specificity and detail.

Usability

We already have noted the potential usefulness of well-done review papers. Some scholars
always are new to the field or domain in question, so review papers also need to help them gain
foundational knowledge. Key constructs, definitions, assumptions, and theories should be laid
out clearly (for which purpose summary tables are extremely helpful). An integrated conceptual
model can be useful to organize cited works. Most scholars integrate the knowledge they gain
from reading the review paper into their plans for future research, so it is also critical that review
papers clearly lay out implications (and specific directions) for research. Ideally, readers will
come away from a review article filled with enthusiasm about ways they might contribute to the
ongoing development of the field.

Helpful format

Because such a large body of research is being synthesized in most review papers, simply
reading through the list of included studies can be exhausting for readers. We cannot overstate
the importance of tables and figures in review papers, used in conjunction with meaningful
headings and subheadings. Vast literature review tables often are essential, but they must be
organized in a way that makes their insights digestible to the reader; in some cases, a sequence of
more focused tables may be better than a single, comprehensive table.

Conclusion

In summary, articles that review extant research in a domain (topic, theory, or method) can be
incredibly useful to the scientific progress of our field. Whether integrating the insights from
extant research through a meta-analysis or synthesizing them through a systematic assessment,
the promised benefits are similar. Both formats provide readers with a useful overview of
knowledge about the focal phenomenon, as well as insights on key dilemmas and conflicting
findings that suggest future research directions. Thus, the editorial team at JAMS encourages
scholars to continue to invest the time and effort to construct thoughtful review papers.

© 2022 Springer Nature Switzerland AG. Part of Springer Nature.

 CHECKLIST OF REVIEW CRITERIA

Problem Statement, Conceptual Framework, and Research

Question

 The introduction builds a logical case and context for the problem statement.
 The problem statement is clear and well-articulated.
 The conceptual framework is explicit and justified.
 The research question (research hypothesis where applicable) is clear, concise, and
complete.
 The variables being investigated are clearly identified and presented.

Reference to the Literature and Documentation

 The literature review is up-to-date.

 The number of references is appropriate and their selection is judicious.
 The review of the literature is well integrated.
 The references are mainly primary sources.
 Ideas are acknowledged appropriately (scholarly attribution) and accurately.
 The literature is analyzed and critically appraised.

Relevance

 The study is relevant to the mission of the journal or its audience.

 The study addresses important problems or issues; the study is worth doing.
 The study adds to the literature already available on the subject.
  The study has generalizability because of the selection of subjects, setting, and
educational intervention or materials.

Research Design

 The research design is defined and clearly described, and is sufficiently detailed to permit
the study to replicate.
 The design is appropriate (optimal) for the research question.
 The design has internal validity, potential confounding variables or biases are addressed.
 The design has external validity, including subjects, settings, and conditions.
 The design allows for unexpected outcomes or events to occur.
 The design and conduct of the study are plausible.

Instrumentation, Data Collection, and Quality Control

 The development and content of the instrument are sufficiently described or referenced,
and are sufficiently detailed to permit the study to be replicated.
 The measurement instrument is appropriate given the study's variables; the scoring
method is clearly defined.
 The psychometric properties and procedures are clearly presented and appropriate.
 The data set is sufficiently described or referenced.
 Observers or raters were sufficiently trained.
 Data quality control is described and adequate.

Population and Sample

 The population is defined clearly, both for subjects (participants) and stimulus
(intervention), and is sufficiently detailed to permit the study to be replicated.
 The sampling procedures are sufficiently described.
 Subject samples are appropriate to the research question.
  Stimulus samples are appropriate to the research question.
 Selection bias is addressed.

Data Analysis and Statistics

 Data analysis procedures are sufficiently described, and are sufficiently detailed to permit
the study to be replicated.
 Data analysis procedures conform to the research design; hypotheses, models, or theory
drives the data analyses.
 The assumptions underlying the use of statistics are fulfilled by the data, such as
measurement properties of the data and normality of distributions.
 Statistical tests are appropriate (optimal).
 If statistical analysis involves multiple tests or comparisons, proper adjustment of
significance level for chance outcomes was applied.
 Power issues are considered in statistical studies with small sample sizes.
 In qualitative research that relies on words instead of numbers, basic requirements of data
reliability, validity, trustworthiness, and absence of bias were fulfilled.

Reporting of Statistical Analyses

 The assumptions underlying the use of statistics are considered, given the data collected.
 The statistics are reported correctly and appropriately.
 The number of analyses is appropriate.
 Measures of functional significance, such as effect size or proportion of variance
accounted for, accompany hypothesis-testing analyses.

Presentation of Results

 Results are organized in a way that is easy to understand.

 Results are presented effectively; the results are contextualized.
 The results are complete.
 The amount of data presented is sufficient and appropriate.
  Tables, graphs, or figures are used judiciously and agree with the text.

Discussion and Conclusion: Interpretation

 The conclusions are clearly stated; key points stand out.

 The conclusions follow from the design, methods, and results; justification of conclusions
is well articulated.
 Interpretations of the results are appropriate; the conclusions are accurate (not
misleading).
 The study limitations are discussed.
 Alternative interpretations for the findings are considered.
 Statistical differences are distinguished from meaningful differences.
 Personal perspectives or values related to interpretations are discussed.
 Practical significance or theoretical implications are discussed; guidance for future
studies is offered.

Title, Authors, and Abstract

 The title is clear and informative.

 The title is representative of the content and breadth of the study (not misleading).
 The title captures the importance of the study and the attention of the reader.
 The number of authors appears to be appropriate given the study.
 The abstract is complete (thorough); essential details are presented.
 The results in the abstract are presented in sufficient and specific detail.
 The conclusions in the abstract are justified by the information in the abstract and the
text.
 There are no inconsistencies in detail between the abstract and the text.
 All of the information in the abstract is present in the text.
 The abstract overall is congruent with the text; the abstract gives the same impression as
the text.
Presentation and Documentation

 The text is well written and easy to follow.

 The vocabulary is appropriate.
 The content is complete and fully congruent.
 The manuscript is well organized.
 The data reported are accurate (e.g., numbers add up) and appropriate; tables and figures
are used effectively and agree with the text.
 Reference citations are complete and accurate.

Scientific Conduct

 There are no instances of plagiarism.

 Ideas and materials of others are correctly attributed.
 Prior publication by the author(s) of substantial portions of the data or study is
appropriately acknowledged.
 There is no apparent conflict of interest.
 There is an explicit statement of approval by an institutional review board (IRB) for
studies directly involving human subjects or data about them.