Intelligence 61 (2017) 130–139

Contents lists available at ScienceDirect

Intelligence

Checking the “Academic Selection” argument. Chess players outperform

non-chess players in cognitive skills related to intelligence:
A meta-analysis
Giovanni Sala a,⁎,1, Alexander P. Burgoyne b,1, Brooke N. Macnamara c, David Z. Hambrick b,
Guillermo Campitelli d, Fernand Gobet a
a
University of Liverpool, United Kingdom
b
Michigan State University, United States
c
Case Western Reserve University, United States
d
Edith Cowan University, Australia

a r t i c l e i n f o a b s t r a c t

Article history: Substantial research in the psychology of expertise has shown that experts in several fields (e.g., science, math-
Received 27 May 2016 ematics) perform better than non-experts on standardized tests of intelligence. This evidence suggests that intel-
Received in revised form 23 December 2016 ligence plays an important role in the acquisition of expertise. However, a counter argument is that the difference
Accepted 24 January 2017
between experts and non-experts is not due to individuals' traits but to academic selection processes. For in-
Available online 3 February 2017
stance, in science, high scores on standardized tests (e.g., SAT and then GRE) are needed to be admitted to a uni-
Keywords:
versity program for training. Thus, the “academic selection process” hypothesis is that expert vs. non-expert
Chess differences in cognitive ability reflect ability-related differences in access to training opportunities. To test this
Intelligence hypothesis, we focused on a domain in which there are no selection processes based on test scores: chess. This
Cognitive ability meta-analysis revealed that chess players outperformed non-chess players in intelligence-related skills (d =
Deliberate practice 0.49). Therefore, this outcome does not corroborate the academic selection process argument, and consequently,
Meta-analysis
supports the idea that access to training alone cannot explain expert performance.
© 2017 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
1.1. Chess and expertise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
1.2. Purpose of the present study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
2. Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
2.1. Literature search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
2.2. Inclusion criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3. Description of the included studies (Table 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3.1. Anderson (2004) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3.2. Campitelli and Labollita (2016) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3.3. Doll and Mayr (1987). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3.4. Hänggi, Brütsch, Siegel, and Jäncke (2014) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3.5. Unterrainer, Kaller, Halsband, and Rahm (2006) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3.6. Unterrainer, Kaller, Leonhart, and Rahm (2011) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.3.7. Vidaurreta (2011) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
2.4. Effect size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

⁎ Corresponding author at: Department of Psychological Sciences, Bedford Street South, University of Liverpool, L69 7ZA, United Kingdom.
E-mail address: [email protected] (G. Sala).
1
Sala G. and Burgoyne A. P. are co-first authors of this article.

http://dx.doi.org/10.1016/j.intell.2017.01.013
0160-2896/© 2017 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
 G. Sala et al. / Intelligence 61 (2017) 130–139 131

3.1. Main model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

3.2. Outlier analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
3.3. Publication bias analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
3.4. Additional meta-analytic models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.1. What is the size of the “True Effect?” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.2. Limitations of the study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.3. Recommendations for future research and conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

1. Introduction (Hambrick et al., 2014), chess is widely considered an archetypical do-

main for the study of expertise, i.e., the ability to perform better than
Whether intelligence plays a significant role in determining expert the majority of people in a particular domain (Gobet, 2016). In fact, to
performance has been intensely debated for more than a century. emphasize the idea that chess is an ideal model environment for re-
Starting with Galton's (1869) theory of hereditary genius, the idea search in the psychology of expertise, Simon and Chase (1973) indicat-
that expert performance needs intelligence, and hence, that experts ed that the role chess plays in cognitive science is comparable to that of
are on average more intelligent than laypeople, has received substantial Drosophila (i.e., fruit fly) for the field of genetics.
support. Barron (1963) suggested that in some areas, such as physics Chess is also an excellent model environment to investigate the im-
and mathematics, the minimum threshold to achieve graduation is an portance of cognitive ability and deliberate practice––the engagement
IQ of 120. More recently, a longitudinal study showed that SAT scores in specific structured activities designed to improve performance in a
of gifted children at the age of 13 predicted future success (e.g., number field––in the acquisition of expertise (Gobet, 2016; Gobet, De Voogt, &
of scientific publications and patents) in their respective fields Retschitzki, 2004; Grabner, 2014). Achieving mastery in chess requires
(Lubinski, 2009; Robertson, Smeets, Lubinski, & Benbow, 2010). Finally, intensive practice, but other factors seem to play an important role,
in a meta-analytic review, Hunter and Hunter (1984) reported a strong too. For example, Campitelli and Gobet (2011) found that the amount
positive correlation (r = 0.53) between job performance and general in- of deliberate practice necessary to reach expertise in chess varied
telligence (g). Moreover, the strength of this relationship was positively massively between players, suggesting that deliberate practice
related to job complexity. These results were confirmed in successive alone cannot explain individual differences in expert performance
meta-analyses (Hunter, Schmidt, & Le, 2006; Schmidt & Hunter, 1998; in chess. Since chess is a complex and intellectual activity, general in-
Schmidt, Shaffer, & Oh, 2008). telligence and more specific cognitive abilities—such as visuospatial
Ericsson (2014) has recently offered a competing view of expertise in ability, short-term and working memory, planning, processing
which he argues that the difference in cognitive ability between experts speed, and problem-solving skills—have been considered as poten-
and non-experts may be due to academic selection processes. Admission tial predictors of individual differences in chess skill (Bilalić,
into undergraduate and postgraduate education programs is often based McLeod, & Gobet, 2007; Burgoyne et al., 2016; de Bruin, Kok,
on test scores correlating with general intelligence measures (e.g., SAT Leppink, & Camp, 2014; Frydman & Lynn, 1992; Jastrzembski,
for undergraduate studies, GRE for graduate studies, LSAT for law Charness, & Vasyukova, 2006; Schneider, Gruber, Gold, & Opwis,
school). Therefore, Ericsson (2014) claims that if experts—such as scien- 1993; Waters, Gobet, & Leyden, 2002).
tists or musicians—score better on intelligence tests than non-experts, Unlike other fields of expertise, access to chess practice and instruc-
this may only be because individuals with higher cognitive ability had ac- tion does not depend on performance on standardized tests. That is,
cess to the training that made them experts. chess is an activity open to many people without restriction. Joining a
Both views recognize the evidence that experts tend to have greater chess club does not require any preliminary testing, and participation
cognitive ability than their non-expert counterparts in many fields. in courses and tournaments is often open to players of any level and rel-
However, they differ in what is assumed to account for the relationship atively affordable. Certainly, a few chess players are funded by chess
between expertise and cognitive ability. The traditional view holds that federations and admitted to elite tournaments, but the selection is al-
cognitive ability is directly predictive of expertise. That is, experts have ways based mainly (if not completely) on players' chess performance.
greater cognitive ability than non-experts because cognitive ability ac- Chess is thus an ideal domain to test Ericsson's (2014) argument that
counts for some of the variance in performance. The competing view differences in cognitive ability between experts and non-experts are
(Ericsson, 2014) holds that cognitive ability predicts access to training, due to performance on standardized tests limiting access to the domain,
which in turn predicts expertise. That is, according to this view, experts given that no such limitation applies to the field of chess.
have greater cognitive ability than non-experts because those with
greater cognitive ability were selected for training; cognitive ability is 1.2. Purpose of the present study
not directly predictive of expert performance.
To test these competing views, we chose a field—chess—in which To test the academic selection hypothesis, we investigated whether
there is no selection process that limits access to training. According to cognitive ability differences emerge between those who have entered a
the academic selection process hypothesis (Ericsson, 2014), if there field and those who have not in a field where there is no selection test:
are no barriers to training based on cognitive ability, then there should chess. We conducted a meta-analysis across all the studies meeting our in-
be no difference in cognitive ability between chess players who are clusion criteria that measured cognitive ability. This meta-analysis exam-
training in this area of expertise and non-chess players who are not en- ines studies of natural groups—those who have chosen to enter the field
gaged in training. and those who have not2—and thus our results cannot address the causal
structure of a relationship between chess player status and overall cogni-
1.1. Chess and expertise tive ability. However, finding a relationship would support the notion that

Thanks to a reliable indicator of players' strength (Elo rating; Elo, 2

For an analysis of the correlation between chess skill and intelligence in chess players
1978) and a balance between tactical calculation and strategic thinking of different skill levels, see Burgoyne et al. (2016).
132 G. Sala et al. / Intelligence 61 (2017) 130–139

cognitive ability is a strong candidate factor to play a role in expert perfor- Reviews and Meta-Analyses (PRISMA) statement (Moher, Liberati,
mance and that expert vs. non-expert differences in cognitive ability can- Tetzlaff, & Altman, 2009). The entire procedure is summarized in Fig. 1.
not be assumed to only be due to academic selection processes.
2.1. Literature search

2. Method A systematic search strategy was used to find the relevant studies.
Google Scholar, ERIC, Psyc-Info, JSTOR, Scopus, and ProQuest Disserta-
We designed the random-effects meta-analysis and report the re- tions & Theses databases were searched to identify potentially relevant
sults in accordance with the Preferred Reporting Items for Systematic studies. We used two specific combinations of keywords: (a) chess AND

Fig. 1. Flow diagram of the studies included in the meta-analysis.

 G. Sala et al. / Intelligence 61 (2017) 130–139 133

Table 1
The descriptive features of the seven included studies.

Study Variable Test Age Rating Control group matching

Anderson Processing speed d2 Test of Attention 11–14 Unranked Age and level of
(2004) education
Campitelli and Planning; cognitive reflection Tower of London; cognitive 15–55; Ranked (Elo): Age and level of
Labollita reflection test M = 27.8 2159–2607; M = 2405.4 education
(2016) (9.70) (122.0)
Doll and Mayr Numerical; verbal; figural intelligence; processing Berlin Intelligence Structure 18–51; Ranked (Elo): Age
(1987) speed; memory; creativity; information processing M = 25.7 2220–2425; M = 2301.1
capacity (4.9) (54.3)
Hänggi et al. Fluid intelligence; mental rotation ability; visuospatial RAPM; V&K; block-tapping 19–41; Ranked (Elo): Age and level of
(2014) short-term memory test M = 28.9 2187–2560; M = 2366 education
(6.02) (107)
Unterrainer et Planning; fluid intelligence; verbal working memory; Tower of London; SPM; digit M = 29.3 Ranked (DWZa): Age and level of
al. (2006) visuospatial short-term memory span; Corsi block-tapping test (8.6) 1250–2100; education
M = 1683.32
Unterrainer et Planning Tower of London 20–50; Ranked (DWZa): Age, level of education,
al. (2011) M = 32.9 1209–2303; M = 1808.6 and general intelligence
(9.1) (272.8)
Vidaurreta Problem-solving WCST 7–11 Unranked Age and level of
(2011) education
a
DWZ (Deutsche Wertungszahl) is the German equivalent of Elo rating.

(intelligence OR memory OR planning OR cognitive OR ability); and (b) 2.3.2. Campitelli and Labollita (2016)
(chess OR Elo OR DWZ OR Fide) AND (intelligence OR memory OR plan- In this study, chess players (n = 25) and non-chess players (n = 25),
ning OR cognitive OR ability). In addition, previous narrative reviews aged 15–55 years, were administered the Tower of London (a measure
were examined, and we e-mailed (n = 7) researchers in the field asking of planning ability; Shallice, 1982) and the Cognitive Reflection Test
for unpublished studies and data. (Campitelli & Gerrans, 2014). According to a commonly accepted metric
(Elo N 2000; see Gobet & Simon, 2000), all the chess players can be con-
2.2. Inclusion criteria sidered chess experts, and the sample included some international
grandmasters (Elo N 2500). The chess players and non-chess players
To be included in our meta-analysis the study needed to: were matched on age and level of education.6

1. report a comparison between chess players and non-chess players;3 2.3.3. Doll and Mayr (1987)
2. include at least one measure of intelligence;4 and In this study, chess players (n = 27) and non-chess players (n = 88),
3. report an effect size or the authors needed to provide enough infor- aged 18–51 years, were administered the Berlin Intelligence Structure
mation to calculate an effect size. (BIS; Jäger, 1982), a test of general intelligence. The BIS consists of
seven subscales measuring three content-related skills (verbal, numer-
We found seven studies, conducted between 1987 and April, 1, 2016,
ical, and figural) and four operational skills (processing speed, memory,
that met all the inclusion criteria. These studies included seven indepen-
creativity, and information processing capacity). All the chess players in
dent samples and 19 effect sizes,5 with a total of 485 participants.
the sample were titled players (Masters, Fide Masters, and International
Masters). The chess players and non-chess players were matched on
2.3. Description of the included studies (Table 1)
age.
2.3.1. Anderson (2004)
2.3.4. Hänggi, Brütsch, Siegel, and Jäncke (2014)
In this study, chess players (n = 80) and non-chess players (n = 46),
In this study, chess players (n = 20) and non-chess players (n = 20),
aged 11–14 years, were administered the d2 Test of Attention
aged 19–41 years, were administered the Raven's Advanced Progressive
(Brickenkamp & Zillmer, 1998). This test measures processing speed and
Matrices (Raven, 1998), a mental rotation task (Peters et al., 1995), and
discrimination of similar visual stimuli. No measure of chess skill was pro-
a block-tapping test (a measure of visuospatial short-term memory;
vided because the chess players, probably due to their young age, did not
Schellig, 1997). The chess players were all experts, and there were sev-
have an Elo rating (or equivalent). The participants were from two middle
eral international grandmasters (Elo N 2500). The two groups were
schools with open chess courses to their students. The chess players were
matched on age and level of education.
those who had selected to enroll in a chess course. The chess players and
non-chess players were matched on age and years of education.
2.3.5. Unterrainer, Kaller, Halsband, and Rahm (2006)
3 In this study, chess players (n = 25) and non-chess players (n = 25),
Studies with experimental treatments were not included. See Sala and Gobet (2016)
for a meta-analytic review of studies having incorporated an experimental chess mean age 29.3 years, were administered the Tower of London, the Stan-
treatment. dard Progressive Matrices (Raven, 1960), the digit span test, and the
4
The measures of cognitive ability included in the present meta-analysis are varied, but Corsi block-tapping test (Milner, 1971). Chess rating ranged from
the majority correlate moderately or strongly with general intelligence. For example, the 1250 to 2100 (amateurs, intermediates, and experts). The chess players
approximate correlation between full-scale IQ and Raven's Progressive Matrices is 0.80
(Jensen, 1998); between full-scale IQ and mental rotation is 0.41 (Ozer, 1987); between
and non-chess players were matched on age and level of education.
full-scale IQ and block-tapping is 0.38 (Orsini, 1994); between full-scale IQ and the Tower
of London is 0.46 (Morice & Delahunty, 1996); between the SAT and the Cognitive Reflec- 2.3.6. Unterrainer, Kaller, Leonhart, and Rahm (2011)
tion Test is 0.44 (Frederick, 2005); between full-scale IQ and perseverative errors in Wis- In this study, chess players (n = 30) and non-chess players (n = 30),
consin Card Sorting Task is −0.30 (Ardila, Pineda, & Rosselli, 2000); and, between full-
aged 20–50 years, were administered the Tower of London with a time
scale IQ and 2-choice processing speed tasks is 0.41 (Vernon, 1983).
5
To account for effect sizes from dependent samples, we adjusted (i.e., lowered) the
6
weight of such samples using the method designed by Cheung and Chan (2004). For a list Note that the chess players received a compensation equivalent to US$12, whereas the
of the adjusted Ns, see the data file openly available at https://osf.io/t3uk2/. non-chess players received unit credits valid for academic courses.
134 G. Sala et al. / Intelligence 61 (2017) 130–139

Fig. 2. Forest plot of the main model. Cohen's ds (circles) and 95% CIs (lines) are shown for all the effects entered into the meta-analysis. The ds are sorted by magnitude (from smallest to
largest). The diamond at the bottom indicates the meta-analytically weighted mean d. When studies used multiple outcome measures, the plot indicates the result of each measure (M1,
M2, etc.) separately. These effect sizes were adjusted for dependent samples (see footnote 5).

constraint. The chess rating range was 1209–2303. The chess players 3. Results
and non-chess players were matched on age, level of education, and
general intelligence.7 3.1. Main model

The random-effects meta-analytic overall effect size was d = 0.49, CI

2.3.7. Vidaurreta (2011)
[0.26; 0.72], k = 19, p b 0.001. The forest plot is shown in Fig. 2. The I2—a
In this study, chess players (n = 22) and non-chess players (n = 22),
statistic that specifies the percentage of between-study variability in ef-
aged 7–11 years, were administered the Wisconsin Card Sorting Task
fect sizes due to heterogeneity rather than random error—was 66.56.
(WCST).8 This test measures problem-solving and the ability to adapt
To evaluate whether any effect size exerted a strong effect on the
to changing rules. Probably due to the young age of the participants,
meta-analytic overall effect size, we performed a one-study-removed
no measure of chess skill was available. The chess players were from a
analysis (Borenstein, Hedges, Higgins, & Rothstein, 2009; Kepes,
local chess club. The chess players and non-chess players were matched
McDaniel, Brannick, & Banks, 2013). This analysis showed that the max-
on age and years of education.
imum difference between the overall effect size (d = 0.49) and the sev-
eral point estimates (min: d = 0.45, CI [0.22; 0.67]; max: d = 0.53, CI
2.4. Effect size [0.31; 0.76]; Fig. 3) was small (±0.04 standard deviations).

The standardized mean difference (Cohen's d) in cognitive ability 3.2. Outlier analysis
scores between the chess players and non-chess players was used as
the effect size. The effect sizes were then corrected for attenuation due To test whether some effect sizes had an unusually large influence
to measurement unreliability (Schmidt & Hunter, 2015) by using the on the overall results, Viechtbauer and Cheung's (2010) outlier detec-
two following formulas: tion analysis was performed. No outliers were found.

d 3.3. Publication bias analysis

0
d ¼ ð1Þ
a
Publication bias occurs when studies with small samples and small
effect sizes are systematically suppressed from the literature. To investi-
Se gate whether our results were affected by publication bias, we exam-
Se0 ¼ ð2Þ
a ined a contour-enhanced funnel plot (Peters, Sutton, Jones, Abrams, &
Rushton, 2008) depicting the relation between effect size and standard
error, and performed Duval and Tweedie's (2000) trim-and-fill analysis.
where d′ is the corrected Cohen's d, Se′ the corrected standard error In the presence of publication bias, effect sizes are missing from the bot-
(Se), and a the square root of the reliability coefficient (see the data tom left part of the funnel plot (small effect sizes with high standard
file openly available at https://osf.io/t3uk2/ for a list of the coefficients error). That is, when standard error is high, larger-than-average effects
and corrected effect sizes and standard errors). Finally, the Comprehen- sizes (those on the bottom right) are more likely to be published than
sive Meta-Analysis (Version 3.0; Biostat, Englewood, NJ) and Metafor smaller-than-average effect sizes (those on the bottom left). The trim-
(Viechtbauer, 2010) software packages were used for conducting the and-fill analysis estimates the number of missing studies from the fun-
meta-analytic, publication bias, and outlier analyses. nel plot and imputes the missing effect sizes based on the observed
data's asymmetry to create a more symmetrical funnel plot.
The funnel plot was approximately symmetrical around the meta-
7
While the groups were matched on general intelligence, an additional measure of cog- analytic mean (d = 0.49; Fig. 4), suggesting no presence of publication
nitive ability was administered (Tower of London) and thus we included it. Given the pat- bias. The trim-and-fill analysis estimated no missing smaller-than-aver-
tern of results—chess players tending to have higher intelligence than non-chess age effect sizes either left or right of the mean.
players—it is likely that our overall effect size would be larger had the study authors not Cumulative meta-analysis (Borenstein et al., 2009; Schmidt &
matched the chess players and non-chess players on general intelligence.
8
Perseverative errors score is the only measure from the WCST correlating with mea-
Hunter, 2015) showed that the small-N effect sizes did not sensibly af-
sures of full-scale IQ (see Ardila, Pineda, & Rosselli, 2000). Thus, only the data concerning fect the overall effect size, suggesting no presence of publication bias
perseverative errors were used to extrapolate an effect size. (Fig. 5).
 G. Sala et al. / Intelligence 61 (2017) 130–139 135

Fig. 3. Forest plot of the one-study-removed analysis. Each row reports the overall effect size (with 95% CI and p-value) that the model would estimate if the effect size of the corresponding
study were removed from the analysis.

Fig. 4. Contour-enhanced funnel plot of standard errors and effect sizes (ds). The black circles represent the effect sizes included in the meta-analysis. Contour lines are at 1%, 5%, and 10%
levels of statistical significance.

Fig. 5. Forest plot of the cumulative meta-analysis. The effect sizes are sorted by adjusted Ns (from largest to smallest).
136 G. Sala et al. / Intelligence 61 (2017) 130–139

Table 2
Meta-analytic and publication bias outcomes.

Meta-analysis Publication bias analyses

N k d 95% CI 80% Cr. I Q I2 τ osr t&f smm sms PET PEESE

485 19 0.49 0.26, 0.72 0.00, 1.09 53.82 66.56 0.17 0.45, 0.53, 0.50 0.49 0.47 0.40 0.91 (0.28) 0.65 (0.35)

Note. N = number of the participants; k = number of the effect sizes; d = random effects meta-analytic mean; 95% CI = 95% confidence intervals; 80% Cr. I = 80% credibility intervals; Q =
weighted sum of square deviations from the mean; I2 = ratio of true heterogeneity; τ = between-sample standard deviation; osr = one-sample removed: minimum, maximum, and me-
dian overall effect sizes; t&f = trim-and-fill; smm = two-tailed moderate selection model; sms = two-tailed moderate selection model; PET = precision effect test (one-tailed p-value in
brackets); PEESE = precision effect estimate with standard error (one-tailed p-value in brackets).

Fig. 6. Forest plot of the one-study-removed analysis without Doll and Mayr (1987). Each row reports the overall effect size (with 95% CI and p-value) that the model would estimate if the
effect size of the corresponding study were removed from the analysis. The ds are sorted by magnitude (from smallest to largest).

Table 3
Meta-analytic and publication bias outcomes (without Doll & Mayr, 1987).

Meta-analysis Publication bias analyses

N k d 95% CI 80% Cr. I Q I2 τ osr t&f smm sms PET PEESE

a
370 12 0.38 0.10, 0.66 −0.18, 0.93 27.79 60.41 0.15 0.29, 0.44, 0.39 0.44 0.35 0.30 0.35 (0.33) 0.08 (0.34)

Note. N = number of the participants; k = number of the effect sizes; d = random effects meta-analytic mean; 95% CI = 95% confidence intervals; 80% Cr. I = 80% credibility intervals; Q =
weighted sum of square deviations from the mean; I2 = ratio of true heterogeneity; τ = between-sample standard deviation; osr = one-sample removed: minimum, maximum, and me-
dian overall effect sizes; t&f = trim-and-fill; smm = two-tailed moderate selection model; sms = two-tailed moderate selection model; PET = precision effect test (its one-tailed p-value in
brackets); PEESE = precision effect estimate with standard error (its one-tailed p-value in brackets).
a
One effect size filled right of the mean. No missing effect sizes left of the mean.

Finally, using multiple methods for the detection of publication bias analytical model without the effect sizes extracted from that study.
is important to test the robustness of results (Kepes & McDaniel, 2015). The random-effects meta-analytic overall effect size was d = 0.38, CI
We thus ran two additional publication bias analyses: (a) selection [0.10; 0.66], k = 12, p = 0.009. The degree of heterogeneity was I2 =
models (moderate two-tailed selection and severe two-tailed 60.41. One-study-removed analysis showed that the maximum differ-
selection;9 Vevea & Woods, 2005); and (b) PET-PEESE (Stanley & ence between the overall effect size (d = 0.38) and the several point es-
Doucouliagos, 2014). The results of the two analyses showed either
timates (min:d = 0.29, CI [0.05; 0.53]; max: d = 0.44, CI [0.16; 0.72]; Fig.
minimal differences between the overall effect size (d = 0.49) and the 6) was small (− 0.09 standard deviations). Consistent with the one-
point estimates, or non-significant point estimate (PET-PEESE). All the study-removed analysis, Viechtbauer and Cheung's (2010) outlier de-
results are summarized in Table 2. tection analysis found no outliers.
With regard to publication bias analyses, the pattern of results was
3.4. Additional meta-analytic models similar to the main model, that is small differences between the overall
effect sizes and the point estimate (PET-PEESE point estimates were still
Doll and Mayr (1987) was the only study not controlling for the level not significant). All the results are summarized in Table 3, while cumu-
of education of the participants. Thus, we ran a random-effects meta- lative meta-analysis is shown in Fig. 7.
Finally, four studies (i.e., Doll & Mayr, 1987; Campitelli & Labollita,
9
The two-tailed selections were preferred over the one-tailed selections because of the 2016; Hänggi et al., 2014; Unterrainer et al., 2006) reported statistically
particular features of the studies included in this meta-analysis. In fact, several studies dependent effect sizes (k = 16). To evaluate whether those effect sizes
used the data analyzed here as control variables (e.g., Hänggi et al., 2014), or tried to sup- meaningfully affected the overall effect size, we calculated the weighted
port the null hypothesis (i.e., no significant difference between chess players and non- average effect sizes for three of these studies, while, for Doll and Mayr
chess players; Unterrainer et al., 2011). Moreover, some studies reported more than one
variable regardless of the significance of the outcomes (e.g., Doll & Mayr, 1987). Thus,
(1987), we used the reported full-scale measure of the BIS (see
there is no reason to think that high p-values were more likely to be suppressed from Section 2.3 and Table 1). We thus inserted the merged effect sizes in a
the literature (as assumed by one-tailed selections). new meta-analytic model. The random-effects meta-analytic overall
 G. Sala et al. / Intelligence 61 (2017) 130–139 137

Fig. 7. Forest plot of the cumulative meta-analysis without Doll and Mayr (1987). The effect sizes are sorted by adjusted Ns (from largest to smallest).

effect size was slightly smaller but still significant, d = 0.39, CI [0.06; lies between 0.49 and 0.40 (the effect size of the main model and its
0.73], k = 7, p = 0.022. Due to scarcity of the effect sizes, no additional smallest point estimate from publication bias analysis, respectively). If
analysis was performed for this model. we assume that these results are unreliable, then the mean estimate is
reasonably between 0.44 (i.e., the estimation of the trim-and-fill analy-
4. Discussion sis) and 0.35 (i.e., the estimation of the two-tailed moderate selection
model).10 Either way, the size of the effect representing the superiority
Ericsson (2014) argued that observed differences in cognitive ability of chess players over non-chess players in overall cognitive ability ap-
between experts and non-experts are likely due to academic selection pears to be approximatively medium (Cohen, 1988).
processes. That is, those who have higher cognitive ability are more like-
ly to become experts because admission tests limit opportunities for
those with lower cognitive ability to enter the field and receive training. 4.2. Limitations of the study
Chess does not use academic selection mechanisms, which makes it an
ideal domain to test whether cognitive ability differences emerge with- The total numbers of effect sizes (k = 19) and studies (n = 7; n = 6
out the presence of confounding access limitations. We analyzed the without Doll & Mayr, 1987) are relatively small. For this reason, the out-
performance of chess players compared to non-chess players in cogni- come of the study, although statistically significant, must be interpreted
tive abilities related to general intelligence. The results demonstrated with caution. In addition, although chess players outperformed non-
that chess players' overall cognitive ability is higher than age-matched chess players in almost all the reviewed cases, regardless of the partici-
pants' age and type of cognitive ability measured, the limited number of
comparison groups (d = 0.49, p b 0.001). Importantly, the difference
studies and effect sizes prevented us from testing for moderating effects
in cognitive ability between chess players and non-chess players does
of age and type of ability. For example, the difference between chess
not seem to be due to level of education, because this variable was con-
players and non-chess players may be more pronounced in youth
trolled for in almost all the reviewed studies. When the only study not
than in adulthood (or vice versa). Moreover, this meta-analysis esti-
controlling for level of education (i.e., Doll & Mayr, 1987) was excluded
mates an overall effect size from indicators of different cognitive abili-
from the analysis, the overall effect size remained significant (d = 0.38,
ties. In fact, the moderate degree of heterogeneity between effect sizes
p = 0.009). Thus, we found no evidence for Ericsson's (2014) claim that
(I2 = 66.56) indicates that the distribution is sufficiently homogenous
expert vs. non-expert differences in cognitive ability only reflect ability-
to be aggregated in the same model (see footnote 4). Thus, the results
related differences in access to training opportunities.
support the hypothesis that chess players tend to have higher overall
This finding, combined with the results of Burgoyne et al. (2016), a
cognitive ability, but little can be inferred about the difference between
meta-analysis that found a significant positive correlation between
chess players and non-chess players in particular cognitive abilities. For
chess skill and cognitive ability, provides evidence in favour of the
example, chess players may excel only in specific cognitive abilities (e.g.,
idea that cognitive ability, to some extent, accounts for the acquisition
visuospatial ability, working memory) but not in measures of verbal
of chess skill. Given that deliberate practice appears to be necessary,
reasoning. Additionally, it is yet to be determined whether the differ-
but not sufficient, to achieve high levels of expert performance in
ence between chess players' and non-chess players' cognitive ability is
chess (Campitelli & Gobet, 2011; Hambrick et al., 2014), it is important
stronger when non-chess players are compared to chess experts
to identify what other factors influence expertise acquisition in chess.
(Elo N 2000) and masters (Elo N 2200), or to amateurs and intermediate
players.
4.1. What is the size of the “True Effect?” Finally, our results specifically speak to the domain of chess. It is pos-
sible that the academic selection hypothesis holds in other intellectual
The advantage of chess players in overall cognitive ability remains domains. However, our results indicate that, at a minimum, the academ-
statistically significant in all the meta-analytic models. However, the es- ic selection hypothesis does not hold for chess and therefore cannot be
timation of the effect size ranges between d = 0.49 (i.e., the effect size of generalized to all intellectual domains. Future research is needed to test
the main model) and d = 0.30 (i.e., the point estimate from the severe
selection model without Doll & Mayr, 1987): a not negligible difference 10
The value provided by the severe two-tailed selection model (i.e., 0.30) is very likely to
of 0.19 standard deviations. be an underestimation. First, the trim-and-fill analysis found a missing study right of the
Evaluating the most likely estimate of the effect size depends mainly mean (point estimate 0.44), which means the overall meta-analytic mean (d = 0.38)
on the reliability of the results of Doll and Mayr (1987). Unfortunately, may be an underestimation rather than an overestimation. Second, as mentioned in foot-
note 9, the assumption that non-significant effect sizes––especially the ones with p-values
not enough information was provided to verify whether the educational between 0.10 and 0.90––have been systematically suppressed from the literature is prob-
level of the participants affected the results. If we assume that Doll and ably wrong. For these reasons, we consider the moderate selection model point estimate
Mayr's (1987) results are trustworthy, then the probable mean estimate (i.e., 0.35) more reliable than the severe selection model one and sufficiently conservative.
138 G. Sala et al. / Intelligence 61 (2017) 130–139

whether the academic selection hypothesis accurately reflects expertise Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel plot based method of testing
and adjusting for publication bias in meta-analysis. Biometrics, 56, 276–284. http://dx.
in any domain. doi.org/10.1111/j.0006-341X.2000.00455.x.
Elo, A. (1978). The rating of chessplayers, past and present. New York: Arco.
Ericsson, K. A. (2014). Why expert performance is special and cannot be extrapolated
4.3. Recommendations for future research and conclusions from studies of performance in general population: A response to criticisms.
Intelligence, 45, 81–103. http://dx.doi.org/10.1016/j.intell.2013.12.001.
Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic
As mentioned above, the number of studies comparing chess players Perspectives, 19, 25–42. http://dx.doi.org/10.1257/089533005775196732.
to non-chess players on cognitive skills is quite small. Additional studies Frydman, M., & Lynn, R. (1992). The general intelligence and spatial abilities of gifted
young Belgian chess players. British Journal of Psychology, 83, 233–235. http://dx.
are needed that compare chess players of different ages and ratings to doi.org/10.1111/j.2044-8295.1992.tb02437.x.
matched non-chess players on a variety of cognitive tests. A larger num- Galton, F. (1869). Hereditary genius: An inquiry into its laws and consequences. London:
ber of studies examining variables such as level of skill, the age of MacMillan.
Gobet, F. (2016). Understanding expertise: A multi-disciplinary approach. London: Palgrave.
players, and a broad range of cognitive abilities would allow us to test Gobet, F., & Simon, H. A. (2000). Five seconds or sixty? Presentation time in expert mem-
more sophisticated models and hypotheses. ory. Cognitive Science, 24, 651–682. http://dx.doi.org/10.1016/S0364-
Based on the available evidence, we demonstrated that chess players 0213(00)00031-8.
Gobet, F., de Voogt, A., & Retschitzki, J. (2004). Moves in mind: The psychology of board
tend to perform better than non-chess players on measures of cognitive games. Hove, UK: Psychology Press.
ability. Consequently, cognitive ability is a strong candidate to play an Grabner, R. H. (2014). The role of intelligence for performance in the prototypical exper-
important role, together with deliberate practice, in skill acquisition in tise domain of chess. Intelligence, 45, 26–33. http://dx.doi.org/10.1016/j.intell.2013.
07.023.
chess. Contrary to Ericsson's (2014) hypothesis, this difference in cogni-
Hambrick, D. Z., Oswald, F. L., Altmann, E. M., Meinz, E. J., Gobet, F., & Campitelli, G. (2014).
tive ability cannot be explained by academic selection processes, be- Deliberate practice: Is that all it takes to become an expert? Intelligence, 45, 34–45.
cause no such processes operate in chess. As previously mentioned, http://dx.doi.org/10.1016/j.intell.2013.04.001.
⁎Hänggi, J., Brütsch, K., Siegel, A. M., & Jäncke, L. (2014). The architecture of the chess
deliberate practice alone is insufficient to account for differences in
player's brain. Neuropsychologia, 62, 152–162. http://dx.doi.org/10.1016/j.
chess skill among chess players (Campitelli & Gobet, 2011; Hambrick neuropsychologia.2014.07.019.
et al., 2014). In line with that finding, this study suggests that cognitive Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job per-
ability may also play an important role. To determine whether the effect formance. Psychological Bulletin, 96, 72–98. http://dx.doi.org/10.1037/0033-2909.96.
1.72.
found in this study goes above and beyond the effect of deliberate prac- Hunter, J. E., Schmidt, F. L., & Le, H. (2006). Implications of direct and indirect range re-
tice on chess skill, studies should follow Bilalić et al.'s (2007) recom- striction for meta-analysis methods and findings. Journal of Applied Psychology, 91,
mendation of measuring both deliberate practice and general abilities. 594–612. http://dx.doi.org/10.1037/0021-9010.91.3.594.
Jäger, A. O. (1982). Mehrmodale Klassifikation von Intelligenzleistungen: Experimentell
kontrollierte Weiterenwicklung eines deskriptiven Intelligenzstrukturmodells. [Mul-
timodal classification of intelligence achievement: Experimentally controlled further
Acknowledgements development of a descriptive structure model of intelligence]. Diagnostica, 28,
195–225.
Jastrzembski, T. S., Charness, N., & Vasyukova, C. (2006). Expertise and age effects on
We thank Josef Unterrainer for providing unpublished data, and knowledge activation in chess. Psychology and Aging, 21, 401. http://dx.doi.org/10.
Sven Kepes and Tom Stanley for their assistance with the publication bi- 1037/0882-7974.21.2.401.
as analyses. Jensen, A. R. (1998). The g factor: The science of mental ability. Westport, CT: Praeger.
Kepes, S., & McDaniel, M. A. (2015). The validity of conscientiousness is overestimated in
the prediction of job performance. PLoS ONE, 10, e0141468. http://dx.doi.org/10.
References11 1371/journal.pone.0141468.
Kepes, S., McDaniel, M. A., Brannick, M. T., & Banks, G. C. (2013). Meta-analytic reviews in
⁎Anderson, T. L. (2004). The relation between gender, age, giftedness, and chess activity and the organizational sciences: Two meta-analytic schools on the way to MARS (the
attention in middle school student. TX: Houston: University of Houston (Doctoral Meta-analytic Reporting Standards). Journal of Business and Psychology, 28,
dissertation). 123–143. http://dx.doi.org/10.1007/s10869-013-9300-2.
Ardila, A., Pineda, D., & Rosselli, M. (2000). Correlation between intelligence test scores Lubinski, D. (2009). Exceptional cognitive ability: The phenotype. Behavior Genetics, 39,
and executive function measures. Archives of Clinical Neuropsychology, 15, 31–36. 350–358. http://dx.doi.org/10.1007/s10519-009-9273-0.
http://dx.doi.org/10.1016/S0887-6177(98)00159-0. Milner, B. (1971). Interhemispheric differences in the localization of psychological pro-
Barron, F. (1963). Creativity and psychological health. Princeton, NJ: Van Nostrand. cesses in man. British Medical Bulletin, 27, 272–277.
Bilalić, M., McLeod, P., & Gobet, F. (2007). Does chess need intelligence? A study with Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for system-
young chess players. Intelligence, 35, 457–470. http://dx.doi.org/10.1016/j.intell. atic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151,
2006.09.005. 264–269. http://dx.doi.org/10.7326/0003-4819-151-4-200908180-00135.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta- Morice, R., & Delahunty, A. (1996). Frontal/executive impairments in schizophrenia.
analysis. London: Wiley. Schizophrenia Bulletin, 22, 125–137. http://dx.doi.org/10.1093/schbul/22.1.125.
Brickenkamp, R., & Zillmer, E. (1998). The d2 Test of Attention (1st US ed.). Seattle, WA: Orsini, A. (1994). Corsi's block-tapping test: Standardization and concurrent validity with
Hogrefe & Huber Publishers. WISC-R for children aged 11 to 16. Perceptual and Motor Skills, 79, 1547–1554. http://
Burgoyne, A. P., Sala, G., Gobet, F., Macnamara, B. N., Campitelli, G., & Hambrick, D. Z. dx.doi.org/10.2466/pms.1994.79.3f.1547.
(2016). The relationship between cognitive ability and chess skill: A comprehensive Ozer, D. J. (1987). Personality, intelligence, and spatial visualization: Correlates of mental
meta-analysis. Intelligence, 59, 72–83. http://dx.doi.org/10.1016/j.intell.2016.08.002. rotations test performance. Journal of Personality and Social Psychology, 53, 129–134.
Campitelli, G., & Gerrans, P. (2014). Does the cognitive reflection test measure cognitive http://dx.doi.org/10.1037/0022-3514.53.1.129.
reflection? A mathematical modeling approach. Memory & Cognition, 42, 434–447. Peters, M., Laeng, B., Latham, K., Jackson, M., Zaiyouna, R., & Richardson, C. (1995). A redrawn
http://dx.doi.org/10.3758/s13421-013-0367-9. Vandenberg and Kuse mental rotations test – Different versions and factors that affect
Campitelli, G., & Gobet, F. (2011). Deliberate practice: Necessary but not sufficient. Current performance. Brain and Cognition, 28, 39–58. http://dx.doi.org/10.1006/brcg.1995.1032.
Directions in Psychological Science, 20, 280–285. http://dx.doi.org/10.1177/ Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-en-
0963721411421922. hanced meta-analysis funnel plots help distinguish publication bias from other
⁎Campitelli, G., & Labollita, M. (2016). Campitelli-Labollita_Datset_Chessplayers&Students. causes of asymmetry. Journal of Clinical Epidemiology, 61, 991–996. http://dx.doi.
figshare. http://dx.doi.org/10.6084/m9.figshare.3172813.v1. org/10.1016/j.jclinepi.2007.11.010.
Cheung, S. F., & Chan, D. K. (2004). Dependent effect sizes in meta-analysis: Incorporating Raven, J. (1960). Guide to the standard progressive matrices. London: H. K. Lewis.
the degree of interdependence. Journal of Applied Psychology, 89, 780–791. http://dx. Raven, J. (1998). Advanced progressive matrices. Oxford: Oxford Psychology Press.
doi.org/10.1037/0021-9010.89.5.780. Robertson, K. F., Smeets, S., Lubinski, D., & Benbow, C. P. (2010). Beyond the threshold hy-
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale; NJ: Law- pothesis. Current Directions in Psychological Science, 19, 346–351. http://dx.doi.org/10.
rence Erlbaum. 1177/0963721410391442.
de Bruin, A. B., Kok, E. M., Leppink, J., & Camp, G. (2014). Practice, intelligence, and enjoy- Sala, G., & Gobet, F. (2016). Do the benefits of chess instruction transfer to academic and
ment in novice chess players: A prospective study at the earliest stage of a chess ca- cognitive skills? A meta-analysis. Educational Research Review, 18, 46–57. http://dx.
reer. Intelligence, 45, 18–25. http://dx.doi.org/10.1016/j.intell.2013.07.004. doi.org/10.1016/j.edurev.2016.02.002.
⁎Doll, J., & Mayr, U. (1987). Intelligenz und Schachleistung—Eine Untersuchung an Schellig, D. (1997). Block-tapping-test. Frankfurt: Swets Test Services.
Schachexperten [Intelligence and performance in chess—A study of chess experts]. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in per-
Psychologische Beiträge, 29, 270–289. sonnel psychology: Practical and theoretical implications of 85 years of research find-
ings. Psychological Bulletin, 124, 262–274. http://dx.doi.org/10.1037/0033-2909.124.2.
11
References marked with an asterisk indicate studies included in the meta-analysis. 262.
 G. Sala et al. / Intelligence 61 (2017) 130–139 139

Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in ⁎Unterrainer, J. M., Kaller, C. P., Leonhart, R., & Rahm, B. (2011). Revising superior planning
research findings (3rd ed.). Newbury Park, CA: Sage. performance in chess players: The impact of time restriction and motivation aspects.
Schmidt, F. L., Shaffer, J. A., & Oh, I. S. (2008). Increased accuracy for range restriction cor- American Journal of Psychology, 124, 213–225. http://dx.doi.org/10.5406/amerjpsyc.
rections: Implications for the role of personality and general mental ability in job and 124.2.0213.
training performance. Personnel Psychology, 61, 827–868. http://dx.doi.org/10.1111/j. Vernon, P. A. (1983). Speed of information processing and general intelligence.
1744-6570.2008.00132.x. Intelligence, 7, 53–70. http://dx.doi.org/10.1016/0160-2896(83)90006-5.
Schneider, W., Gruber, H., Gold, A., & Opwis, K. (1993). Chess expertise and memory for Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity anal-
chess positions in children and adults. Journal of Experimental Child Psychology, 56, ysis using a priori weight functions. Psychological Methods, 10, 428–443. http://dx.doi.
328–349. http://dx.doi.org/10.1006/jecp.1993.1038. org/10.1037/1082-989X.10.4.428.
Shallice, T. (1982). The Tower of London test. Colorado, US: Colorado Assessment Tests ⁎Vidaurreta, L. R. (2011). Aproximación al estudio de la flexibilidad cognitiva en niños
(CAT). ajedrecistas [Approach to the study of cognitive flexibility in children chess players].
Simon, H. A., & Chase, W. G. (1973). Skill in chess. American Scientist, 61, 393–403. Rev. Cub. Med. Dep. & Cul. Fís., 6.
Stanley, T. D., & Doucouliagos, H. (2014). Meta-regression approximations to reduce pub- Viechtbauer, W. (2010). Conducting meta-analysis in R with the metafor package. Journal
lication selection bias. Research Synthesis Methods, 5, 60–78. http://dx.doi.org/10. of Statistical Software, 36, 1–48 Retrieved from http://brieger.esalq.usp.br/CRAN/web/
1002/jrsm.1095. packages/metafor/vignettes/metafor.pdf
⁎Unterrainer, J. M., Kaller, C. P., Halsband, U., & Rahm, B. (2006). Planning abilities Viechtbauer, W., & Cheung, M. W. L. (2010). Outlier and influence diagnostics for meta-
and chess: A comparison of chess and non-chess players on the Tower of London analysis. Research Synthesis Methods, 1, 112–125. http://dx.doi.org/10.1002/jrsm.11.
task. British Journal of Psychology, 97, 299–311. http://dx.doi.org/10.1348/ Waters, A. J., Gobet, F., & Leyden, G. (2002). Visuospatial abilities in chess players. British
000712605X71407. Journal of Psychology, 30, 303–311. http://dx.doi.org/10.1348/000712602761381402.