University of Wollongong

Research Online

Faculty of Social Sciences - Papers (Archive) Faculty of Arts, Social Sciences & Humanities

1-1-2018

Effect of Neurocognitive Training for Children With ADHD at Improving

Academic Engagement in Two Learning Settings
Han Jiang
Zhejiang Normal University, [email protected]

Stuart J. Johnstone
University of Wollongong, [email protected]

Li Sun
Peking University

Dawei Zhang
University of Wollongong, [email protected]

Follow this and additional works at: https://ro.uow.edu.au/sspapers

Part of the Education Commons, and the Social and Behavioral Sciences Commons

Research Online is the open access institutional repository for the University of Wollongong. For further information
contact the UOW Library: [email protected]
Effect of Neurocognitive Training for Children With ADHD at Improving Academic
Engagement in Two Learning Settings

Abstract
Objective: This preliminary study investigated effectiveness of neurocognitive training on academic
engagement (AET) for children with ADHD. The training approach targeted working memory, inhibitory
control, and attention/relaxation (via brain electrical activity).

Method: A reversal design with a 2-week follow-up was used to assess the effectiveness of the treatment
on two children with diagnosed ADHD in two learning settings. Direct observation was used to collect
academic-related behavior.

Results: Improvements in on-task expected behavior (ONT-EX) and general AET, as well as reductions in
off-task motor activity (OFF-MA) and off-task passive behavior (OFF-PB) were observed for both students
over baselines and across the settings. Moreover, differences in behavioral change were found between
participants and settings.

Conclusion: These findings support using the treatment for improving academic performance of children
with ADHD. Future studies may investigate influences of contextual differences, nontreatment variables,
or adult's feedback during the training session on treatment effectiveness.

Disciplines
Education | Social and Behavioral Sciences

Publication Details
Jiang, H., Johnstone, S. J., Sun, L. & Zhang, D. (2018). Effect of Neurocognitive Training for Children With
ADHD at Improving Academic Engagement in Two Learning Settings. Journal of Attention Disorders,
online first 1-18.

This journal article is available at Research Online: https://ro.uow.edu.au/sspapers/4422

This article originally published as: Jiang, H., Johnstone, S. J., Sun, L. & Zhang, D. (2018). Effect of Neurocognitive Training for
Children With ADHD at Improving Academic Engagement in Two Learning Settings. Journal of Attention Disorders, online first 1-18.

Neurocognitive training for improving academic engagement

Abstract

This preliminary study investigated effectiveness of neurocognitive training on

academic engagement for children with AD/HD. The training approach targeted working

memory, inhibitory control, and attention/relaxation (via brain electrical activity). A reversal

design with a two-week follow-up was used to assess the effectiveness of the treatment on

two children with diagnosed AD/HD in two learning settings. The results of direct

observation indicated improvements in on-task expected behavior (ONT-EX) and general

academic engagement (AET), as well as, reductions in off-task motor activity (OFF-MA) and

off-task passive behavior (OFF-PB) for both students over baselines and across the settings.

Moreover, differences in behavioral change were found between participants and settings.

These findings support using the treatment for improving academic performance of children

with AD/HD. Future studies may investigate influences of contextual differences, non-

treatment variables or adult’s feedback during the training session on treatment effectiveness.

Keyword: AD/HD; neurocognitive; training; EEG; neurofeedback; academic

engagement; learning settings

1
Neurocognitive training for improving academic engagement

Effect of neurocognitive training for children with AD/HD at improving academic

engagement in two learning settings

Attention-deficit hyperactivity disorder (AD/HD) is one of the most common

developmental disorders of childhood. About 5% of school children are affected by this

disorder, with boys two times more likely to be diagnosed than girls (American Psychiatric

Association, 2013). The core symptoms are age-inappropriate and persistent patterns of

inattentive, impulsive, and hyperactive behaviors. Children with AD/HD often encounter

impairments in school performance, including lower academic achievement (Langberg et al.,

2011; Rodriguez et al., 2007), poorer relatedness (e.g., low belongingness with the teachers)

in the classroom (Rogers & Tannock, 2013), a higher level of graduation failure (Pingault,

Côté, Vitaro, Falissard, Genolini, & Tremblay, 2014), and a larger proportion of school

suspension or expulsion (Parker et al., 2015).

During a lesson, typically-developing children can remain on-task during academic

activities, and quickly and easily switch from off-task to on-task. Children with AD/HD,

however, may display dysfunctions in these attentive traits (Imeraj et al., 2013). In addition,

when compared with typically-developing peers, children with AD/HD have demonstrated

2
Neurocognitive training for improving academic engagement

significantly lower levels of sustained attention (Egeland, Johansen, & Ueland, 2009), and

significantly higher levels of off-task behavior (Kofler, Rapport, & Alderson, 2008), motoric

activities (e.g., rolling chair, leaving seat; Sarver, Rapport, Kofler, Raiker, & Friedman,

2015), and disruptive behaviors (Liu, Huang, Kao, & Gau, 2017). Severities of off-task

behavior are associated with different levels of inattention displayed by children with

AD/HD. Rapport, Kofler, Alderson, Timko, and DuPaul (2009) found that children with

severe inattentive symptoms, in comparison with peers with less severe inattentive

symptoms, spent twice as long off-task. According to Rapport, Scanlan, and Denney’s (1999)

research on developmental pathways of AD/HD, inattentive behaviors, rather than other

symptoms (e.g., disruption) are related to later academic underachievement.

Observational studies have indicated that specific learning contexts have

differentiated impacts on academic engagement. While children with AD/HD exhibited less

on-task behavior compared to their typically developed peers, Vile Junod, DePaul, Jitendra,

Volpe, and Cleary (2006) reported that the most significant discrepancy was in activities of

passive engagement (e.g., silently reading academic material), rather than in activities of

active engagement (e.g., talking to a teacher about academic material). In a regular lesson,

3
Neurocognitive training for improving academic engagement

Lauth, Heubeck, and Mackowiak (2006) reported that children with AD/HD exhibited less

time on-task in inconspicuous tasks (e.g., doing math) as compared to the non-AD/HD peers.

Surprisingly, the engagement rate of the AD/HD group was higher in self-initiated activities

(e.g., correcting a peer), and more than two times as high as the non-AD/HD group in other-

initiated activities (e.g., answering a question). When academic content was concerned,

Imeraj and colleagues (2013) reported that children with AD/HD were on-task less during

academic lessons (e.g., mathematics, language, sciences) compared to typically developing

peers. Besides, such a discrepancy was not found in non-academic lessons (e.g., music, arts).

These studies suggest that off-task behavior in children with AD/HD may be influenced by

the learning context. DuPaul and Joshua (2015) indicated that children with AD/HD tend to

respond poorly in contexts that lack scaffolding, have high executive functioning demand, or

require high self-regulation.

Academic engagement in children with AD/HD has been shown to vary based on

different instructional contexts. In a regular lesson, children with AD/HD exhibited a higher

rate of inattentive off-task behavior (e.g., day-dreaming) during whole class instruction,

whereas these children displayed a higher level of disruptive off-task behavior (e.g., leaving

4
Neurocognitive training for improving academic engagement

seat) during instructions with minimal interaction (e.g., silent work; Lauth et al., 2006). It is

possible that instructional settings such as silent work (that involve less external support such

as teacher feedback) or instructional settings such as whole class teaching (that involve less

teacher supervision), are more likely to trigger off-task behaviors in children with AD/HD.

More recently, several studies (Hart, Massetti, Fabiano, Pariseau, & Pelham, 2011; Imeraj et

al., 2013) have identified effects of group size on academic performance in children with

AD/HD. For example, Hart et al. (2011) reported that the rate of on-task behavior was higher

in small group than in independent silent work or whole class instruction. However, the

positive relationship between small group instruction and academic achievement (via

calculating test accuracy) in children with AD/HD was not established in this study. In fact,

test accuracy was lower in small group instruction than the other two group sizes.

Neurocognitive training for children with AD/HD

Non-pharmacological treatments for AD/HD have been developed in several

modalities, such as cognitive training (CT) and neurofeedback training (NF). CT involves the

use of purpose-designed computer software to exercise particular cognitive abilities (e.g.,

working memory, inhibitory control). These tasks typically include performance feedback,

5
Neurocognitive training for improving academic engagement

with task difficulty varied according to performance to promote challenge, engagement, and

learning. Some positive effects of this approach have been reported in specific cognitive

abilities, behavioral outcomes, and academic achievements (Rapport, Orban, Kofler,

Friedman, & Bolden, 2015).

NF is an innovative approach that builds on the premise that functional states of brain

activity (e.g., attention) can be modified through self-regulation of brain electrical activity

(electroencephalogram [EEG]; Lofthouse, Arnold, Hersch, Hurt, & DeBeus, 2012). Research

has identified certain types of EEG activity that are associated with the core symptoms of

AD/HD (see the review by Barry, Clarke, & Johnstone, 2003). The typical goal of NF in

AD/HD is enhancing higher frequency brain activity (e.g. alpha and beta) and inhibiting

lower frequency activity (e.g. theta and delta). Relatively new consumer-level EEG recording

devices allow for simple, valid, and reliable measurement of brain activity (Johnstone,

Blackman, & Bruggemann, 2012; Rogers, Aminov, Wilson, & Johnstone, 2016) that can be

used in conjunction with computer software to achieve NF goals. A growing body of research

has reported that NF promotes cognitive functions, improves AD/HD symptoms, and

psychosocial outcomes (e.g., self-regulation skills; see reviews by Lofthouse et al., 2012;

6
Neurocognitive training for improving academic engagement

Mayer, Wyckoff, & Strehl, 2013). A meta-analytic review of non-pharmacological treatments

for AD/HD (Hodgson, Hutchinson, & Denson, 2014) suggested that NF outperforms some

other types of treatments (e.g., behaviour modification, parent training) in the average

weighted effect size of outcome measures.

The training approach used in the current study brings together CT and NF to exercise

and improve cognitive and state-control functions; due to this combination of training targets

it is referred to as “neurocognitive” training. The neurocognitive approach is built on the

cognitive energetic model (CEM) of AD/HD (Sergeant, 2005a, 2005b), which proposes that

AD/HD stems from a state-regulation dysfunction that affects efficient engagement of

computational/cognitive processes and executive functions. The neurocognitive approach

targets fundamental cognitive processes such as working memory and inhibitory control, as

well as the psychological state factors of attention and relaxation via NF (Johnstone, 2013).

When functioning effectively, these processes interact to provide a foundation for an

individual’s effective engagement with information in their external world. It is thought that

targeting these three areas in each training session will take advantage of the dynamic

7
Neurocognitive training for improving academic engagement

interplay between them, as outlined by CEM, resulting in improved processing abilities in

these areas and better training outcomes.

Early work examined the efficacy of CT targeting both working memory and

inhibitory control in children with AD/HD (Johnstone, Roodenrys, Phillips, Watt, & Mantz,

2010). Thirty-eight children diagnosed with the combined presentation (AD/HD-C)

undertook 25 training sessions at home using purpose-built software over a 5-week period.

Each training session included six games of a response inhibition task and six games of a

working memory task. After training, the participants had improved resting EEG, and their

parents and another potentially less-biased adult observer (such as grandparent, aunt/uncle, or

family friend with 1-2 contact hours with the child per week) rated significant improvements

on AD/HD symptoms. A subsequent CT study added passive attention-monitoring to the

cognitive training via a portable, wireless, dry-sensor EEG recording device (Johnstone,

Roodenrys, et al., 2012) to allow feedback based on task performance and attention level

during the task. After training, significant improvements in spatial working memory, ignoring

distracting stimuli, and sustained attention were measured in the training cohorts compared to

8
Neurocognitive training for improving academic engagement

the waitlist. In addition, the training cohorts showed significant improvements in AD/HD

symptoms as rated by their parents and another potentially less-biased adult observer.

A preliminary study of the combined CT and NF (i.e. neurocognitive) approach

reported behavioral and academic outcomes of 5 Chinese primary school children with

AD/HD. The participants undertook 25 training sessions using purpose-built software over a

5- to 7-week period at home. Each training session contained 14 games: 4 working memory,

4 inhibitory control, and 6 NF (2 for attention, 2 for relaxation, and 2 for combined attention

and relaxation). After training, the participants showed reduced AD/HD symptoms and other

problem behaviors (e.g., social problems) as rated by their parents and teachers. The

participants also showed increased rates of assignment completion during the training and a

4-week follow-up, in comparison with baseline. Recently, in a randomized control study of

the neurocognitive approach with a larger sample size (n=85; Johnstone, Roodenrys,

Johnson, Bonfield, & Bennett, 2017), children in the training condition showed substantial

improvements in AD/HD symptoms and related behaviors such executive functions,

aggression, and externalizing behaviors. There were minor improvements in two of six near-

9
Neurocognitive training for improving academic engagement

transfer tasks, evidence of far-transfer of training effects in four of five far-transfer tasks, and

indications of normalization of atypical EEG features after training.

At present, most studies have reported the effects of neurocognitive training on

academic behaviors (except dependent variable ‘homework completion’ in Jiang &

Johnstone, 2015) based on adult’s reports of child behavior via checklists (Rapport, Orban,

Kofler, & Friedman, 2013). No studies have been conducted to establish a causal relationship

between neurocognitive training and observed learning behaviors. Studies examining the

effect of NF on academic engagement behaviors via direct observation are also limited (see

reviews Cortese et al., 2016; Hodgson et al., 2014). Two randomized control studies

examined effects of computer-based NF or computer-based attention (cognitive) training in

elementary (Steiner, Frenette, Rene, Brennan, & Perrin, 2014) and middle school students

(Steiner, Sheldrick, Gotthelf, & Perrin, 2011). In both studies, academic engagement (active

or passive) and off-task behaviors (motor, verbal, and passive) were observed at three phases,

i.e., before, during, and 6-months after the interventions. The results of two studies showed

improvements in academic engagement and off-task behaviors for the NF conditions

compared to waitlist. Nonetheless, without illustrating observed data for each behavior across

10
Neurocognitive training for improving academic engagement

phases, the findings of these studies could not indicate deviations among these behaviors. It

was unclear which specific behavior(s) were improved by the use of NF.

The purpose of this study

Despite a growing body of research supporting the use of neurocognitive training for

improving cognitive functions and reducing AD/HD symptoms (Chacko, Kofler, & Jarrett,

2014), relatively little research has been conducted examining classroom learning behaviors.

The primary purpose of this study was to extend evaluation of neurocognitive training

outcomes into a real-life educational context. Chacko and colleagues (2014) proposed “next

generation neurocognitive training” would provide “the cortical foundation to improve

children’s ability to fully benefit from adjunctive, skill-based approaches intended to

ameliorate the behavioral, academic, and interpersonal manifestations of the complex

interactions between underlying neurocognitive impairments and the child’s environment” (p.

369). This study was inspired by such a prospect. Further, we wanted to investigate

behavioral outcomes of the current training protocol in learning contexts. It was expected that

findings of this study might provide implications for scaffolding updated training protocols.

11
Neurocognitive training for improving academic engagement

Thus, the primary research objective of present study was to examine the effects of

neurocognitive training on academic engagement behavior. Specifically, we anticipated

increasing on-task behaviors and reducing off-task behaviors through neurocognitive training

and tested whether the effects occurred through a reversal design with a 2-week follow-up.

The secondary research objective was to examine the behavioral outcomes of the

training in different learning contexts. In the present study, we purposefully selected a self-

study room and small classroom as the research contexts. The main difference between the

two contexts was the presence (or not) or external distraction. The small class context had

potential distractors (e.g., interruption by other students, discussion between teacher and

students for solving an academic problem) as it contained other students, while the self-study

room did not. It was hoped that assessing the impact of the training in these two contexts

would have relevance for learning practice.

Method

Setting and Participants

The training was conducted in an after-school care center attached to a public primary

school in an urban area in East China. The center provided after school services for students

12
Neurocognitive training for improving academic engagement

who had difficulties with their homework (between 3 pm and 7 pm), including assistance

with academic learning, supervision of homework completion, and extra-curricular teaching.

These services were allocated by two types of placement: small class (between 10 to 15

students) and one-to-one instruction (one teacher to one student). Students who exhibited

more disruptive behavior and/or severe learning difficulties might be placed in the one-to-one

instruction. To attend this program, students were initially referred by their teachers due to

difficulties in completing their homework (e.g., procrastination, low accuracy). Assessment

of the students’ academic outcomes and behavioral problems were then conducted for their

placements. Students might be involved in more than one placement based on their

performance on different tasks. The program contained 6 teachers and 41 students ranging

from grade 1 to grade 6.

The participants were randomly selected from a pool of qualified students who were

diagnosed with AD/HD and studying in inclusive classrooms. These students often had

learning difficulties. Some of them were at risk of academic failure. Student A was a 6-year

old first-grader, and had attended the after-school program for two months. He received one-

to-one instruction for most of his time in the center due to his disruptive behavior. His

13
Neurocognitive training for improving academic engagement

teachers and parents were concerned about his problem behaviors in engaging classroom

teaching, class work and homework completion, organization, and inappropriate social

interactions with peers (e.g., pushing). According to the homeroom teacher, the student was

off-task most of the time during class. His off-task behaviors included playing with toys,

interrupting teachers/peers, and hiding/wandering in class. He could not complete any in-

class assignment without teacher supervision. Before attending this program, his father spent

about two or three hours every day helping with his assignments at home. The student was

diagnosed with AD/HD-C without comorbidities. On his recent evaluation, he obtained a

full-scale IQ of 68 on the Wechsler Intelligence Scale for Children–Third Edition (Chinese

version; WISC-IV; Wechsler, 2003), and Inattention subscale of 1.67,

Hyperactivity/Impulsivity subscale of 1.56, and Oppositional subscale of 0.5 on parent-rating

SNAP-IV (Chinese version; Swanson et al., 2001). The student was not taking medication or

receiving other treatment for AD/HD during this study.

Student B was a 10-year old five-grader. He had attended the program for one year,

and spent most of his time in small class instruction with occasional one-to-one instruction.

His classroom misbehavior included daydreaming, playing small toys, avoiding tasks, and

14
Neurocognitive training for improving academic engagement

sleeping. According to the homeroom teacher, the student was easily distracted by noises,

people, or things that were irrelevant. He also avoided assignments that he was not good at

(e.g., English grammar) or uninterested in (e.g., writing). The student was diagnosed with the

predominantly inattentive presentation without comorbidities (AD/HD-I). On his recent

evaluation, he obtained a full-scale IQ of 87 on the WISC IV (Chinese version; Wechsler,

2003), and Inattention subscale of 2.11, Hyperactivity/Impulsivity subscale of 0.67, and

Oppositional subscale of 0.75 on parent-rating SNAP-IV (Chinese version; Swanson et al.,

2001). The student was not taking medication or receiving other treatment for AD/HD during

this study.

Independent variable

The independent variable in this study was the computerized neurocognitive training,

implemented with adult feedback. The training protocol consisted of 25 sessions. Each

session consisted of 14 games: 8 CT (i.e. 4 working memory, 4 inhibitory control) and 6 NF.

The 6 NF games included 2 that were controlled by attention level, 2 controlled by relaxation

level, and 2 controlled by combined attention and relaxation index (termed Zen). The

working memory games involved holding information in memory with subsequent recall to

15
Neurocognitive training for improving academic engagement

complete an action. The inhibitory-control games required a tap/press response to frequently

presented “Go” stimuli and the withholding of responses to infrequent “Nogo” stimuli. The

NF games required children to be attentive, relaxed, or in a “Zen” state (i.e., both attentive

and relaxed), with game-play linked to levels of these EEG-derived factors. Completing a

session took 15 to 20 minutes. The difficulty level of each game increased with successful

completion of the previous level and decreased if the previous level was not successfully

completed. All games started at the lowest level of difficulty. Feedback was provided to the

child at the end of each game in the form of a star rating. Zero to 5 stars were awarded

based on performance, linked to Go/Nogo errors, reaction time, and attention level in the

inhibitory-control games, search errors and attention level in the working memory games,

and threshold level and time-above-threshold in the NF games.

The training was conducted in a consistent environment to minimize external

distraction. All sessions were conducted between 3 and 5 pm. The student had a break time of

45min (including had snack and beverage, and play time) between the school and training

sessions. If a student felt tired during the training, he was allowed to pause the training and

16
Neurocognitive training for improving academic engagement

rest for a few minutes before he restarted the training. Each participant completed the

sessions in a quiet self-study room.

A teacher was trained to provide positive feedback to the participants 5 times per

session. The feedback was of the following types: 1. Student performance on previous task,

2. Advice on how to achieve a better game score, 3. Encouragement for the student to

complete a task at a more difficult level, or 4. Answers to the student’s questions. Types and

examples of teacher feedback are listed in Table 1.

Dependent Variables

On-task behaviors. Academic engagement was the primary dependent variable and

was measured in terms of on-task without inappropriate body movements (ONT-EX) and on-

task with spontaneous body movement (ONT-SBM). ONT-EX was operationally defined as

completing an assigned academic task as expected for at least six consecutive seconds of an

interval of ten seconds. Under this status, the student sat still with eyes focusing on the

material, with their hands holding the proper stationary. ONT-SBM was operationally

defined as when a student was completing an assigned academic task with spontaneous and

unrelated body movements for at least six consecutive seconds of an interval of ten seconds.

17
Neurocognitive training for improving academic engagement

During ONT-SBM, the student’s eyes were focusing on the material and doing the

assignment, but they were also engaging in unrelated activities with other parts of their body,

such as rocking motions, kneeling, or playing with a piece of stationary or gadget). Thus, this

type of behavior could be conceptualized as a mid-point between ONT-EX and off-task

behaviors. It may be the case this type of behavior triggers off-task behavior by weakening

student’s attentional focus (For mechanism of brain’s attention networks and problems of

attentive switch between multitasks, see Rothbart & Posner, 2015).

Off-task behaviors. Off-task behavior was separated into off-task motor activity

(OFF-MA) and off-task passive behavior (OFF-PB). The operational definitions were

adapted from the study by Vile and colleagues (2006). OFF-MA was defined as when the

student had exhibited any motor activity that interrupted completion of the assigned academic

task for at least six consecutive seconds of an interval of ten seconds. Examples included

leaving their seat, playing with stationary/toys, or hiding under the desk. OFF-PB was

defined as when a student was passively not doing the assigned task for at least six

consecutive seconds of an interval of ten seconds. Examples included sleeping, day-

dreaming, or staring at the window/door/wall/ceiling. We excluded off-task verbal behavior

18
Neurocognitive training for improving academic engagement

(e.g., talking to peers without teacher’s permission; Vile Junod et al., 2006) because this

behavior was not common in the observed settings.

Data Collection and Data Analysis

Behavioral data. Direct observations were conducted to assess academic engagement

and on/off-task behavior during completion of academic assignments in the self-study room

and small class setting (see below). In an observation, the student independently completed

an assignment (10 minutes for Student A, 20 minutes for Student B) that matched his

academic ability. While doing the assignments, they were allowed to skip items they felt

unsure about.

The self-study room setting was a quiet small room (about 10m2) with a desk and

chair. Before the observation, the researcher guided the student into the room and provided

him with the academic material. The researcher made sure that the student understood the

task requirements before they left the room. The student was required to complete the task on

his own. The student was provided with sufficient stationary (e.g., pencil, ruler) for

completing the academic task, and reminded to use the toilet room before starting the task.

19
Neurocognitive training for improving academic engagement

The small class was set in a larger room (about 25m2). It was a mixed class of

students of all grades, with 15 students who were seated in four rows by four columns. While

the students were doing their academic assignments in the class, a teacher sat in the front of

the class. A student having difficulty could raise their hand to inform the teacher – who

would then assist them. The students were expected to stay in their seat and be quiet.

Whispering, talking, or discussing with other students without teacher’s permissions were not

expected behaviors.

To minimize potential disturbance caused by observation in the self-study room

setting, the student’s performance was audio- and video-recorded. Later, a research assistant

coded presence of target behaviors by watching the videos. Onsite observation was the

primary method of observing the target behaviors in the small class setting. A research

assistant sat in an aisle and remained unobtrusive to the observed student. Each observation

cycle consisted of 10 seconds of observation followed by 2 seconds for recording. A “beep”

sound recorded in a MP3 player with earphones was used to remind the observer of the 10

and 2 second intervals. In addition, video records were used as backup data in case there was

a need to review past scenarios.

20
Neurocognitive training for improving academic engagement

To calculate the percentage of ONT-EX, the number of occurrences of ONT-EX was

divided by the total number of responses in a session and multiplied by 100. The percentage

of ONT-SBM was calculated in the same way. The percentage of academic engagement

(AET) was calculated by dividing the sum of the number of occurrences of ONT-EX and

ONT-SBM by the total number of responses in a session and then multiplied by 100. To

calculate the percentage of OFF-MA, the number of occurrences of OFF-MA was divided by

the total number of responses in a session and then multiplied by 100. The percentage of

OFF-PB was calculated in the same way. Visual inspection of the level and trend of the

targeted behaviors was used to determine individual students’ performance in each phase.

Effect size PAND/Phi was calculated to interpret change of each dependent variables

by reflecting exact non-overlapping proportion between the baseline and other phases.

PAND/Phi is an ES interpretation schemes that is suitable for indicating the magnitude of the

training effect within single-case designs, especially for multiple baseline and multiple

phases designs (Schneider, Goldstein, & Parker, 2008). Strengths of PAND/Phi to interpret

outcomes of single-case designs include: (a) intuitive appeal and link to visual analysis, (b)

data overlap is exact, and (c) no requirement for data normal distribution (Parker & Hagan-

21
Neurocognitive training for improving academic engagement

Burke, 2007, p. 102). The procedure for calculating PAND/Phi within single subjects with a

reverse design was provided by Schneider et al. (2008, pp. 155-157). According to Cohen

(1992), small, medium, and large effect size of Chi-square for a two by two contingency table

for a= 0.05 are 0.10, 0.30, and 0.50 respectively. In the present study, effect sizes were

calculated within two conditions: within the reversal design, and from reversal to follow-up.

The outcomes were used to determine effects of the neurocognitive training on each of the

four behavioral variables (ONT-EX, ONT-SBM, OFF-MA, and OFF-PB) in the two

conditions.

A graduate student with a major in Special Education was trained as the second

observer. Inter-observer agreement was assessed on approximately 26% of observations of

ONT-EX, ONT-SBM, OFF-MA, and OFF-PB of each participant. Kappa (k) indices were

calculated for each observation to determine agreement for Student A (0.84, range= 0.66-1)

and Student B (0.85, range= 0.66-1); for the self-study room setting (0.84, range= 0.66-1) and

the small class setting (0.84, range= 0.66-1); and for ONT-EX (0.86, range= 0.70-0.97),

ONT-SBM (0.83, range= 0.66-1), OFF-MA (0.84, range= 0.66-1), and OFF-PB (0.85, range=

0.66-1). Overall k averaged 0.84 (range= 0.66-1).

22
Neurocognitive training for improving academic engagement

EEG data. The dry-sensor EEG recording device constantly measured EEG activity

during training and was used to (a) control game-play during the state-control games, and (b)

quantify attention level during the working memory and impulse-control games. The device

consists of microchips, embedded firmware, a 10 mm active electrode and ear-clip reference

ground electrode (ThinkGear, Neurosky, San Jose, California) contained within a headset

(MindWave, Neurosky, USA). The EEG was recorded continuously from site Fp1 at 256 Hz

and has been shown to be reliable and valid (Johnstone, Roodenrys, et al., 2012; Rogers et

al., 2016). The device converted the raw signal from the time- to the frequency-domain via an

Fast Fourier Transform, to calculate EEG power in the delta, theta, alpha, and beta frequency

bands (see Johnstone, Roodenrys, et al., 2012 for more information). Based on conversion of

the raw EEG signal to power in the delta, theta, alpha, and beta frequency bands via Fast

Fourier Transform, proprietary algorithms then calculate values representing two independent

psychological state dimensions of ‘‘attention’’ (low to high; highly correlated with power in

the beta EEG band) and ‘‘relaxation’’ (tense to calm; highly correlated with power in the

alpha EEG band). These measures are presented as a value between 0 and 100, enabling the

provision of generalized feedback about ongoing brain activity in a form understood by

23
Neurocognitive training for improving academic engagement

children. This method provides a robust and universal index of ongoing EEG activity that

does not require individual calibration. An additional index, termed “Zen”, was calculated in

the software by averaging the attention and relaxation indices. The state indices were sent to

the PC wirelessly via a Bluetooth connection. This EEG has been reported to be sensitive to

psychological state variations that are relevant to the neurofeedback training goals contained

within the state-control component of this training approach, i.e. high vs. low attention; high

vs. low relaxation (Johnstone et al, 2012). Note that the EEG is used actively in the state-

control training (see Training Session section), but passively during the impulse-control and

working memory training where it simply monitors background attention level and

categorizes attention into low/medium/high/very-high as a multiplier for game points

achieved (x1/x2/x3/x4, respectively). The device constantly monitors electrode impedance

and provides an ongoing numerical representation of its quality. The neurocognitive training

software monitors this value and if sub-standard impedance occurs at any point (e.g. device is

removed, or as a result of substantial head movement), the training game is paused until

acceptable impedance in once again achieved.

Design and Procedures

24
Neurocognitive training for improving academic engagement

A single-subject design with a reversal phase and a follow-up phase was applied to

evaluate the effects of neurocognitive training on academic engagement and off-task

behavior. Written consents were obtained from the parents and the manager of the after-

school care center. During the pre-training phase, a two-hour information session about the

software and EEG device was delivered to the students at the center. The first author served

as the trainer, instructing students on how to play each game, and assisting them to play the

games in a trail session.

The first baseline occurred over a 2-week period prior to commencement of the

training, with behavioral data collected once every 2 days in each setting for each student.

The first intervention phase occurred over a 2-week period during which each student

completed 3 sessions of neurocognitive training each week; behavioral data were collected

once every 2 days in each setting for each student. The second baseline occurred over a 2-

week period prior to commencement of the training, with behavioral data collected once

every 2 days in each setting for each student. The second intervention phase occurred over a

7-week period during which the student completed a session of training 3 times/week, with

behavioral data collected 2 times/week in each setting for each student. Student B missed 2

25
Neurocognitive training for improving academic engagement

sessions in the setting of small class because of family scheduling needs. The follow-up

phase was conducted 2 weeks after termination of the second intervention phase, and

occurred over a 2-week period in which observations were conducted 2 times/week in each

setting for each student. During these research procedures, the students attended the center

and received the service as normal.

Results

Results are described for academic engagement behaviors and off-task behaviors

during completion of academic assignments across five phases (first baseline, first training

phase, second baseline, second training phase, and follow-up) in two academic settings. Each

variable was graphed independently by phase (see Figure 1, 2) and then interpreted by visual

analysis for immediacy, level, and trend. Additionally, effect size PAND/Phi coefficient was

calculated to determine the magnitude of change.

EEG data during the training

Average Focus, Relax, or Zen scores across the 25 training sessions are shown in

Figure 1. Visual inspection of the data suggest some common trends for the three

psychological states for both participants: 1) scores increased over the 25 training sessions, 2)

scores were lowest in the first intervention phase, 3) there were large increases occurred

26
Neurocognitive training for improving academic engagement

during the first intervention phase, 4) scores reduced at sessions 7 and 8, after a period of no-

training, and then started to increase again, and 5) the improvement tended to be more stable

in the last 8-10 sessions (particularly for Relax and Zen). Other trends to note include that

larger increases were observed for Focus than other states, and that compared with Student B,

Student A had lower scores during the first four sessions but his improvements were larger

and more sustained in the later sessions.

On-task behavior

The self-study room setting. Rates of targeted behaviors for each phase are shown in

Figure 2. When completing their academic assignments during the first baseline, both

students demonstrated low rates of ONT-EX and ONT-SBM. Student A’s performances were

lower. All of his ONT-EX and ONT-SBM sessions were below 50% and 10%, respectively.

These resulted in a low level of academic engagement (AET) for Student A (mean = 29%,

range = 15-52%) and a medium level of AET for Student B (mean = 70%, range = 46-94%).

Once the neurocognitive training was introduced, immediate increases and positive growth in

ONT-EX and AET were observed. For Student A, all rates of ONT-EX and AET were higher

27
Neurocognitive training for improving academic engagement

in this phase than the first baseline. Similar cases (except for the 6th session, see Figure 2)

occurred in Student B.

During the second baseline, decreasing trends in ONT-EX and AET were observed.

All rates of AET and most rates of ONT-EX (except for the 13th session for Student A and

the 12th session for Student B, see Figure 2) were lower than in the first intervention phase.

When the training was re-introduced, immediate and sustained improvements in ONT-EX

and AET were noted. This resulted in medium levels of ONT-EX for Student A (mean =

82%, range = 51-97%) and Student B (mean = 80%, range = 59-97%), as well as high levels

of AET for Student A (mean = 91%, range = 82-97%) and Student B (mean = 93%, range =

82-100%). Two weeks after termination of the training gains in ONT-EX were maintained

for Student A (mean = 79%, range = 69-84%) and Student B (mean = 77%, range = 67-75%).

Despite slight decreases, the rates were steadier in this follow-up phase than the second

intervention phase. As for AET, larger and more sustained increases were found in Student B

(mean = 96%, range = 89-100%) than Student A (mean = 83%, range = 76-92%).

A different pattern of changes was demonstrated in ONT-SBM. All rates were lower

than 40% (except for the 5th session of Student B), with changes in this variable were minor

28
Neurocognitive training for improving academic engagement

throughout the phases. Decreasing trends were observed the in two intervention phases. In

particular, an extremely low level (mean = 4%, range = 0-12%) occurred in the late 15

sessions for Student A.

Large effect sizes were obtained for ONT-EX within the reversal design for Student A

(Phi = 0.79) and Student B (Phi = 0.67), as well as throughout the reversal design to follow-

up for Student A (Phi = 0.80) and Student B (Phi = 0.69). Slightly larger effect sizes in the

latter condition suggested maintenance of the positive effect 2-weeks after termination of the

training. Small effect sizes in ONT-SBM for Student A were calculated within the reversal

design (Phi = 0.04) and from the reversal design to follow-up (Phi = 0.10). Negative effect

sizes in this variable for Student B were obtained within the reversal design (Phi = -0.10) and

from the reversal design to follow-up (Phi = -0.02).

The small class setting. Rates of targeted behaviors for each phase are shown in

Figure 3. When completing their academic assignments during the first baseline, both

students demonstrated low rates of ONT-EX and ONT-SBM. In particular, ONT-EX for both

students was lower than 30% (except that the 5th session of Student A was 30%). This

resulted in low rates of AET for Student A (mean = 22%, range = 9-35%) and Student B

29
Neurocognitive training for improving academic engagement

(mean = 44%, range = 40-48%). Once the neurocognitive training was introduced, increases

in ONT-EX were demonstrated for both students. The rate of ONT-EX was three times

(mean = 43%, range = 24-68%) and two times (mean = 47%, range = 32-63%) those

observed in the first baseline for Student A and Student B, respectively. Further, AET values

for all sessions were higher in this training phase than the first baseline for both students.

During the second baseline, decreasing trends in ONT-EX and AET were observed

for both students. When the training was re-introduced, despite increases in the average

percentage of ONT-EX for Student A (mean = 41%, range = 13-86%) and Student B (mean =

45%, range = 27-69%), slight decreasing trends with notable fluctuations were observed for

the two variables. For AET, a slight decreasing trend was found for Student A (mean = 61%,

range = 42-86%), whereas an increasing trend was found for Student B (mean = 63%, range

= 41-75%). During the 2-week follow-up phase, increases with notable fluctuations in ONT-

EX and AET were noted for both students.

A different pattern of changes was demonstrated in ONT-SBM. Most rates were

lower than 40% (except for the 25th and 28th sessions for Student A and the 32nd session for

Student B). More notable fluctuations were found in Student A. For both students, slight

30
Neurocognitive training for improving academic engagement

decreasing trends in the first intervention phase but increasing trends in the second

intervention phase and 2-week follow-up were observed.

Medium effect sizes in ONT-EX were found for Student A within the reversal design

(Phi = 0.30) and from the reversal design to the follow-up (Phi = 0.34), with large effect sizes

found for Student B within the reversal design (Phi = 0.66) and from the reversal design to

the follow-up (Phi = 0.69). As for ONT-SBM, there were small effect sizes within the

reversal design for Student A (Phi = 0.20) and Student B (Phi = 0.18), as well as from the

reversal design to the follow-up for Student A (Phi = 0.15) and Student B (Phi = 0.13).

In summary, these results suggest that neurocognitive training had immediate and

positive effects on ONT-EX and AET for both students in two learning settings. This finding

has been confirmed by effect size coefficients. However, these increasing trends that

maintained throughout from the reversal design to the follow-up occurred in the self-study

setting than the small class. Effectiveness of the training was not demonstrated for ONT-

SBM in both settings.

Off-task Behavior

31
Neurocognitive training for improving academic engagement

The self-study room setting. In regards to OFF-MA, each student performed quite

differently. Student A showed a high rate (mean = 45%, range = 34-79%) during the first

baseline, and once the training was introduced there were immediate and large decreases in

this variable. After some increases in the rate of OFF-MA in the second baseline, immediate

and sustained decreases were found after the training was re-introduced; a trend maintained

in the 2-week follow-up. Most of the rates (except for the 19th and 20th sessions) during these

phases were below 10%. Student B initially showed a low rate of OFF-MA (mean = 5%,

range = 0-10%) and maintained low levels throughout the phases. After minor decreases

during the first intervention phase, a slight increasing trend was found during the second

baseline. Later, slight decreases maintained in the second intervention phase and 2-week

follow-up.

As for OFF-PB, relatively high rates with notable fluctuations were found for Student

A (mean = 26%, range = 14-44%) and Student B (mean = 25%, range = 6-48%). Immediate

and large decreases were found after the training was introduced. Rates of this variable were

at or below 10% for both students. After some increases during the second baseline,

immediate and sustained decreases were observed for both students when the training was re-

32
Neurocognitive training for improving academic engagement

introduced. Most rates of this variable were below 5% (except for the 23rd session for Student

A, and the 22nd and 27th sessions for Student B) for both students. However, such a trend was

not maintained in the 2-week follow-up.

Large effect sizes in OFF-MA were calculated for Student A within the reversal

design (Phi = 0.68) and from the reversal design to the follow-up (Phi = 0.70). Medium effect

sizes were obtained for Student B within the reversal design (Phi = 0.34) and from the

reversal design to the follow-up (Phi = 0.30). As for OFF-PB, large effect sizes were obtained

within the reversal design for Student A (Phi = 0.79) and Student B (Phi = 0.67), as well as

from the reversal design to follow-up for Student A (Phi = 0.60) and Student B (Phi = 0.69).

The small class setting. For OFF-MA, each student performed differently throughout

the phases. Student A demonstrated a high rate of OFF-MA (mean = 47%, range = 18-65%)

during the first baseline. When the training was introduced, despite decreases in the average

percentage (mean = 31%, range = 20-37%), a slight increasing trend was observed for this

variable. After slight increases in the second baseline, immediate and sustained decreases

with notable fluctuations in OFF-MA were observed during the second intervention phase.

Later, steadier improvements occurred in the 2-week follow-up. Student B initially showed a

33
Neurocognitive training for improving academic engagement

low rate of OFF-MA (mean = 6%, range = 1-12%) and maintained low levels throughout the

phases. After minor increases in the first intervention phase, decreasing trends were observed

throughout the second intervention phase (mean = 1%, range = 0-5%) and 2-week follow-up

(mean = 1%, range = 0-3%).

Rates of OFF-PB also differed between the students. During the first baseline, Student

A showed lower rates with larger fluctuations (mean = 28%, range = 12-67%), while the rates

for Student B were higher and more stable (mean = 44%, range = 40-51%). After the training

was implemented, sustained improvements were found in both students. Considerably lower

averaged rates were found for Student A (mean = 3%, range = 0-7%) than Student B (mean =

24%, range = 16-29%) in the first intervention phase. During the second baseline, increasing

trends were demonstrated for both students. After the training was re-introduced, a sustained

decreasing trend was found in Student B (mean = 27%, range = 12-43%), whereas an

increasing trend was observed in Student A (mean = 15%, range = 3-37%). These trends

maintained in the 2-week follow-up. In particular, the average rate of Student A’s OFF-PB in

the follow-up phase (mean = 31%, range = 23-39%) was higher than in the first baseline.

34
Neurocognitive training for improving academic engagement

Medium effect sizes were obtained for OFF-MA within the reversal design for

Student A (Phi = 0.40) and Student B (Phi = 0.44), as well as for the reversal design to

follow-up for Student A (Phi = 0.43) and Student B (Phi = 0.47). For OFF-PB, medium effect

sizes were obtained for Student A within the reversal design (Phi = 0.30) and from the

reversal design to the follow-up (Phi = 0.34). Larger effect sizes were obtained for Student B

within the reversal design (Phi = 0.78) and from the reversal design to the follow-up (Phi =

0.79).

In summary, these results suggest that the neurocognitive training had immediate and

negative effects on OFF-MA and OFF-PB in the two settings for both students. This finding

has been confirmed by effect size coefficients. The decreasing trends from the reversal design

to the follow-up were more likely to sustain in the self-study setting than the small class.

Intervention Fidelity

The computer-delivered training program allowed for a robust protocol. In order to

complete a session, the students were required to complete assigned tasks one by one. This

procedure resulted in warranted treatment fidelity and reliability. Furthermore, increased task

difficulty levels in the last five compared to the first five training sessions (see Table 2)

35
Neurocognitive training for improving academic engagement

indicated that the students showed good compliance; i.e. they put in effort and improved at

the training games

Discussion

Previous research has reported positive effects of neurocognitive training in AD/HD.

However, previous studies (e.g., Johnstone et al., 2017) that reported improved behavior

mostly relied on indirect source of evidence (e.g., parent or teacher report). The purpose of

this study was to examine effects of neurocognitive training on specific academic behaviors

in the self-study setting and small class setting. Through the use of withdrawn design,

specific academic behaviors were directly observed and reported across phases in two

children with AD/HD. Previous studies (Hart et al., 2011; Imeraj et al., 2013) indicate that

children with AD/HD tend to perform inattentive or disruptive behaviors in contexts that

require high self-regulation or lack adult supervision. Thus, we set up two contexts in the

present study by selecting tasks with academic contents and requiring the students to

complete tasks independently and silently. Informed by previous studies (Hart et al., 2011;

Imeraj et al., 2013), it was assumed that the students in the present study would need to

engage executive functions and/or self-regulative abilities to sustain attention and remain on-

36
Neurocognitive training for improving academic engagement

task. It is worth noting that the small class setting was more challenging to the students as

they had to deal with potential interferences (e.g., inappropriate communication) from other

students.

Overall, improvements in academic engagement and reductions in off-task behaviors

were demonstrated during the training compared to baseline phases. In particular, both

students demonstrated immediate and positive growths in ONT-EX and AET in the two

settings. More sustained and post-training improvements were observed in the setting of self-

study room. Despite of these outcomes, visual inspection of the trend in the diagram and

calculation of effect size coefficient suggested that the training had limited or no

effectiveness on ONT-SBM. The present study contributes to research supporting

neurocognitive training as an effective treatment for children with AD/HD to improve their

academic engagement in day-to-day school activities.

Substantial improvements in awareness and control of psychological states factors

(Focus, Relax, and Zen) occurred during the first training phase, with reduced achievement

(after 2 weeks of no training) followed by sustained improvements during the 2nd training

phase. “Early response” refers to trajectories of rapid symptom changes within the first half

37
Neurocognitive training for improving academic engagement

of the treatment (Linardon, Brennan, & de la Piedad Garcia, 2016). Early response is a

favorable pattern in treatment as it has been reported to predict better outcomes at treatment

termination and short-/long-term follow-ups (Barb, Siegle, Young, & Huppert, 2018;

Kleinstäuber, Lambert, & Hiller, 2017). Ideally, our observed early response trajectory for

the psychological states predicts (or generalizes) change of expected behavior in a similar

way. Linking this pattern to changes in academic engagement behaviors lead to two

interesting points for consideration.

Firstly, contextual differences may be an influential factor in generalization of

training potency to expected behavior. Changes in ONT-EX and AET in the self-study room

setting met the criteria for early change, and showed an increasing trend across training

sessions. In contrast, changes in these variables in the small class setting did not show an

initial increasing trend nor maintain improvement in later sessions. As for OFF-MA and

OFF-PB, an ideal pattern might be ‘early reduction with sustained decreases’, which was in

line with the pattern of psychological states. This expected pattern was observed in the self-

study room (except for Student B’s OFF-MA) but not the small class setting. Thus is seems

that simple contexts (e.g., self-contained, with minimal distractors) are likely to facilitate the

38
Neurocognitive training for improving academic engagement

generalization from state control/regulation improvements to behavioral performance. In

contrast, complicated contexts, such as small/whole classrooms, may contain factors that

hinders this generalization. This finding is somewhat contrary to previous studies which have

suggested that on-task behavior is more likely to occur in a small group setting than during

independent silent work (Hart et al., 2011). In addition, the finding does not support that of

Lauth et al. (2006) who found a higher level of off-task behavior during instructions with

minimal teacher supervision. Indeed, convergence of EEG/state scores and observational

outcomes implies that behavioral generalization from the training is more likely occur in

simpler contexts. It is worthwhile for future studies to investigate the mechanism of

generalizing effects of neurocognitive training to learning-related behavior.

The second consideration is about development of on-task behavior with spontaneous

body movement (i.e., ONT-SBM), which did not meet the pattern of early response and was

not congruent with the expected on-task behavior (i.e., ONT-EX). While a decreasing trend

was found in the self-study room setting, an increasing trend in this behavior was observed in

the small class setting. Beside of the potential influence of context difference, the trait of

spontaneous body movement itself might play a mediating role during transfer from off-task

39
Neurocognitive training for improving academic engagement

behavior to on-task behavior. In the situation where on-task behavior was easy to implement,

the training might facilitate changing on-task behavior with spontaneous activities to the

desired on-task behavior. This possibly explained sustained decreases in this variable in the

setting of self-study room. In another situation where on-task behavior was more difficult to

exert (e.g., in the setting of small class due to external distractors), the training might

facilitate changing off-task behaviors to on-task behavior with spontaneous activities. This

possibly may explain the sustained increase of this behavior in the setting of small class.

Previous studies have reported that the rate of spontaneous activities (e.g., body

movement, mind wandering) during sustained attention tasks are associated with inattentive

symptoms (Frid, Lavner, & Rabinowitz, 2012; Seli, Smallwood, Cheyne, & Smilek, 2015).

This does not seem to be the case in the small class setting in our study. When off-task

behaviors declined, increasing trends in ONT-SBM were observed. This finding, thus, calls

for investigation of the role that spontaneous activities play in shifting from off-task to on-

task. To our knowledge, there is little research on this topic.

Despite the common improvement in academic behaviors, different improvement

patterns in these variables were found between participants. While Student A demonstrated

40
Neurocognitive training for improving academic engagement

improvement with large effect sizes in ONT-EX, OFF-MA, and OFF-PB in the setting of

self-study room, he only showed improvements with medium effect sizes for these variables

in the small class setting. In contrast, Student B made similar improvements in both settings.

Even though the same training protocol was implemented, non-treatment variables might be

account for such discrepancies. Evaluative studies of behavioral treatments (e.g., cognitive-

behavioral therapy, parent training, residential treatment) have reported that various

pretreatment factors (e.g., contextual variables, comorbidity, readiness to change) influence

outcomes for children with AD/HD (Beauchaine et al., 2015; den Dunnen, St. Pierre, Stewart,

Johnson, Cook, & Leschied, 2012; Jarrett, 2013). Neurocognitive training has revealed its

own arena of promising effectiveness, which is different from behavioral treatment nor

stimulus medication (for a review, see Chacko et al., 2014). Thus, one direction for future

research may be to explore pretreatment variables that enhance training potency. Coherently,

training protocols need to be tailored to incorporate variable(s) for catering for children with

a diversity of backgrounds.

While previous studies have shown some preferable outcomes after neurocognitive

training, these studies often overlooked contextual factors. Here we examined the efficacy of

41
Neurocognitive training for improving academic engagement

neurocognitive training in differentiated instructional contexts. Taking effect size coefficient

phi of each variable into account, more positive outcomes were found in the setting of self-

study room than small class. This finding, although preliminary, suggests that future research

may assess the extent to which learning context may affect training effectiveness. Implying

from the results of the present study, it is likely that children are more capable of maintaining

the training effects in contexts that are free from external distractions. A proposed solution of

enhancing potency of neurocognitive training in varied learning contexts may incorporate

behavioral strategies that are responsive to environmental variables (Chacko et al., 2014).

However, substantial research need to be conducted to prove validity of this proposition.

In the present study the neurocognitive training was conducted with teacher feedback

– this has not been the case in previous studies. Teacher feedback has been found to have

powerful impacts on student performance. Hattie (2009) reviewed 23 meta-analyses and

reported an overall effect size d = 0.73 for teacher feedback on student performance, with the

most effective form of feedback being the provision of cues or reinforcement. Further,

providing positive and specific feedback about student’s performance is a recommended

instructional strategy for children with AD/HD (DuPaul, Weyandt, & Janusis, 2011; Fowler,

42
Neurocognitive training for improving academic engagement

2010). Teacher feedback in conjunction with other techniques/interventions is effective in

decreasing inappropriate classroom behavior (Price, Martella, Marchand-Martella, &

Cleanthous, 2002) and increasing on-task behavior (Jurbergs, Palcic, & Kelley, 2010) for

children with AD/HD in previous studies. Thus, the third direction for exploration is

assessing the relative contribution of adult feedback on the effects of neurocognitive training.

It would be interesting to examine whether or how the training outcomes may be enhanced by

collaborating with behavioral/instructional strategies such as teacher feedback.

Limitations

There are several limitations that must be considered when interpreting the results.

Firstly, this study did not involve children with the predominantly hyperactive-impulsive

presentation (AD/HD-HI). Additionally, it did not involve participants with comorbid

disorders. Although other disorders (e.g., oppositional defiant disorder, anxiety disorder) may

co-occur with AD/HD (American Psychiatric Association, 2013), manifestation of

comorbidity disorders may be more complicated and is beyond research interest of the

present study. Therefore, findings of this study may have a limited implication for

educational interventions for the subtype of AD/HD-HI or AD/HD with comorbidities.

43
Neurocognitive training for improving academic engagement

Second, the participants were randomly drawn from a pool that consisted of students

with AD/HD with learning difficulties in mainstream classrooms. In this situation, some

students in the pool might have low aptitude in learning, for example, Student A in the

present study (relatively low IQ). Moreover, possible co-morbid diagnoses in categories such

as intellectual disability or learning disability were not considered. It is worthwhile to collect

more specific case studies to report the progress and effectiveness of neurocognitive training

on this subgroup.

Third, although gender difference was not a research interest of the present study, no

female participants were involved. Considering potential differences of the training outcomes

between female and male children, findings form this study may have a limited implication

for girls with AD/HD. It is worthwhile for future studies to include female participants.

Fourth, the training was conducted after school (between 3 to 5pm). Although the

participants were given a break and food before each session, they might still feel tired after a

whole day of school. This issue might affect the training outcomes. Nevertheless, this

situation represented somewhat the reality of conducting the training in school context.

44
Neurocognitive training for improving academic engagement

School administrators may consider achieving academic teaching plans as a priority over

providing extra-curricular training.

Fifth, although the present study was designed to examine a two-week follow-up

phase, to assess the maintenance of training outcomes, the findings may have a limited

implication for sustained real-world improvement. Future studies may adopt a longer follow-

up phase to determine the sustained improvements. Fourth, this study was a preliminary study

reporting effects of neurocognitive training on academic engagement in real-life settings.

Given that a small number of participants was used, it is worthwhile for future studies to

adopt designs with large sample sizes (e.g., blinded treatment-control design; Lofthouse et

al., 2012)

Conclusion

While an emerging body of research has assessed behavioral outcomes of

neurocognitive training in children with AD/HD, most of those studies relied on adults’

report of children’s behavior or clinically-based data (Chacko et al., 2014; Rapport et al.,

2013). This multiple case-study extended the literature by evaluating the efficacy of

neurocognitive training on academic engagement in real-life educational contexts through

45
Neurocognitive training for improving academic engagement

using direct observation. These findings support using this treatment for improving academic

engagement in children with AD/HD. Whether or not contextual differences, non-treatment

variables (e.g., children’s motivation), or adult’s feedback, may affect treatment efficacy

warrants future research.

46
Neurocognitive training for improving academic engagement

References

American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental

Disorders: Dsm-5. Arlington, VA: American Psychiatric Association.
Barb, S., Siegle, G., Young, K., & Huppert, T. (2018). Augmenting Cbt with Real-Time Fmri
Amygdala Neurofeedback Training Increases Early Response to Therapy.
BIOLOGICAL PSYCHIATRY, 83(9), S188-S188.
Barry, R. J., Clarke, A. R., & Johnstone, S. J. (2003). A Review of Electrophysiology in
Attention-Deficit/Hyperactivity Disorder: I. Qualitative and Quantitative
Electroencephalography. Clinical Neurophysiology, 114, 171-183. doi:10.1016/S1388-
2457(02)00362-0
Beauchaine, T. P., Neuhaus, E., Gatzke-Kopp, L. M., Reid, M. J., Chipman, J., Olliges, A., . . .
Webster-Stratton, C. (2015). Electrodermal Responding Predicts Responses to, and
May Be Altered by, Preschool Intervention for Adhd. Journal of Consulting & Clinical
Psychology, 83(2), 293-303. doi:10.1037/a0038405
Chacko, A., Kofler, M., & Jarrett, M. (2014). Improving Outcomes for Youth with Adhd: A
Conceptual Framework for Combined Neurocognitive and Skill-Based Treatment
Approaches. Clinical Child & Family Psychology Review, 17(4), 368-384.
doi:10.1007/s10567-014-0171-5
Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112(1), 155-159. doi:10.1037/0033-
2909.112.1.155
Cortese, S., Ferrin, M., Brandeis, D., Holtmann, M., Aggensteiner, P., Daley, D., . . . Sonuga-
Barke, E. J. S. (2016). Neurofeedback for Attention-Deficit/Hyperactivity Disorder:
Meta-Analysis of Clinical and Neuropsychological Outcomes from Randomized
Controlled Trials. Journal of the American Academy of Child & Adolescent Psychiatry,
55(6), 444-455. doi:10.1016/j.jaac.2016.03.007
den Dunnen, W., St. Pierre, J., Stewart, S. L., Johnson, A., Cook, S., & Leschied, A. W. (2012).
Predicting Residential Treatment Outcomes for Emotionally and Behaviorally
Disordered Youth: The Role of Pretreatment Factors. Residential Treatment for
Children & Youth, 29(1), 13-31.
DuPaul, G. J., & Joshua, M. L. (2015). Educational Impairements in Children with Adhd. In R.
A. Barkley (Ed.), Attention-Deficit Hyperactivity Disorder: A Handbook for Diagnosis
47
Neurocognitive training for improving academic engagement

and Treatment (4th ed., pp. 311-349). New York: The Guilford Press.
DuPaul, G. J., Weyandt, L. L., & Janusis, G. M. (2011). Adhd in the Classroom: Effective
Intervention Strategies. Theory Into Practice, 50(1), 35-42.
Egeland, J., Johansen, S. N., & Ueland, T. (2009). Differentiating between Adhd Sub-Types on
Ccpt Measures of Sustained Attention and Vigilance. Scandinavian Journal of
Psychology, 50(4), 347-354. doi:10.1111/j.1467-9450.2009.00717.x
Fowler, M. (2010). Increasing on-Task Performance for Students with Adhd. Education Digest:
Essential Readings Condensed for Quick Review, 76(2), 44-50.
Frid, A., Lavner, Y., & Rabinowitz, I. (2012). Analysis of Finger Tapping Parameters in People
with Adhd. Paper presented at the IEEE 27th Convention of Electrical and Electronics
Engineers, Israel. Conference retrieved from
http://ezproxy.uow.edu.au/login?url=https://search.ebscohost.com/login.aspx?direct=t
rue&db=edseee&AN=edseee.6377064&site=eds-live
Hart, K. C., Massetti, G. M., Fabiano, G. A., Pariseau, M. E., & Pelham, W. E., Jr. (2011).
Impact of Group Size on Classroom on-Task Behavior and Work Productivity in
Children with Adhd. Journal of Emotional and Behavioral Disorders, 19(1), 55-64.
Hattie, J. (2009). Visible Learning : A Synthesis of over 800 Meta-Analyses Relating to
Achievement London : Routledge.
Hodgson, K., Hutchinson, A. D., & Denson, L. (2014). Nonpharmacological Treatments for
Adhd: A Meta-Analytic Review. Journal of attention disorders, 18(4), 275-282.
doi:10.1177/1087054712444732
Imeraj, L., Antrop, I., Sonuga-Barke, E., Deboutte, D., Deschepper, E., Bal, S., & Roeyers, H.
(2013). The Impact of Instructional Context on Classroom on-Task Behavior: A
Matched Comparison of Children with Adhd and Non-Adhd Classmates. Journal of
School Psychology, 51, 487-498. doi:10.1016/j.jsp.2013.05.004
Jarrett, M. A. (2013). Treatment of Comorbid Attention-Deficit/Hyperactivity Disorder and
Anxiety in Children: Processes of Change. Psychological Assessment, 25(2), 545-5558.
doi:10.1037/a0031749
Jiang, H., & Johnstone, S. J. (2015). A Preliminary Multiple Case Report of Neurocognitive
Training for Children with Ad/Hd in China. SAGE Open, 5(2).
doi:10.1177/2158244015586811
Johnstone, S. J. (2013). Computer Gaming and Adhd: Potential Positive Influences on Behavior.
IEEE Technology & Society Magazine, 32(1), 20-22. doi:10.1109/mts.2013.2249355
48
Neurocognitive training for improving academic engagement

Johnstone, S. J., Blackman, R., & Bruggemann, J. M. (2012). Eeg from a Single-Channel Dry-
Sensor Recording Device. Clinical EEG and Neuroscience, 43(2), 112-120.
doi:10.1177/1550059411435857
Johnstone, S. J., Roodenrys, S., Blackman, R., Johnston, E., Loveday, K., Mantz, S., & Barratt,
M. F. (2012). Neurocognitive Training for Children with and without Ad/Hd. Attention
Deficit and Hyperactivity Disorders, 4(1), 11-23. doi:10.1007/s12402-011-0069-8
Johnstone, S. J., Roodenrys, S., Phillips, E. A., Watt, A. J., & Mantz, S. (2010). A Pilot Study
of Combined Working Memory and Inhibition Training for Children with Ad/Hd.
Attention Deficit and Hyperactivity Disorders, 2(1), 31-42. doi:10.1007/s12402-009-
0017-z
Johnstone, S. J., Roodenrys, S. J., Johnson, K., Bonfield, R., & Bennett, S. J. (2017). Game-
Based Combined Cognitive and Neurofeedback Training Using Focus Pocus Reduces
Symptom Severity in Children with Diagnosed Ad/Hd and Subclinical Ad/Hd.
International Journal of Psychophysiology, 116, 32-44.
doi:10.1016/j.ijpsycho.2017.02.015
Jurbergs, N., Palcic, J. L., & Kelley, M. L. (2010). Daily Behavior Report Cards with and
without Home-Based Consequences: Improving Classroom Behavior in Low Income,
African American Children with Adhd. Child & Family Behavior Therapy, 32(3), 177-
195.
Kleinstäuber, M., Lambert, M. J., & Hiller, W. (2017). Early Response in Cognitive-Behavior
Therapy for Syndromes of Medically Unexplained Symptoms. BMC Psychiatry, 17(1),
195-195. doi:10.1186/s12888-017-1351-x
Kofler, M. J., Rapport, M. D., & Alderson, R. M. (2008). Quantifying Adhd Classroom
Inattentiveness, Its Moderators, and Variability: A Meta-Analytic Review. Journal Of
Child Psychology And Psychiatry, And Allied Disciplines, 49(1), 59-69.
doi:10.1111/j.1469-7610.2007.01809.x
Langberg, J. M., Molina, B. S. G., Arnold, L. E., Epstein, J. N., Altaye, M., Hinshaw, S. P., . . .
Hechtman, L. (2011). Patterns and Predictors of Adolescent Academic Achievement
and Performance in a Sample of Children with Attention-Deficit/Hyperactivity
Disorder. Journal of Clinical Child and Adolescent Psychology, 40(4), 519-531.
Lauth, G. W., Heubeck, B. G., & Mackowiak, K. (2006). Observation of Children with
Attention-Deficit Hyperactivity (Adhd) Problems in Three Natural Classroom Contexts.
British Journal of Educational Psychology, 76(2), 385-404.
49
Neurocognitive training for improving academic engagement

doi:10.1348/000709905X43797
Linardon, J., Brennan, L., & de la Piedad Garcia, X. (2016). Rapid Response to Eating Disorder
Treatment: A Systematic Review and Meta-Analysis. International Journal of Eating
Disorders, 49(10), 905-919. doi:10.1002/eat.22595
Liu, C.-Y., Huang, W.-L., Kao, W.-C., & Gau, S. S.-F. (2017). Influence of Disruptive Behavior
Disorders on Academic Performance and School Functions of Youths with Attention-
Deficit/Hyperactivity Disorder. Child Psychiatry and Human Development.
doi:10.1007/s10578-017-0710-7
Lofthouse, N., Arnold, L. E., Hersch, S., Hurt, E., & DeBeus, R. (2012). A Review of
Neurofeedback Treatment for Pediatric Adhd. Journal of Attention Disorders, 16(5),
351-372. doi:10.1177/1087054711427530
Mayer, K., Wyckoff, S. N., & Strehl, U. (2013). One Size Fits All? Slow Cortical Potentials
Neurofeedback: A Review. Journal of attention disorders, 17(5), 393-409.
doi:10.1177/1087054712468053
Parker, C., Whear, R., Ukoumunne, O. C., Bethel, A., Thompson-Coon, J., Stein, K., & Ford,
T. (2015). School Exclusion in Children with Psychiatric Disorder or Impairing
Psychopathology: A Systematic Review. Emotional & Behavioural Difficulties, 20(3),
229-251. doi:10.1080/13632752.2014.945741
Parker, R., & Hagan-Burke, S. (2007). Useful Effect Size Interpretations for Single Case
Research. Behavior Therapy, 38(1), 95-105. doi:10.1016/j.beth.2006.05.002
Pingault, J.-B., Côté, S. M., Vitaro, F., Falissard, B., Genolini, C., & Tremblay, R. E. (2014).
The Developmental Course of Childhood Inattention Symptoms Uniquely Predicts
Educational Attainment: A 16-Year Longitudinal Study. Psychiatry Research, 219(3),
707-709. doi:https://doi.org/10.1016/j.psychres.2014.06.022
Price, A. T., Martella, R. C., Marchand-Martella, N. E., & Cleanthous, C. C. (2002). A
Comparison of Immediate Feedback Delivered Via an Fm Headset Versus Delayed
Feedback on the Inappropriate Verbalizations of a Student with Adhd. Education and
Treatment of Children, 25(2), 159-171.
Rapport, M. D., Kofler, M. J., Alderson, R. M., Timko, T. M., Jr., & DuPaul, G. J. (2009).
Variability of Attention Processes in Adhd: Observations from the Classroom. Journal
of Attention Disorders, 12(6), 563-573.
Rapport, M. D., Orban, S. A., Kofler, M. J., & Friedman, L. M. (2013). Do Programs Designed
to Train Working Memory, Other Executive Functions, and Attention Benefit Children
50
Neurocognitive training for improving academic engagement

with Adhd? A Meta-Analytic Review of Cognitive, Academic, and Behavioral

Outcomes. Clinical Psychology Review, 33(8), 1237-1252.
doi:https://doi.org/10.1016/j.cpr.2013.08.005
Rapport, M. D., Orban, S. A., Kofler, M. J., Friedman, L. M., & Bolden, J. (2015). Executive
Function Training for Children with Adhd. In R. A. Barkley (Ed.), Attention-Deficit
Hyperactivity Disorder: A Handbook for Diagnosis and Treatment (4th ed., pp. 1131-
1171). New York: The Guilford Press.
Rapport, M. D., Scanlan, S. W., & Denney, C. B. (1999). Attention-Deficit/Hyperactivity
Disorder and Scholastic Achievement: A Model of Dual Developmental Pathways.
Journal Of Child Psychology And Psychiatry, And Allied Disciplines, 40(8), 1169-1183.
Rodriguez, A., Järvelin, M.-R., Obel, C., Taanila, A., Miettunen, J., Moilanen, I., . . . Olsen, J.
(2007). Do Inattention and Hyperactivity Symptoms Equal Scholastic Impairment?
Evidence from Three European Cohorts. BMC Public Health, 7, 327-327.
Rogers, J. M., Aminov, A., Wilson, P. H., & Johnstone, S. J. (2016). Retest Reliability of a
Single-Channel, Wireless Eeg Device. International Journal of Psychophysiology, 106,
87-96.
Rogers, M., & Tannock, R. (2013). Are Classrooms Meeting the Basic Psychological Needs of
Children with Adhd Symptoms? A Self-Determination Theory Perspective. Journal Of
Attention Disorders.
Rothbart, M. K., & Posner, M. I. (2015). The Developing Brain in a Multitasking World.
Developmental Review, 35(Supplement C), 42-63.
doi:https://doi.org/10.1016/j.dr.2014.12.006
Sarver, D. E., Rapport, M. D., Kofler, M. J., Raiker, J. S., & Friedman, L. M. (2015).
Hyperactivity in Attention-Deficit/Hyperactivity Disorder (Adhd): Impairing Deficit or
Compensatory Behavior? Journal Of Abnormal Child Psychology, 43(7), 1219-1232.
doi:10.1007/s10802-015-0011-1
Schneider, N., Goldstein, H., & Parker, R. (2008). Social Skills Interventions for Children with
Autism: A Meta-Analytic Application of Percentage of All Non-Overlapping Data
(Pand). Evidence-Based Communication Assessment & Intervention, 2(3), 152-162.
doi:10.1080/17489530802505396
Seli, P., Smallwood, J., Cheyne, J. A., & Smilek, D. (2015). On the Relation of Mind Wandering
and Adhd Symptomatology. Psychonomic Bulletin & Review, 22(3), 629-636.
doi:10.3758/s13423-014-0793-0
51
Neurocognitive training for improving academic engagement

Sergeant, J. A. (2005a). The Dynamic Developmental Theory of Adhd: Reflections from a

Cognitive Energetic Model Standpoint. Behavioral and Brain Sciences, 28(3), 442-443.
doi:10.1017/s0140525x05380071
Sergeant, J. A. (2005b). Modeling Attention-Deficit/Hyperactivity Disorder: A Critical
Appraisal of the Cognitive-Energetic Model. Biological Psychiatry, 57(11), 1248-1255.
doi:10.1016/j.biopsych.2004.09.010
Steiner, N. J., Frenette, E. C., Rene, K. M., Brennan, R. T., & Perrin, E. C. (2014). In-School
Neurofeedback Training for Adhd: Sustained Improvements from a Randomized
Control Trial. Pediatrics, 133(3), 483-492. doi:10.1542/peds.2013-2059
Steiner, N. J., Sheldrick, R. C., Gotthelf, D., & Perrin, E. C. (2011). Computer-Based Attention
Training in the Schools for Children with Attention Deficit/Hyperactivity Disorder: A
Preliminary Trial. Clinical Pediatrics, 50(7), 615-622.
doi:10.1177/0009922810397887
Swanson, J. M., Kraemer, H. C., Hinshaw, S. P., Arnold, L. E., Conners, C. K., Abikoff, H.
B., . . . Wu, M. (2001). Clinical Relevance of the Primary Findings of the Mta: Success
Rates Based on Severity of Adhd and Odd Symptoms at the End of Treatment. Journal
Of The American Academy Of Child And Adolescent Psychiatry, 40(2), 168-179.
Vile Junod, R. E., DuPaul, G. J., Jitendra, A. K., Volpe, R. J., & Cleary, K. S. (2006). Classroom
Observations of Students with and without Adhd: Differences across Types of
Engagement. Journal of School Psychology, 44, 87-104. doi:10.1016/j.jsp.2005.12.004
Wechsler, D. (2003). Wechsler Intelligence Scale for Children-Fourth Edition. San Antonio,
TX: Psychological Corporation.

52