Integrating Innovative Neuro-educational Technologies

(I-Net) into K-12 Science Classrooms

Ronald H. Stevens1, Trysha Galloway1, and Chris Berka2

1
UCLA IMMEX Project, 5601 W. Slauson Ave. #255, Culver City, CA 90230
[email protected],
[email protected]
2
Advanced Brain Monitoring, Inc, Carlsbad, CA 92008
[email protected]

Abstract. With the U.S. facing a decline in science, math and engineering skills,
there is a need for educators in these fields to team with engineers and cognitive
scientists to pioneer novel approaches to science education. There is a strong
need for the incorporation problem solving and emerging neuroscience
technologies into mainstream classrooms, and for students and teachers to
experience what it means at a very personal level, to engage in and struggle with
solving difficult science problems. An innovating and engaging way of doing
this is by making the problem solving process visible through the use of real-
time electroencephalography cognitive metrics. There are educational, task, and
measurement challenges that must be addressed to accomplish this goal. In this
paper we detail some of these challenges, and possible solutions, to develop a
framework for a new set of Interactive Neuro-Educational Technologies (I-Net).

Keywords: EEG, Problem solving, Skill Acquisition, Cognitive Workload.

1 Introduction
Science educators are increasingly being pressured to make their efforts more
effective, efficient and relevant to the needs of today’s workforce. Promoting "…an
individual's capacities to use cognitive processes to confront and resolve real, cross-
disciplinary situations where the solution path is not immediately obvious" is a
worldwide educational priority (National Research Council, 2005, OECD, 2004). As
tasks and problems become more complex, students face greater demands to hold and
manipulate many forms of data in working memory. The burden placed on working
memory (i.e., cognitive load) in these situations can have a significant impact on
students’ abilities to perform learning tasks and to benefit from them, because
working memory capacity is limited (Baddeley, 2003, Sweller, 1989, 1994). Given
the increasing consolidation of job functions within the workforce, the reality is that
students will be assuming more responsibility for learning and decision making in the
future, and will be increasingly confronted by data / information overload.
While students are individually struggling with increased mental demands, it is
becoming more difficult for teachers to support their learning in meaningful cognitive

D.D. Schmorrow, L.M. Reeves (Eds.): Augmented Cognition, HCII 2007, LNAI 4565, pp. 47–56, 2007.
© Springer-Verlag Berlin Heidelberg 2007
48 R.H. Stevens, T. Galloway, and C. Berka

ways. With limited training resources and staff size, rapid identification of students
who are / are not developing problem solving skills has remained elusive (National
Research Council, 2004). Part of the challenge is cognitive. Strategic problem solving
is complex with skill level development being influenced by the balance of the task
and the experience and knowledge of the student. Another challenge is observational.
Assessment of problem solving requires real-world tasks that are not immediately
resolvable and require movement among structural, visual and auditory
representations. Lastly, there are challenges of how rapidly valid inferences of the
cognitive state of a student can be made and reported from the performance data.
Application of neurophysiologic solutions, especially electroencephalography
(EEG) to education offers cutting-edge approaches to these educational challenges.
First, with a temporal resolution of seconds or milliseconds the goal of real-time
feedback could optimize engagement and cognitive workload during learning. As
EEG data collection occurs without interfering with normal task-driven cognition, it
can provide an assessment of cognitive load that is not confounded by strong
measurement effects. Further on the horizon would be the use of brain-computer
interfaces to drive learning in ways that remain in the domain of science fiction.
With today’s technologies, classroom-wide application of EEG during educational
tasks could individualize the learning experience in specific ways. For example:
Existing monitoring devices could detect situations where the student is simply not
ready to learn. This could arise through sleep deprivation, a common occurrence
among pre-teens and teens (Meijer et al, 2000), or excessive stress / poor stress
management. Under these situations classroom instruction may fail, and errors in the
learning process will increase
There may also be situations where the task exceeds the individual’s immediate
ability to mentally model the problem due to the difficulty of the task with respect to
content / language and the student cannot be adequately engaged in the learning
process although they are motivated to do so. This would be evidenced either by
disengagement or ‘thrashing’, (acquiring excessive data while not engaged or while
not processing it), conditions that could lead to poor decision making. Finally there
are conditions where distractions and constraints in the environment combine to make
a task that a student would normally easily learn, difficult. Such constraints include
time pressure, noise, distraction and/or peer pressure. Were these situations quickly
recognized and corrected then learning could become accelerated and more
personalized.
Throughout education, there is a wide gap in translating the findings of cognitive
neuroscientists into everyday educational practice (Goswami, 2006). The challenges
are multi-factorial involving issues of diffusion of innovations, disconnects between
scientists and educators, and issues of technology and turf. In this paper we draw from
multiple perspectives of technology and education to explore possible pathways for
taking I-NET into the messy world of real tasks and classroom environments. We first
outline the major challenges along the way and close with a near vision of what these
approaches can contribute to improving the learning of students.
Our starting framework for situating I-NET tools in education includes the:
• Parameters for measures and devices
• Approaches for situating neurophysiologic measures in education,
 Integrating Innovative Neuro-educational Technologies 49

• Specifications for the design and delivery of meaningful tasks, and

guidelines regarding the capture of cognitive measurements and reporting of
findings.
In doing so, we draw from programmatic lessons of the initial DARPA Augmented
Cognition Program which emphasized the need for these technologies to “extract
meaningful cognitive measures for identified periods of interest”, as well as to
“situate themselves within environments where they demonstrate high visibility
effects that satisfy near as well as long-term needs” (“The End of the Beginning”,
Washington DC, 1/26/07).

2 EEG as a Neurophysiologic Method of Choice

For routine use in classroom settings I-NET instrumentation should be non-obtrusive
for the students and the classroom environment, easy for students to apply, applicable
to multiple task formats, inexpensive, and easy to collect meaningful data from. Real-
time EEG monitoring satisfies many of these requirements. Technically, EEG-based
cognitive load measurements offer the advantage of extremely high temporal
resolution. Electrophysiological data is collected at a grain-size of tenths of a second,
in contrast to the tens or hundreds of seconds required for traditional measures. This
opens the possibility for effective monitoring of workload fluctuations during very
rapid decision-making processes that are unobservable using traditional methods.
Although there is a large and growing literature on the EEG correlates of attention,
memory, and perception (Fabiani, 2001) there is a relative dearth of EEG
investigations of the process skill acquisition and learning in classrooms
(Smith,1999). Recording and analysis of EEG has traditionally been confined to
laboratory settings and training to criterion tasks due to the technical obstacles of
recording high quality data and the computational demands of real-time analysis.
Advances in electronics and data processing set the stage for ambulatory EEG
applications. A recently developed wireless EEG sensor headset facilitates easy
acquisition of high quality EEG combining battery-powered hardware with a sensor
placement system to provide a lightweight, easy-to-apply method to acquire and
analyze six channels of high-quality EEG. These headsets have been developed by
Advanced Brain Monitoring, Inc. and are useful for in-the-field studies where the
setup and acquisition of high quality recordings can be obtained by the user without
technical assistance. Such units capture EEG data and provide real-time reports of (a)
working cognitive load (EEG-WL), (b) distraction (EEG-DT) and (c) engagement
(EEG-E). Quantification of the EEG in real time, referred to as the B-Alert system, is
achieved using signal analysis techniques to identify and decontaminate eye blinks,
and identify and reject data points contaminated with electromyography, amplifier
saturation, and/or excursions due to movement artifacts (Berka et al., 2004). The EEG
sensor headset requires no scalp preparation and provides a comfortable and secure
sensor-scalp interface for 12 to 24 hours of continuous use.
Recent validation studies that included tests of forward and backward digit span, 3-
choice vigilance, standard image recognition, verbal paired associate recognition, etc
suggest that EEG-E reflects information-gathering, visual scanning and sustained
attention. EEG-WL increases with increasing working memory load and during
50 R.H. Stevens, T. Galloway, and C. Berka

problem-solving, integration of information, analytical reasoning and may be more

reflective of executive functions. Inspection of EEG on a second-by-second timescale
has revealed associations between workload and engagement levels when aligned
with specific task events providing preliminary evidence that second-by-second
classifications reflect parameters of task performance (Berka et al., 2004). These
associations have been demonstrated with adults in tasks such as driving and weapons
training simulation. We have extended this approach to investigate the role of
cognitive effort (workload, distraction, and engagement) in high school students’
math and science learning.

3 Challenges in the Educational System

An innovation has to be important to the teachers and administrators of a school
system if it is going to be used; if it is just based on pure research protocols it
probably won’t be (Rogers, 1995). A proposed educational innovation should align
with existing educational activities and practice; or be used to extend or expand the
existing curriculum. Our interactions with educators have approached these
challenges from both perspectives.
Maximizing the benefits of EEG technologies, involves developing an
understanding of what students and teachers are looking at. Developing a firm
grounding in the basic neuroscience and in the clinical neurology of EEG is needed.
One approach for expanding the curriculum is through the development of a Brain,
Mind and Learning Tool-kit for high school science teachers that features systems and
software for acquisition, analysis, and interpretation of the EEG. Introducing brain
monitoring into the classroom creates the opportunity for content-rich training in
neuroscience, psychology, and a hands-on experience for students and teacher to
observe their brain activity in real-time and understand how different states of
mind are reflected in the electric fields of the brain. Study of EEG can contribute
to a better understanding of the physics of energy, electricity, and waves, as well
as the chemistry of biosynthetic pathways and signaling cascades. These workshop
/ hands-on activities build awareness and understanding of the science and the goals
of the project including broad issues of investigation and experimentation.
An alternative method of providing value to the educational system is to
incorporate tools that complement existing curricular activities and extend them by
providing fast, accurate and more detailed information to support student learning.
Using second-by-second EEG and detailed descriptions of cognitive events during the
performance of complex tasks provides models for mediating the tasks, and/or the
stress on an individual when workload exceeds an optimal threshold. By studying the
dynamics of cognitive patterns within different classroom environments new insights
are gained about complex problem solving, collaborative learning tasks and hands-on
and minds-on science activities.
Example questions that could be approached include:
• What are the ranges and dynamics of cognitive workload and engagement
with students demonstrating different levels of problem solving proficiency?
Are these features stable, or do they change with experience, various
domains and the acquisition of skills?
 Integrating Innovative Neuro-educational Technologies 51

• What would be the likely classroom impacts if a teacher were able to

monitor a graphic display tracking the engagement, distraction and workload
of an entire class in real time?
• How do cognitive workload and engagement levels differ for students
performing a simulation vs. observing a simulation?

4 IMMEX™ Scientific Problem Solving Tasks

From this spectrum of educational possibilities we have aligned our research activities
with complex problem solving tasks that are being used extensively throughout school
districts across this country and other nations such as China. This problem solving
simulation system, termed IMMEX (Interactive Multi-Media Exercises) contains
dozens of standards-based science simulations suitable for middle and high school
classrooms. Over 600,000 student performances have been obtained on these
simulations leading to the construction of probabilistic models of performance and
skill acquisition (Stevens et al, 1999, Stevens & Casillas, 2006). IMMEX simulations
are also cognitively complex and most student activities while engaged in problem
solving can be mapped to specific elements of cognition (Chung et al, 2002). A final
advantage of IMMEX is that simulations exist in many science domains and across
grade levels; students will often perform over 100 simulations each year in domains
as diverse as math, chemistry, earth science and health. Aligning I-NET tools into
such an activity could provide high project visibility throughout the school district.
Deriving EEG measures during open-ended problem solving is not easy. The
development of problem solving skills is a gradual process and not all novices solve
problems in the same way, nor do they follow the same path toward understanding at
the same pace. Given the diversity of the student population it can be difficult to
assess what their relative levels of competence are when performing a task, making it
difficult to accurately relate EEG measures to other measures of task skill. In real-
world educational settings there is not anything approaching the control over the
relevant experimental parameters like others studying the sensory and motor systems.
There one knows exactly when light hits the retina; however, how does one determine
when a memory has occurred on a millisecond timescale?
Complex problem solving activities can critically depend upon the student’s
motivational and emotional states, creating additional complexity and variability
when compared with automatic behavioral responses, or even voluntary motor
responses. The memory traces of learning are not necessarily formed with any
reproducible temporal offset relative to the time the student demonstrates the memory
behaviorally, and this temporal ambiguity makes the job of ascribing meaning to a
particular firing correlate especially difficult as we are less sure of when to look, or
indeed what a neural correlate of problem solving might look like.
Our approach decomposes problem solving into behaviorally relevant segments
and is much like an exercise in situation awareness where data is continuously
acquired and consolidated to provide a ‘big picture’ understanding of the problem.
Components of such awareness include perception of the elements in the
environment, comprehension of the elements, and prediction of future interaction and
52 R.H. Stevens, T. Galloway, and C. Berka

events. The analogous components in problem solving are framing, navigation and
investigation, and closure.

Framing. The perception analog in IMMEX tasks is a framing event. Text or

multimedia descriptions are presented describing a situation, and posing one or more
challenges. For example, in the middle school problem set Phyto Phyasco the
prologue describes a newspaper clip indicating that a local farmer’s potato crop is
mysteriously dying; the challenge for the student becomes to determine if this is due
to pests, lack of nutrients, infection, etc.
Successful framing is critical for solving problems and experts often spend a
disproportionate amount of time on framing. Similarly, in situation awareness, the
majority of errors are based on errors in perception (Endsley, 1999). If the student
reads the prologue too quickly or does not take notes, or does not sufficiently
understand the terms then the framing process could fail. Occasionally students
become so involved in the details of the problem solving itself that they forget what
the question being asked is and need to reframe the problem. Currently it is very
difficult to know if the student has correctly framed the problem without extensive
verbal questions and answers. EEG can help illuminate the framing process by
determining if the student is engaged or overwhelmed by the task being proposed. We
have also observed that as the student gains problem solving experience on the tasks
the level of workload needed for effective framing begins to decrease (Stevens et al,
2007).

Navigation and Integration. Once the problem is framed the next important step is
the identification of the most relevant resources available to solve the problem. All
IMMEX tasks contain a hierarchy of resources that students navigate while gathering
information. Figure 1 shows the problem space for Phyto Phyasco which consists of 5
major categories, each containing multiple sub categories of information.
The top menus are mainly used for problem space navigation. The sub menu items
to the left are more decision making points as there is a cost or risk associated with
each. In the problem set Phyto Phyasco there are 38 choices. What results is a
dynamic iterative process of data collection and integration from which the solution to
the problem is derived.
This is a complex process as few students solve IMMEX problems using the same
sequence of actions, and currently it is not possible to determine from the framing,
navigation, and decision events alone whether the student will solve the problem.
Some students may have the relevant information in hand but do not understand the
significance of it in the context of the problem. Other students through either bias or
poor framing will never acquire the most useful information. Still others will access all
the available information in a non-discriminating way without a clear hypothesis.
Probabilistic descriptions of navigation and integration can provide a proxy for student
understanding, but the models often lack the precision and temporal updating needed
for a global metric of student’s situation awareness (Stevens & Casillas, 2006).
During efficient navigation and investigation, students often demonstrated a cycling
of the B-Alert cognitive indexes characterized by relatively high workload and low
engagement which then switched to lower workload and higher engagement. These
cycle switches were often, but not always associated with selection of new data items.
 Integrating Innovative Neuro-educational Technologies 53

Fig. 1. Sample IMMEX™ simulation. In the Phyto Phyasco simulation, potato plants are dying
and the student must identify the cause by examining weather conditions, nutrients, etc.
Students navigate throughout the problem space using the Main Menu and select data to make
decisions using the Sub Menu Items.

Multiple EEG correlates exist that are useful for understanding the navigation and
integration events. On complex tasks, humans seem to use prior information to
generate predictive codes for anticipated future events and continually compare
incoming data with these codes (Summerfield et al, 2006). Perception-related evoked
response potentials (ERP) similar to the auditory mismatch negativity (MMN)
(Naatanen, 2001), may be observed when data is displayed that is not consistent with
the current hypothesis. The absence of such signals could also be important if a
deviant source of data was not recognized when presented.
It is also likely that there will be differences in engagement and workload when
specific items of data are seen for the first time vs. subsequent times. The first time a
piece of data is examined the student must categorize both the representation of the
data, as well as associate the importance of the data to the context of the problem.
Once the representation of the data is understood then the subsequent data viewing
should only relate to the problem context and workload and engagement measures
would be expected to be shorter or lower. At the level of ERP, a decrease in the latency
of the P300 wave may be seen once the data representation is learned (Luck, 1998).
54 R.H. Stevens, T. Galloway, and C. Berka

Closure. Navigation and investigation is an open-ended process where the student

can continue until s/he is satisfied that a solution has been obtained and closes the
problem. Here the students first decide to solve the problem, an irreversible step in the
sense that once this decision is made then a solution must be entered. Then the
students select a solution from a list, and confirm the selection. There is no global
cognitive metric for knowing the likely problem solving outcome; however, having
such a metric would be important for learning as student’s success and approaches to
problem solving can persist for months after initial learning.
The steps leading to closure are highly dynamic and rich in uncertainty and errors
due to premature closure (errors perceived as errors), as well as errors due to data
limitations (errors about which there was uncertainty) can be observed which may
evoke error-related negativity ERP (Scheffers & Coles, 2000). On IMMEX tasks, the
more productive closure sequences are preceded by the high workload, rather than
high engagement by B-Alert measures.

5 The Near Future

Figure 2 portrays an interface to the I-NET workbench which will link student’s
neurophysiologic metrics, problem solving actions, and machine learning models of
performance and progress. It shows the screen of a teacher who is monitoring six of
her students for Engagement, Distraction and Workload using the B-Alert sensor
software. Each bar in the histograms represents averaged metrics at 1-second epochs.
Panels A, C and to a lesser extend F most closely represents students who are
productively engaged in problem solving; workload levels are moderate and the levels
are alternating with cycles of high engagement. Many of these cycles are associated
with navigation and interpretation events. Panel B illustrates a student who may be
experiencing difficulties and may not be prepared to learn. The workload and
engagement levels are low and distraction is consistently high. Under these conditions
the student should be asked to terminate the task as productive learning is not likely to
occur.
The student in Panel D has just encountered a segment of the simulation that
induced 10-15 seconds of distraction (middle row) and decreased workload and
engagement. By clicking on those EEG segments the teacher retrieved a view of the
data that the student was looking at, which in this case is an animation of a plant
growing. She could then make a note to discuss with the student to determine the
source of the distraction and / or to see if others in the class have struggled with the
same or similar issues.
Panel E shows a student who, while not distracted, appears to be working at
beyond optimal capacity with workload levels consistently near 100%. The teacher
then retrieved the artificial neural network performance assessment models for this
student (Stevens et al, 2004, Stevens & Casillas, 2006) which shows that this student
does not seem to be developing efficient strategies on his own. Utilizing this
information, the instructor may suggest the adjustment of the difficulty of the task for
this student and continue monitoring to see if more acceptable (in a learning sense)
levels of workload and engagement result.
 Integrating Innovative Neuro-educational Technologies 55

Fig. 2. Prototype I-NET interface for linking EEG measures of workload, distraction and
engagement with details of the tasks and probabilistic models of problem solving performance
and progress. While the interface itself is not yet functional, the tasks, EEG readings and
machine learning model reports are actual student performance data.

Acknowledgements. This project has been funded by several grants from the
National Science Foundation, (DUE- 01236050, REC-02131995, NSF-ROLE
0528840, HRD-0429156, DUE-0512526) whose support is gratefully acknowledged.

References
1. Baddeley, A.: Working Memory: Looking Back and Looking Forward. Nature Reviews |
Neuroscience 4, 829–839 (2003)
2. Berka, C., et al.: Real-time analysis of EEG indexes of alertness, cognition and memory
acquired with a wireless EEG headset. International Journal of Human-Computer
Interaction 17(2), 151–170 (2004)
3. Berka, C., Levendowski, D.J., Cvetinovic, M., Petrovic, M.M., Davis, G.F., Lumicao,
M.N., Popovic, M.V., Zivkovic, V.T., Olmstead, R.E.: Real-Time Analysis of EEG Indices
of Alertness, Cognition and Memory Acquired with a Wireless EEG Headset. International
Journal of Human-Computer Interaction 17(2), 151–170 (2004)
4. Chung, G.K.W.K, deVries, L.F., Cheak, A.M., Stevens, R.H., Bewley, W.L.: Cognitive
Process Validation of an Online Problem Solving Assessment. Computers and Human
Behavior 18, 669 (2002)
5. Endsley, M.R.: Situation awareness in aviation systems. In: Garland, D.J., Wise, J.A.
(eds.) Handbook of aviation human factors. Human factors in transportation, pp. 257–276.
Lawrence Erlbaum Associates, Mahwah (1999)
6. Fabiani, M., Gratton, G., Coles, M.G.: Event-related brain potentials. In: Caciooppo, J.T.,
Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, pp. 53–84.
Cambridge University Press, Cambridge, England (2000)
7. Goswami, U.: Neuroscience and education: from research to practice? Nature Reviews
Neuroscience 7, 2–7 (2006)
8. Luck, S.: Sources of dual-task interference: Evidence from human electro-
encephalography. Physiological Science 9, 223–227 (1998)
56 R.H. Stevens, T. Galloway, and C. Berka

9. Meijer, A.M., Habekothe, H.T., van Den Wittenboer, G.: Time in bed, quality of sleep and
school functioning of children. J. Sleep Res. 9, 145–153 (2000)
10. Naatanen, R.: The perception of speech sounds by the human brain as reflected by the
mismatch negativity (MMN) and its magnetic equivalent (MMNm). Psychophysiology 38,
1–21 (2001)
11. National Research Council, Knowing what students know: The science and design of
educational assessment. Committee on the Foundations of Assessment. In: Pelligrino,J.,
Chudowsky, N., Glaser, R. (eds.) Board on Testing and Assessment, Center for Education.
National Academy Press, Washington. DC (2001)
12. National Research Council, Rising Above the Gathering Storm: Energizing and Employing
America for a Brighter Economic Future (2005)
13. Organization for Economic Cooperation and Development, Problem Solving for
Tomorrow’s World, First Measures of Cross-curricular competencies from PIS 2004. p. 26
(2004)
14. Rogers, E.M.: Diffusion of Innovations, 4th edn. Free Press, New York (1995)
15. Scheffers, M.K., Coles, M.: Performance monitoring in a confusing world: Error related
brain activity, judgments of response accuracy and types of errors. Journal of Experimental
Psychology: Human Perception and Performance 26, 141–151 (2000)
16. Stevens, R., Casillas, A.: Artificial Neural Networks. In: Mislevy, R.E., Williamson, D.M.,
Bejar, I. (eds.) Automated Scoring, pp. 259–312. Lawrence Erlbaum, Mahwah (2006)
17. Stevens, R.H., Ikeda, J., Casillas, A., Palacio-Cayetano, J., Clyman, S.: Artificial neural
network-based performance assessments. Computers in Human Behavior 15, 295–314
(1999)
18. Summerfield, C., Egner, T., Greene, M., Koechlin, E., Mangels, J., Hirsch, J.: Predictive
codes for forthcoming perception in the frontal cortex. Science 314, 1311–1315 (2006)
19. Sweller, J.: Cognitive technology: Some procedures for facilitating learning and problem
solving in mathematics and science. Journal of Cognitive Psychology 81(4), 457–466
(1989)
20. Sweller, J.: Cognitive load theory, learning difficulty, and instructional design. Learning &
Instruction 4(4), 295–312 (1994)