Theoretically framing a complex phenomenon: student success in large enrollment active

learning courses

Robert M. Talbot III, Leanne Doughty, Amreen Nasim,1 Laurel Hartley,

Paul Le,2 Laird Kramer,3 Hagit Kornreich-Leshem,4 and Jeff Boyer5
1
School of Education and Human Development, University of Colorado Denver, Campus Box 106, Denver, CO 80217
2
Department of Integrative Biology, University of Colorado Denver, Campus Box 171, Denver, CO 80217
3
Department of Physics, Florida International University, 11200 SW 8 Street. Miami, FL 33199
4
STEM Transformation Institute, Florida International University, 11200 SW 8 Street. Miami, FL 33199
5
College of Science and Mathematics, North Dakota State University, Dept 2700 PO Box 6050, Fargo, ND 58108
Student success in large enrollment undergraduate science courses which utilize "active learning" and Learn-
ing Assistant (LA) support is a complex phenomenon. It is often ill-defined, is likely impacted by many factors,
and regularly interacts with a variety of treatments or interventions. Defining, measuring, and modeling student
success as a factor of multiple inputs is the focus of our work. Because this endeavor is complex and multi-
faceted, there is a need for strong theoretical framing. Without such explicit framing, we argue that our findings
would be uninterpretable. In this paper we describe our efforts to define that theoretical framework, present the
framework, and describe how it defines our methodological approach, analyses, and future work.

I. INTRODUCTION main research questions (Section II), describe the theoreti-

cal framework we are employing to frame our complex re-
search problem (Section III), and discuss the resulting data
There is strong evidence that the implementation of active
collection and analyses (Section IV) that are currently being
learning methods in undergraduate science courses can lead
conducted to answer our research questions.
to increased student conceptual understanding and course
achievement [1]. There is also evidence that Learning As-
sistants (LAs) [2], a practice embedded resource, can support
the use of active learning methods in the large lecture science
classroom [3]. Active learning is a term generally used to de- II. RESEARCH QUESTIONS
scribe interactive innovations in undergraduate science teach-
ing, similar to the notion of interactive engagement (IE) used Our research questions are: 1) What are the characteristics
in physics education research [4]. Wieman offers a definition of active learning tasks employed in undergraduate science
of active learning, in which "students are spending a signifi- courses?, 2) How can the design and implementation of ac-
cant fraction of the class time on activities that require them to tive learning methods be understood in terms of a theoretical
be actively processing and applying information in a variety framework?, 3) How are LAs involved in supporting active
of ways, such as answering questions using electronic click- learning?, and 4) How does engaging in these active learning
ers, completing worksheet exercises, and discussing and solv- methods with and without LA support contribute to student
ing problems with fellow students" [5]. But we also know that level outcomes? We are engaged in identifying and charac-
not all of these innovations in undergraduate science teach- terizing the use of active learning methods in LA and non-LA
ing practices (i.e., active learning or interactive engagement) supported large lecture science courses, and using a frame-
contribute equally to student level outcomes [6]. Though re- work to map those activities to the learning processes they
searchers have many ideas for why active learning helps stu- target. We then plan to model student level outcomes (e.g.,
dents learn, achieve, and persist in a course, we do not know learning gains, course grades, course retention, and two-year
what specific characteristics of active learning contribute the persistence) quantitatively as a function of variables at both
most to these outcomes, or the mechanisms by which these the course level (e.g., assigned instructional tasks) and stu-
activities work. Further, it is unclear how LAs support con- dent level (e.g., a student’s interaction with other students or
tribute to student success in these courses. It is time to look LAs), and interpret those models with respect to the theoret-
more deeply at the contributions of different observable char- ical framework. Finally, we will build qualitative arguments
acteristics of active learning methods to student success [5]. about the activities and outcomes observed in these courses.
In our work, we observe, characterize, and interpret the We will then be able to synthesize our quantitative model-
active learning methods employed in a large sample of LA ing and qualitative arguments in order to describe the char-
supported and non-LA supported science courses at the Uni- acteristics of LA and non-LA supported classroom activities
versity of Colorado Denver, North Dakota State University, that contribute most to student outcomes, and link those out-
and Florida International University. We define student suc- comes to specific learning processes. Although we have not
cess as concept inventory learning gains, course achievement, yet answered these questions, this paper is about the theoret-
retention, and persistence. ical framing which underpins and binds the entire research
In this paper, we outline our research problem via four program.
2016 PERC Proceedings, edited by Jones, Ding, and Traxler; Peer-reviewed, doi:10.1119/perc.2016.pr.081
Published by the American Association of Physics Teachers under a Creative Commons Attribution 3.0 license.
Further distribution must maintain attribution to the article’s authors, title, proceedings citation, and DOI.

344
 FIG. 1. The Activity System.

III. THEORETICAL FRAMEWORK

Because our ultimate research question is focused on indi-

vidual student level outcomes (learning gains, course grades, FIG. 2. Three interacting activity systems.
course retention, and persistence in major), one might argue
that a purely cognitive theoretical framing might be appropri-
ate for our work. These outcomes can be directly tied to latent
constructs that are conceptualized as residing within an indi- within the classroom Community, wherein sets of Rules and
vidual student’s head (e.g., conceptual understanding within a Division of Labor help to define that Community as they
the domain). As such, framing within something like Con- engage in the Activity. The Mediating Artifacts by which
ceptual Change Theory [7] might be warranted. However, we students work toward success could be at the individual level
are also keenly interested in the social construction of knowl- (e.g. student work, notes, products) or at the community level
edge and student engagement. Many of the "active learning" (e.g., group constructed responses, projects, etc). The link
activities are, by definition, interaction-based. Therefore a between Community and Mediating Artifacts in the activity
more social cognitive theory of learning [8] might be ap- triangle represents the engagement of the class with the activ-
propriate, or even a sociocultural or situated theory [9, 10] ities. The focus in the activity system is on the complex in-
owing to the complexity of classroom and student cultures terrelationships between students and their community, while
which mediate individual student learning. Due to the com- still considering the individual outcome space as a unit of
plex nature of our target phenomenon and our multifaceted analysis [15], hence the need for dialectic thinking.
units of analysis, we find ourselves arriving at the "Learning An important aspect of our work is the notion of interact-
Paradox", the boundary between cognitive and cultural ap- ing networks at various levels as parts of a larger system. For
proaches to learning and development [11, 12]. We are cer- example, students (the Subject in our basic activity system)
tainly not the first to approach this boundary from a practi- are nested within courses, that are in turn nested within dis-
cal, theoretical-need standpoint [13]. What is needed to help ciplines. This nested structure lends itself nicely to quantita-
frame our work is more of a metatheory which draws on ap- tive analysis using Hierarchical Linear Modeling (HLM; dis-
propriate aspects of these different theories of learning. cussed further below) but can that analysis be justified by in-
The metatheoretical framework of Cultural Historical Ac- teracting networks of multiple activity systems? Indeed, the
tivity Theory (CHAT) [11, 14] addresses these complexities "third generation" of CHAT does just this, and considers at
and needs. In particular, the dialectic nature of CHAT affords a minimum two (and in our work three) interacting activity
us the opportunity to consider multiple, seemingly discor- systems (see Fig.2).
dant units of analysis (e.g., individual/collective) which can In this set of three interacting activity systems, the ultimate
be conceptually isolated from one another, but still must be object of interest is still student success. But there are three
understood together to make sense of the target phenomenon. objects (at the student, course, and discipline levels) that in-
In CHAT, the activity system builds on Vygotsky’s notion of teract to contribute to a single, object-oriented conceptualiza-
mediation [10], and in its basic form is represented in the "ac- tion of student success. Students themselves contribute char-
tivity triangle" (see Fig.1). acteristics, experiences, and motivations towards their own
In our basic activity system, the Subject is the student, success. Accordingly, the Student is the Subject and student
the Object is student success, and the Outcome represents success is still the Object in the activity system on the lower
the way student success is operationalized (learning gains, right (see Fig.2). Course structures, curriculum, and instruc-
course grades, course retention, and persistence in major). tion also contribute to student success, therefore the Course
The Activity itself is learning science. Where the activity sys- is the Subject and student success is still the Object in the
tem conceptualization is particularly powerful is in represent- activity system on the lower left (see Fig.2). And finally, dis-
ing the situated nature of the Subject>Object>Outcome space ciplinary knowledge structures, norms, and cultures also con-

345
tribute to student success. In the activity system on the top of [17] who take a situated view of knowledge. They state that
Figure 2, the Discipline is the Subject and student success is practitioners typically engage in reasoning that utilizes causal
again the Object. Communities exist within and across each models to solve ill-defined problems and suggest that stu-
of these interacting systems. The complex interaction of these dents gain conceptual understanding through an apprentice-
three levels of the object comprise our object of ultimate in- ship where they engage in the practices of a domain.
terest: student success within the systems. The concept of three-dimensional learning [18] promotes
the integration of scientific practices, concepts that span the
science disciplines, and disciplinary core ideas in develop-
IV. METHODOLOGY ing tasks. Not only does engaging in scientific practice align
with the way we described authentic activity above, but also
The interacting activity systems depicted in Figure 2 help focusing on a few central concepts creates further alignment
to define our data collection and analyses. Research ques- because it is around those that science practitioners organize
tion 1 (What are the characteristics of active learning tasks and contextualize their knowledge [19]. Although originally
employed in undergraduate science courses?) is focused on developed for K-12, we contend that the concept of three-
mediating artifacts in the course level system. Research ques- dimensional learning applies equally well to post-secondary
tions 2 and 3 (How can the design and implementation of classrooms because of the importance of authenticity. The
active learning methods be understood in terms of a theoret- Three-Dimensional Learning Assessment Protocol [20] was
ical framework? and How are LAs involved in supporting designed, in part, to determine if assessment tasks are likely
active learning?) are focused on community, rules, and di- to prompt students engagement with the three dimensions.
vision of labor (if the activity is social) in the course level We are using the protocol to characterize the active learning
system. Finally, research question 4 (How does engaging in tasks being used in the courses we are observing. Again, we
active learning with and without LA support contribute to stu- will develop this characterization into predictor variables in
dent level outcomes?) is focused on the interacting resultant our quantitative models. We will interpret how well aligned
object of all three activity systems. In this section, we discuss the task is with three-dimensional learning as a measure of
the instruments we have identified, adapted, or developed for the authenticity of the task.
data collection and analysis, and the quantitative analyses we
propose in order to answer each of the research questions.

B. Characterizing Community
A. Characterizing Mediating Artifacts
We are constraining the bounds of Community for our sci-
The Mediating Artifacts of our activity system correspond ence learning activity system to the classroom. This means
to the active learning tasks students engage with in the class- that the Community will consist of the students, the instruc-
room. We are concerned with identifying the types of active tor, and the LAs (if it is an LA-supported course). There
learning being used, and with characterizing the tasks as a are three types of interactions that we are aiming to char-
means of developing a set of predictor variables for our quan- acterize within the Community: student-student interactions
titative models (see Section IV C) and to build descriptors for which we will examine through student surveys to define the
our qualitative arguments. classroom network space and each member’s position in that
To identify the types of active learning methods, we are network [21]; instructor-student interactions which we will
observing classes using the Classroom Observation protocol look at through student action COPUS codes (’answering a
for Undergraduate STEM (COPUS) [16]. The codes from the question’, ’asks a question’, ’engaged in whole class discus-
COPUS that we note as active learning types are ’individual sion’) and instructor action COPUS codes (’answering ques-
thinking or problem solving’, ’discussing clicker questions’, tions’, ’moving through class guiding ongoing student work
’working in groups on a worksheet activity’, ’other assigned during active learning task’, ’one-on-one extended discussion
group activity’, and ’engaged in whole class discussion’. The with one or a few individuals’); and LA-student interactions
types of active learning methods used and their frequency are characterized by codes similar to instructor codes from the
possible variables in our quantitative models. COPUS.
To characterize the Mediating Artifacts further, we need In characterizing the link between Community and the Me-
to analyze the tasks involved. Our decision for what analy- diating Artifacts we are concerned with determining the ex-
sis to execute is based on what task characteristics we think tent to which students engage with each other and the tasks,
will have the greatest impact on student success in science and the extent to which instructors and LAs facilitate that en-
courses. Our hypothesis is that students will be more suc- gagement. Class-level engagement will be measured based
cessful in courses where they engage in ’authentic’ tasks. By on Chi’s Interactive>Constructive>Active>Passive frame-
’authentic’ tasks we mean tasks where students do activities work [22] and the other pieces will be measured through
that scientists actually engage in, in ways that scientists do. the frequency of the Community interaction codes described
This is consistent with the definition given by Brown et al. above.

346
 C. Building Quantitative Models example, any effect due to socioeconomic status (for which
Expected Financial Contribution is a proxy) cannot be disen-
Because our object of interest is viewed as existing within tangled statistically or substantively from any other predic-
three interacting activity systems, and because those systems tor variable unless considered in concert with other factors in
exist at different levels (due to the nestedness of students the interacting activity systems. These potential interactions
within courses within disciplines), we are using Hierarchical must not only be understood statistically, but also theoreti-
Linear Modeling (HLM) [23] to model and help make sense cally.
of the outcome measures. In using HLM, we take very seri-
ously the challenge of defining the model variables with "pre-
V. DISCUSSION AND FUTURE WORK
cise meaning so that statistical results can be related to the
theoretical concerns that motivate the research" [23]. CHAT
helps us to do that in a way that addresses the dialectic nature Although we have yet to answer our research questions, our
of these data and the complexity of the object of interest. current efforts are strengthened by the CHAT framework. Im-
portantly, the dialectic, object-oriented nature of CHAT will
Our initial quantitative models will be run with two levels
help us to make sense of a complex object (student success)
(students within courses), as our current data collection is fo-
in a set of interacting activity systems at multiple levels: stu-
cused there. For now, the discipline level exists only concep-
dent, course, and discipline. Our current challenges are to
tually to help our framing and interpretation of the object of
define our data in meaningful variables which can be mod-
measurement. Outcomes to be modeled include student gains
eled quantitatively. The operationalization of the variables
on concept inventories, course grades and drop/fail/withdraw
will be informed by our qualitative arguments and interpreta-
likelihoods, and persistence in major. At the student level
tions, considering other data sources from each of the inter-
(level 1) the following predictors will be included in the
acting activity systems. In the future, we also plan to charac-
model (as random effects): student engagement, gender, eth-
terize the discipline level in a way that can contribute to the
nicity, Expected Financial Contribution, GPA. At the course
quantitative models.
level (level 2), the following predictors will be included to
model randomly varying slopes at level 1: course activities
variable, class engagement variable, instructor actions vari- ACKNOWLEDGMENTS
able, Learning Assistant actions variable.
Without an explicit theoretical framing, the relative contri- This work is supported by NSF DUE award 1525115, and
bution of each effect (i.e., slopes in regression equations) on this paper is Contribution No. LAA-040 of the International
student outcomes would be only minimally interpretable. For Learning Assistant Alliance.

[1] S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Oko- 1987).

roafor, H. Jordt, and M. P. Wenderoth, Proc. Natl. Acad. Sci. [12] C. Bereiter, Rev. Educ. Res. 55, 201 (1985).
U. S. A. (2014). [13] S. Vosniadou, Educational Psychologist 42, 55 (2007).
[2] V. Otero, N. Finkelstein, R. McCray, and S. J. Pollock, Science [14] W.-M. Roth and Y.-J. Lee, Rev. Educ. Res. 77, 186 (2007).
313, 445 (2006). [15] Y. Engeström, Journal of Education and Work 14, 133 (2001).
[3] R. M. Talbot, L. Hartley, K. Marzetta, and B. Wee, Journal of [16] M. K. Smith, F. H. M. Jones, S. L. Gilbert, and C. E. Wieman,
College Science Teaching 44, 24 (2015). CBE Life Sci. Educ. 12, 618 (2013).
[4] R. Hake, American Journal of Physics 66, 64 (1998). [17] J. S. Brown, A. Collins, and P. Duguid, Educational Researcher
[5] C. E. Wieman, Proc. Natl. Acad. Sci. U. S. A. 111, 8319 18, 32 (1989).
(2014). [18] N. R. Council, A framework for K-12 science education:
[6] M. A. Ruiz-Primo, D. Briggs, H. Iverson, R. M. Talbot, and Practices, crosscutting concepts, and core ideas (National
L. A. Shepard, Science 331, 1269 (2011). Academies Press, 2012).
[7] G. J. Posner, K. A. Strike, P. W. Hewson, and W. A. Gertzog, [19] J. D. Bransford, A. L. Brown, and R. R. Cocking, How peo-
Sci. Educ. 66, 211 (1982). ple learn: Brain, mind, experience, and school. (National
[8] A. Bandura, Social foundations of thought and action : a social Academy Press, 1999).
cognitive theory (Englewood Cliffs, N.J.: Prentice-Hall, 1986). [20] J. T. Laverty, S. M. Underwood, R. L. Matz, L. A. Posey, J. H.
[9] J. Lave and E. Wenger, Situated Learning: Legitimate Pe- Carmel, M. D. Caballero, C. L. Fata-Hartley, D. Ebert-May,
ripheral Participation, Learning in Doing: Social, Cognitive S. E. Jardeleza, and M. M. Cooper (2016), 1604.07770.
and Computational Perspectives (Cambridge University Press, [21] R. M. Talbot, L. M. Hartley, and L. Liddick, Characterizing
1991). student engagement in a learning assistant supported biology
[10] L. Vygotsky, Mind in society: The development of higher psy- course: The classroom as a social network (2014).
chological processes (Harvard University Press, 1978). [22] M. T. H. Chi and R. Wylie, Educ. Psychol. 49, 219 (2014).
[11] Y. Engeström, Learning by Expanding: An Activity-theoretical [23] S. W. Raudenbush and A. S. Bryk, Hierarchical Linear Mod-
Approach to Developmental Research (Orienta-Konsultit Oy, els: Applications and Data Analysis Methods (SAGE, 2002).

347