2019 Book UtilizingLearningAnalyticsToSu

Dirk Ifenthaler · Dana-Kristin Mah
Jane Yin-Kim Yau Editors
Utilizing
Learning
Analytics to
Support Study
Success
Utilizing Learning Analytics to Support Study
Success
Dirk Ifenthaler • Dana-Kristin Mah
Jane Yin-Kim Yau
Editors
Utilizing Learning Analytics

to Support Study Success
Editors
Dirk Ifenthaler Dana-Kristin Mah
University of Mannheim University of Mannheim
Mannheim, BW, Germany Mannheim, BW, Germany
Curtin University
Perth, WA, Australia
Jane Yin-Kim Yau

University of Mannheim
Mannheim, BW, Germany
ISBN 978-3-319-64791-3 ISBN 978-3-319-64792-0 (eBook)

https://doi.org/10.1007/978-3-319-64792-0
Library of Congress Control Number: 2018968406
© Springer Nature Switzerland AG 2019

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, express or implied, with respect to the material contained herein or for any errors
or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims
in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface
Advances in educational technology have enabled opportunities to provide insight

into how learners engage within the learning environment provided. The resulting
availability of vast amounts of educational data can represent how students interact
with higher education resources, and further analysis may provide useful insights
into learning behaviour, processes, and outcomes. From a holistic point of view,
learning analytics use static and dynamic educational information from digital
learning environments, administrative systems, and social platforms for real-time
modelling, prediction, and optimization of learning processes, learning environ-
ments, and educational decision-making. Accordingly, learning analytics are
expected to provide benefits for all stakeholders (e.g. students, teachers, designers,
administrators) in the higher education arena.
In particular, students may benefit from learning analytics through personalized
and adaptive support of their learning journey. For example, students often enter
higher education academically unprepared and with unrealistic perceptions and
expectations of academic competencies for their studies. Both the inability to cope
with academic requirements and unrealistic perceptions and expectations of univer-
sity life, in particular with regard to academic competencies, are important factors
for leaving the institution prior to degree completion.
Still, research in learning analytics and how they support students at higher edu-
cation institutions is scarce. Therefore, this edited volume Utilizing Learning
Analytics to Support Study Success aims to provide insights into how educational
data and digital technologies contribute towards successful learning and teaching
scenarios. We organized the chapters included in this edited volume into three major
parts: (I) Theoretical and Technological Perspectives Linking Learning Analytics
and Study Success, (II) Issues and Challenges for Implementing Learning Analytics,
and (III) Learning Analytics Case Studies – Practices and Evidence, and closing
with an Epilogue.
In Part I, the first chapter, the authors discuss how learning theories and learning
analytics are important components of educational research and concludes by pro-
posing an iterative loop for educational research employing learning analytics in
which learning theories guide data collection and analyses (Jacqueline Wong,
v
vi Preface
Martine Baars, Björn B. de Koning, Tim van der Zee, Dan Davis, Mohammad
Khalil, Geert-Jan Houben, Fred Paas, Chap. 1). The next chapter presents a critical
reflection on empirical evidence linking study success and learning analytics.
Findings are reported and discussed focussing on positive evidence on the use of
learning analytics to support study success, insufficient evidence on the use of
learning analytics, and link between learning analytics and intervention measures to
facilitate study success (Dirk Ifenthaler, Dana-Kristin Mah, Jane Yin-Kim Yau,
Chap. 2). The next chapter describes how the Study Support Centre (SSC) at Aalen
UAS assists first-year students of all faculties and, in particular, improves their
mathematical skills (Miriam Hommel, Armin Egetenmeier, Ulrike Maier, Chap. 3).
The following chapter shows how a prompting application has been implemented
into an existing university environment by adding a plug-in to the local digital learn-
ing platform which injects user-centric prompts to specific objects within their digi-
tal learning environment. The solution is used to perform various educational
research studies, focussing on effects of prompting for self-regulated learning
(Daniel Klasen, Dirk Ifenthaler, Chap. 4). The final chapter of the first part explores
cognitive and motivational differences between students who drop out and students
who persist. From their findings, the authors consider the monitoring and analysing
of error streaks as a promising way for the design of adaptive instructional interven-
tions in courses where the students have to programme code (Anja Hawlitschek, Till
Krenz, Sebastian Zug, Chap. 5).
In Part II, the first chapter focusses on a practical tool that can be used to identify
risks and challenges that arise when implementing learning analytics initiatives and
discuss how to approach these to find acceptable solutions (Philipp Leitner, Markus
Ebner, Martin Ebner, Chap. 6). Next, the LAPS project is introduced, which is able
to analyse progressions of former students and to make statements on possible risks
for currently enrolled students by using machine learning techniques. The chapter
provides insights into how the project is technically developed and how it can be
used in consultation situations (Mathias Hinkelmann, Tobias Jordine, Chap. 7). The
argument that precourse data could be valuable resources for learning analytics is
explored in the following chapter. The authors discuss the difficulties of collecting
data from open web-based learning environments, from missing data to interactions
between cognitive and meta-cognitive variables (Katja Derr, Reinhold Hübl,
Mohammed Zaki Ahmed, Chap. 8). The next chapter addresses issues and chal-
lenges for implementing writing analytics in higher education through theoretical
considerations that emerge from the literature review and an example application
(Duygu Bektik, Chap. 9). Then, a collaborative research project is presented which
explores the short-term and long-term effects, risks, and benefits of the use of
mobile learning analytics in students’ daily life (Luisa Seiler, Matthias Kuhnel, Dirk
Ifenthaler, Andrea Honal, Chap. 10). The following chapter reviews three categories
of algorithms in light of their application to assessment and student success. The
authors discuss an implementation of these algorithms through a new set of digital
tools, designed to support a community of practice in problem-based instruction
(Philippe J. Giabbanelli, Andrew A. Tawfik, Vishrant K. Gupta, Chap. 11). In the
final chapter of the second part, the researchers studied archival data from online
Preface vii
undergraduate course registrants through mining a dataset to determine trends and

patterns of student success, as determined by the final grade earned in the online
courses (Ellina Chernobilsky, Susan Hayes, Chap. 12).
In Part III, the authors of the first chapter present a teacher-friendly “learning
analytics lifecycle” that seeks to address challenges and critically assess the adop-
tion and impact of a unique solution in the form of an learning analytics platform
that is designed to be adaptable by teachers to diverse contexts (Natasha Arthars,
Mollie Dollinger, Lorenzo Vigentini, Danny Y.-T. Liu, Elsuida Kondo, Deborah
M. King, Chap. 13). Next, the presented study identifies key predictors of persis-
tence and achievement amongst students enrolled in an online English language
course. The study is framed in Deci and Ryan’s self-determination theory (SDT)
and uses data from a precourse student readiness survey, LMS log files, and a course
Facebook page (Danny Glick, Anat Cohen, Eitan Festinger, Di Xu, Qiujie Li, Mark
Warschauer, Chap. 14). The following chapter presents a study which investigates
how participants in a massive open online course (MOOC) designed for working
professionals interacted with various key course components of the MOOC and the
usage patterns connected to participants’ profiles and perceptions (Min Liu, Wenting
Zou, ChengLu Li, Yi Shi, Zilong Pan, Xin Pan, Chap. 15). The final chapter of this
part reports a case study focussing on a capstone unit in business at a university in
Western Australia. Instructors used learning analytics of average weekly eTest
scores, overall average eTest scores, a benchmark assessment score, and study mode
extracted from learning management system (LMS) reports to target areas where
assessment integrity could be improved (Michael Baird, Lesley Sefcik, Steve Steyn,
Chap. 16).
The edited volume closes with an Epilogue reflecting on the contributions of this
edited volume and identifying future research and directions in learning analytics
to enhance study success (Dana-Kristin Mah, Jane Yin-Kim Yau, Dirk Ifenthaler,
Chap. 17).
Without the assistance of experts in the field of learning analytics, the editors
would have been unable to prepare this volume for publication. We wish to thank
our board of reviewers for their tremendous help with both reviewing the chapters
and linguistic editing.
Mannheim, BW, Germany Dirk Ifenthaler

Dana-Kristin Mah
Jane Yin-Kim Yau
Contents
Part I Theoretical and Technological Perspectives

Linking Learning Analytics and Study Success
1 Educational Theories and Learning Analytics:
From Data to Knowledge�� 3
Jacqueline Wong, Martine Baars, Björn B. de Koning,
Tim van der Zee, Dan Davis, Mohammad Khalil,
Geert-Jan Houben, and Fred Paas
2 Utilising Learning Analytics for Study Success:
Reflections on Current Empirical Findings�� 27
Dirk Ifenthaler, Dana-Kristin Mah, and Jane Yin-Kim Yau
3 Supporting Stakeholders with Learning Analytics
to Increase Study Success�� 37
Miriam Hommel, Armin Egetenmeier, and Ulrike Maier
4 Implementing Learning Analytics into Existing
Higher Education Legacy Systems �� 61
Daniel Klasen and Dirk Ifenthaler
5 When Students Get Stuck: Adaptive Remote Labs
as a Way to Support Students in Practical
Engineering Education�� 73
Anja Hawlitschek, Till Krenz, and Sebastian Zug
Part II Issues and Challenges for Implementing Learning Analytics

6 Learning Analytics Challenges to Overcome
in Higher Education Institutions�� 91
Philipp Leitner, Markus Ebner, and Martin Ebner
ix
x Contents
7 The LAPS Project: Using Machine Learning

Techniques for Early Student Support�� 105
Mathias Hinkelmann and Tobias Jordine
8 Monitoring the Use of Learning Strategies
in a Web-Based Pre-course in Mathematics�� 119
Katja Derr, Reinhold Hübl, and Mohammed Zaki Ahmed
9 Issues and Challenges for Implementing Writing
Analytics at Higher Education�� 143
Duygu Bektik
10 Digital Applications as Smart Solutions
for Learning and Teaching at Higher Education Institutions�� 157
Luisa Seiler, Matthias Kuhnel, Dirk Ifenthaler, and Andrea Honal
11 Learning Analytics to Support Teachers’ Assessment
of Problem Solving: A Novel Application for Machine
Learning and Graph Algorithms�� 175
Philippe J. Giabbanelli, Andrew A. Tawfik, and Vishrant K. Gupta
12 Utilizing Learning Analytics in Small Institutions:
A Study of Performance of Adult Learners in Online Classes�� 201
Ellina Chernobilsky and Susan Hayes
Part III Learning Analytics Case Studies: Practices and Evidence

13 Empowering Teachers to Personalize Learning Support �� 223
Natasha Arthars, Mollie Dollinger, Lorenzo Vigentini,
Danny Y.-T. Liu, Elsuida Kondo, and Deborah M. King
14 Predicting Success, Preventing Failure�� 249
Danny Glick, Anat Cohen, Eitan Festinger, Di Xu,
Qiujie Li, and Mark Warschauer
15 Using Learning Analytics to Examine Relationships
Between Learners’ Usage Data with Their Profiles
and Perceptions: A Case Study of a MOOC Designed
for Working Professionals �� 275
Min Liu, Wenting Zou, Chenglu Li, Yi Shi, Zilong Pan,
and Xin Pan
16 Learning Analytics Leading to Remote Invigilation
for eTests: A Case Study�� 295
Michael Baird, Lesley Sefcik, Steve Steyn, and Connie Price
17 Epilogue: Future Directions on Learning Analytics
to Enhance Study Success �� 313
Dana-Kristin Mah, Jane Yin-Kim Yau, and Dirk Ifenthaler
Index�� 323
About the Editors
Dirk Ifenthaler is Professor and Chair of Learning, Design, and Technology at the

University of Mannheim, Germany; Adjunct Professor at Deakin University,
Australia; and UNESCO Deputy-Chair of Data Science in Higher Education
Learning and Teaching at Curtin University, Australia. His previous roles include
Professor and Director at Centre for Research in Digital Learning at Deakin
University, Australia; Manager of Applied Research and Learning Analytics at Open
Universities Australia, Australia; and Professor of Applied Teaching and Learning
Research at the University of Potsdam, Germany. He was a 2012 Fulbright Scholar-
in-Residence at the Jeannine Rainbolt College of Education at the University of
Oklahoma, USA. Professor Ifenthaler’s research focusses on the intersection of cog-
nitive psychology, educational technology, learning science, data analytics, and
organizational learning. He developed automated and computer-based methodolo-
gies for the assessment, analysis, and feedback of graphical and natural language
representations as well as simulation and game environments for teacher education.
His research outcomes include numerous co-authored books, book series, book
chapters, journal articles, and international conference papers as well as successful
grant funding in Australia, Germany, and the USA. Professor Ifenthaler is the
Editor-in-Chief of the Springer journal Technology, Knowledge and Learning (www.
springer.com/10758). Dirk is the past president for the AECT (Association for
Educational Communications and Technology) Design and Development Division;
past chair for the AERA (American Educational Research Association) Special
Interest Group Technology, Instruction, Cognition and Learning; and co-programme
chair for the international conference series on Cognition and Exploratory Learning
in the Digital Age (CELDA).
Dana-Kristin Mah is a researcher and consultant in the field of educational tech-

nologies and higher education. In her doctoral thesis, she concentrated on students’
first-year experience in higher education with a focus on academic competencies
and the potential of learning analytics and digital badges to enhance first-year
student retention. She is co-editor of the edited volume Foundations of Digital
Badges and Micro-credentials: Demonstrating and Recognizing Knowledge and
xi
xii About the Editors
Competencies (Springer). Her previous roles include research and teaching assistant
in the Department Educational and Socialization Processes at the University of
Potsdam, Germany, and teaching assistant at the Centre for Scientific Continuing
Education and Cooperation at the Berlin Institute of Technology, Germany. She
studied at the Berlin Institute of Technology, Germany, and Stockholm University,
Sweden, to receive her master’s degree in educational science.
Jane Yin-Kim Yau is a researcher in Learning Analytics and Mobile Learning at

the Chair for Learning, Design and Technology in the Business School at the
University of Mannheim, Germany. She is working on the BMBF-funded project
“Utilising learning analytics for study success” from 2017 to 2018. She completed
a PhD in computer science (mobile learning) at the University of Warwick, UK, in
2010. Her doctoral thesis was entitled “A mobile context-aware learning schedule
framework with Java learning objects”. She has research expertise in context aware-
ness, personalization, and user profiling. She was awarded a postdoctoral research
fellowship at the Centre for Learning and Knowledge Technologies at Linnaeus
University (CeLeKT), Sweden, where she collaborated with multidisciplinary
research teams in the various projects undertaken by the group. Thereafter, she was
a postdoc at the School of Technology, Malmö University, Sweden, and was a co-
applicant in two large successful research applications: Practice-based Experimental
Learning Analytics Research Support project, PELARS (EU FP7, 2014–2017) and
the Internet of Things and People Research Centre at Malmö University. She was
also a visiting researcher at the German Institute for International Educational
Research (DIPF) in 2016 in Frankfurt Am Main, Germany. She is a reviewer in the
IEEE Transactions in Learning Technologies, Educational Technology & Society,
International Journal of Mobile and Blended Learning, and International Journal
on Mobile Learning and Organisation, amongst others. To date, she has published
around 40 peer-reviewed articles including 16 journal articles.
About the Authors
Mohammed Zaki Ahmed is an Associate Professor (Senior Lecturer) in

Information Technology at the School of Computing, Electronics and Mathematics
of Plymouth University. He works in signal processing research and manages the
MEng and BEng programmes in electrical and electronic engineering at Plymouth
University.
Natasha Arthars is a current PhD candidate at the Centre for Research on Learning

and Innovation (CRLI) where she studies epistemic environments and learning
spaces in higher education.
Martine Baars is an Assistant Professor of Educational Psychology at the Erasmus

University Rotterdam. Her research is focussed on instructional strategies to
improve self-regulated learning. She investigates what cues are used for self-
regulated learning. Also, she explores the role of cognitive load, task complexity,
and motivation in self-regulated learning.
Michael Baird is a lecturer in the School of Marketing at Curtin University,

Western Australia. Dr. Baird has taught more than ten different marketing and
advertising units in the Curtin Business School in Perth, Sydney, Singapore,
Malaysia, and Hong Kong since 2004. Dr. Baird’s research interests include con-
tract cheating, academic misconduct, capstone courses, and consumer behaviour
and branding. Dr. Baird has a Bachelor of Commerce with first-class honours, a
masters by research, and a PhD in marketing.
Duygu Bektik is a mixed methods researcher and currently works at the Institute

of Educational Technology, Learning and Teaching Innovation’s academic profes-
sional development team as a lecturer. Her research interests currently lie primarily
in the fields of learning analytics, particularly writing analytics, e-assessment and
automated feedback, artificial intelligence in education (AIED), successful integra-
tion of ICT into the teaching and learning process, and achieving gender equity in
xiii
xiv About the Authors
academia and empowering women in academia/STEM. She has worked on several

European and UK research projects as a researcher with several partners.
Chenglu Li is a doctoral student in Learning Technologies Programme and has a

strong interest in educational games and educational data mining.
Ellina Chernobilsky is a Professor in the School of Education at Caldwell

University. She also serves as the Director of Graduate Studies and the leader of the
Caldwell University Center for Faculty Teaching and Learning. Prior to earning her
PhD, Ellina was a classroom teacher and used action research as means to study and
improve her own teaching in order to help herself and her students to become better
learners. Her research interests include but are not limited to multilingualism, action
research in education, the use of data and data mining in education, and issues of
professional development on all levels of education. She teaches research courses
regularly. She has spent time teaching in China and Russia.
Anat Cohen is a tenured-track senior academic staff member at Tel Aviv University’s

School of Education and Head of the Learning and Technology programme in the
Department of Education in Mathematics, Science and Technology. She is also
Deputy Chair of UNESCO’s Technology Internationalization and Education (TIE), a
research and pedagogical coordinator of Web-Supported Academic Instruction at Tel
Aviv University’s Virtual TAU, and a co-PI of a research project funded by the
Ministry of Education’s Chief Scientist. Dr. Cohen has vast experience in research and
teaching in the fields of learning and cyber technologies, training of academic staff,
and the design of learning management systems. Her research interests include, but
are not limited to, social networks and privacy perception in cyber space, implementa-
tion of web-based courses in higher education, issues related to cost-effectiveness of
Web-based learning, innovative pedagogical practices, open educational resources,
learning analytics, and educational data mining. She has published over 90 research
papers in leading peer-reviewed journals and national and international conference
proceedings.
Dan Davis is a PhD candidate at Delft University of Technology. He develops

methods to gain a deeper understanding about how the design of online learning
environments affects learner success and engagement, often by designing, developing,
and evaluating instructional interventions at scale.
Björn B. de Koning is an Assistant Professor of Educational Psychology in the

Department of Psychology, Education, and Child Studies at Erasmus University
Rotterdam, Rotterdam, Netherlands. De Koning’s research concentrates on design-
ing effective instructional support for learners and using process-oriented mea-
sures such as eye tracking to uncover the learning process. His research projects
focus on optimizing learning from dynamic visualizations, designing effective
multimedia instruction, using cognitive strategies to foster reading comprehension,
supporting mental model construction in mathematical word problem-solving, and
About the Authors xv
understanding the influence of collaboration on learning processes and perfor-

mance in online and offline learning environments. These projects involve learners
from primary education to higher education. His work is characterized by a multi-
disciplinary approach that aims to provide insights for improving educational prac-
tice by integrating cognitive science, developmental, and educational psychology
research perspectives.
Katja Derr worked in the field of e-learning design and development before com-
pleting a degree in education at Freiburg University of Education. Since 2007, she
has been involved in mathematics e-learning projects in tertiary education. Since
2012, she is a research staff member in the joint project optes.
Mollie Dollinger is an associate lecturer at La Trobe University where she

researches student success and the student experience.
Markus Ebner previously worked with the Institute of Interactive Systems and
Data Science and currently works as a junior researcher in the Department
Educational Technology at Graz University of Technology. His doctoral research
deals with e-learning, mobile learning, technology-enhanced learning, and open
educational resources. His specific focus is on learning analytics at the K–12 level.
In this framework, he is contributing to an EU project with the aim to analyse and
promote the language acquisition of children. In addition, he has published several
publications in the area of learning analytics and held workshops on the topic.
Martin Ebner is currently Head of the Department Educational Technology at

Graz University of Technology and, therefore, responsible for all university-wide
e-learning activities. He holds an adjunct professorship in media informatics
(research area, educational technology) and also works at the Institute for Interactive
Systems and Data Science as a senior researcher. His research focusses strongly on
seamless learning, learning analytics, open educational resources, making, and
computer science for children. Martin has given a number of lectures in this area as
well as workshops and keynote lectures at international conferences. To view his
publications as well as further research activities, please visit his website: http://
martinebner.at.
Armin Egetenmeier studied business mathematics at the University of Ulm. He

graduated in 2013 with a master’s degree in science. Since 2013, he is an academic
assistant at the Study Support Centre of Aalen University of Applied Sciences.
At the SSC, he is responsible for the professional supervision of students in the
study entry phase, primarily in (business) mathematics. Another focus of his work
is scientific accompanying research. In particular, he is responsible for develop-
ments of algorithms and visualizations. His research interests lie in the areas of
transition from school to university, learning analytics, modelling of teaching and
learning processes, as well as educational data mining and data analysis.
xvi About the Authors
Eitan Festinger is a second-year MA student at the Tel Aviv University’s School

of Education, majoring in mathematics, science, and technology. He received a BA
in computer science from the Interdisciplinary Center in Herzliya and an MA in
business administration from Bar Ilan University. He has 15 years’ experience
working for privately held software companies in the fields of quality monitoring
and analytics.
Philippe J. Giabbanelli, PhD is an Assistant Professor of Computer Science at

Furman University. He directs the Data Analytics in Complex Human Behaviors
(DACHB) laboratory. He has published close to 50 articles using data science tech-
niques such as machine learning, network science, and simulation. He develops and
applies these techniques to problems in human behaviours. His research on mental
models in particular has appeared in the Winter and Spring simulation conferences.
Danny Glick is an educational researcher, learning technologist, and TMLL

implementation specialist with nearly two decades of global experience in planning,
designing, implementing and evaluating small- and large-scale technology-
enhanced programmes for the K–12 market, academic institutions, and government
agencies and organizations in both developing and developed countries. Dr. Glick is
a research affiliate at the University of California, Irvine’s Digital Learning Lab,
where he explores ways to improve learning experiences and student success in
online courses by identifying factors that predict students’ performance and persis-
tence in online learning environments. Since 2004, he has worked as Director of
Pedagogy and Research at Edusoft, a subsidiary of Educational Testing Services
(ETS), which develops blended and distance programmes for ministries of educa-
tion and universities in more than 30 countries. Dr. Glick is the pedagogical director
and PI of several large-scale technology-enhanced programmes implemented across
Latin America reaching 750,000 students. He specializes in developing early warn-
ing systems to identify at-risk students in online courses using learning analytics.
Vishrant K. Gupta is a graduate student supervised by Dr. Giabbanelli. His thesis

focusses on the comparison of student and expert maps. His work on visualizing and
recommending alignments between maps is due to appear in the proceedings of the
Human Computer Interaction (HCI) conference. One of his key contributions is the
ongoing development of the ITACM software, which combines technologies (e.g.
Java, D3, Spring, MySQL, Neo4J) to offer a user-centric environment supporting a
community of practice in assessing ill-structured problems.
Anja Hawlitschek is currently working as research associate at Magdeburg-

Stendal University of Applied Sciences in the project “Industrial eLab”. This proj-
ect which is funded by the German Federal Ministry of Education and Research
addresses the usage of remote laboratories for teaching and learning. Her research
focusses on instructional design for blended learning, e-learning, and game-based
learning.
About the Authors xvii
Susan Hayes is the Director of Institutional Research and Assessment at Caldwell

University. She also serves as Caldwell’s Accreditation Liaison Officer to the
Middle States Commission on Higher Education. Prior to working in higher educa-
tion, Susan conducted research in non-profit and public settings and earned a mas-
ter’s degree in public administration from New York University. She is currently a
doctoral student in education leadership, and her research interests include student
persistence, learning analytics, organizational culture, and institutional effective-
ness. She also serves on the steering committee of the New Jersey Association for
Institutional Research, coordinating professional development and networking
opportunities in area of higher education research professionals.
Mathias Hinkelmann received a diploma (Dipl-Ing) in mechanical engineering in

1990 and PhD (Dr-Ing) in 2001 at the University of Stuttgart. He has been working
for 7 years as a business consultant and manager in a large consulting company. His
project work was focussed on Data Warehousing and Business Intelligence. Since
2003, he is a Professor for database systems at the Hochschule der Medien, Stuttgart.
Since 2007, he is the Vice President of academic affairs at the Hochschule der
Medien. Mathias Hinkelmann is the project manager of the LAPS project.
Miriam Hommel received her diploma in geodesy and geoinformatics from

University Karlsruhe (TH), Germany, in 2006, and her PhD from Karlsruhe Institute
of Technology (KIT) in 2010. From 2006 to 2010, she was a staff member of the
Institute of Photogrammetry and Remote Sensing (IPF) at KIT and from 2010 to
2011 of the Fraunhofer Institute for Manufacturing Engineering and Automation
(IPA). Since 2012, she is an academic assistant at the Study Support Centre of Aalen
University of Applied Sciences. At the SSC, she is responsible for the professional
supervision of students in the introductory phase, primarily in mathematics. Another
focus of her work is scientific accompanying research. In particular, she is respon-
sible for the feedback emails as well as the study course reports. Her research inter-
ests lie in the areas of transition from school to university, modelling of teaching and
learning processes, as well as statistical data analysis, classification, and learning
analytics.
Andrea Honal is a Business Professor of Management, Marketing, and Media at

Baden-Wuerttemberg Cooperative State University Mannheim since 2012. She is
currently involved in various research projects focussing on mobile learning and
new technologies in the field of higher education. Within the project “Mobile
Learning Analytics” of the Cooperative State University Mannheim and the
University of Mannheim, her role concentrates on the strategic management of the
project and on the pedagogical part of it.
Geert-Jan Houben is Full Professor of Web Information Systems at the Software

Technology Department at Delft University of Technology (TU Delft). His main
research interests are in web engineering, web science, and user modelling, adapta-
tion, and personalization. He is the Managing Editor of the Journal of Web
xviii About the Authors
Engineering (JWE), an editorial board member for the Journal of Web Science
(JWS), the International Journal of Web Science (IJWS), User Modeling and User-
Adapted Interaction (UMUAI), and ACM Transactions on the Web (ACM TWEB).
In Delft, he is the scientific director of Delft Data Science (DDS), TU Delft’s coor-
dinating initiative in the field of data science, holding the KIVI chair Big Data
Science, leading TU Delft’s research programme on Open & Online Education in
TU Delft Extension School, and the principal investigator in AMS, Amsterdam
Institute for Advanced Metropolitan Solutions. He is currently serving as Director
of Education at the Faculty of Electrical Engineering, Mathematics and Computer
Science at TU Delft.
Reinhold Hübl, PhD (1987), is a Professor at Baden-Wuerttemberg Cooperative

State University Mannheim where he teaches mathematics and acts as scientific
coordinator of the centre of basic knowledge in mathematics and sciences.
Dirk Ifenthaler is a Professor and Chair of Learning, Design and Technology at

the University of Mannheim, Germany, and UNESCO Deputy Chair of Data Science
in Higher Education Teaching and Learning at Curtin University, Australia. His
research focusses on the intersection of cognitive psychology, educational technol-
ogy, data analytics, and organizational learning. Dirk’s research outcomes include
numerous co-authored books, book series, book chapters, journal articles, and inter-
national conference papers as well as successful grant funding in Australia,
Germany, and the USA. He is the Editor-in-Chief of the Springer journal Technology,
Knowledge and Learning (www.ifenthaler.info).
Tobias Jordine received his BSc and MSc in computer science and media at the
Stuttgart Media University in 2009 and 2011. In the beginning of 2013, he started
his PhD studies in computer science education in cooperation with the University of
the West of Scotland and the Hochschule der Medien. He finished his PhD in
November 2017. He presented at the Frontiers in Education (FIE) Conference,
Madrid; the European Conference in the Applications of Enabling Technologies,
Glasgow; and the European Conference on Games-Based Learning, Paisley, where
he presented his PhD topic. Tobias Jordine is responsible for the technical develop-
ment of the LAPS project.
Mohammad Khalil is a postdoctoral candidate at Delft University of Technology,

funded by the Leiden-Delft-Erasmus Centre for Education and Learning (LDECEL)
consortium. Mohammad has a doctoral degree in computer science from Graz
University of Technology. His PhD dissertation was about learning analytics in
massive open online courses (MOOCs). At the moment, his research is strongly
related to MOOCs, online learning, and learning analytics. For publications as well
as further research activities, visit his website: http://mohdkhalil.wordpress.com.
Deborah M. King is an Associate Dean (undergraduate programmes) in the

Faculty of Science. She is committed to enabling sector-wide improvement in
About the Authors xix
tertiary learning and teaching, particularly through the professional development of

staff and creation of communities of practice. In her native discipline of mathemat-
ics, Deborah has recently completed two national projects, MathsAssess and
FYiMaths, from which a large network of educators has developed to disseminate
and adopt best practice in tertiary mathematics education.
Elsuida Kondo is a developer of learning and teaching for the Faculty of Science

at the University of Melbourne. Elsuida’s background is in analytical chemistry,
mathematical modelling, and learning and teaching development. In her current
role, her main interest is in learning analytics evaluation, student engagement, and
interactive learning development.
Till Krenz has been working as a research assistant at the department for methods
of social sciences at the University of Magdeburg from 2011 to 2017 and is cur-
rently part of the research project “Industrial e-Lab”. While engaging in research
activities in different fields, as social network analysis, social capital, and higher
education studies, he is focussed on the data science aspects of social science, pro-
viding data transformation, analysis, and visualization skills to his research part-
ners. He is also developing extensions for the statistics software R.
Matthias Kuhnel studied information design and media management at the

University of Applied Sciences Merseburg. Since January 2017, he is academic
assistant at the University of Mannheim at the Chair of Economic and Business
Education, Learning, Design and Technology. Within the project “Mobile Learning
Analytics” of the Cooperative State University Mannheim and the University of
Mannheim, he is mainly responsible for the technical parts. His actual research field
is in the scope of mobile learning and learning analytics.
Philipp Leitner is currently working for the Department of Educational Technology

at Graz University of Technology as a junior researcher at the Institute of Interactive
Systems and Data Science. His doctoral research focusses on learning analytics in
higher education and, specifically, on technology-enhanced learning, learning ana-
lytics, data privacy, and recommender systems. Philipp has already published sev-
eral publications and has held workshops in those research areas. For further
information on publications and research activities, please visit his website: https://
philipp-leitner.at.
Qiujie Li is a PhD candidate in the Department of Education at the University of

California, Irvine. She completed her BA in educational technology in 2011 at
Beijing Normal University and received an MA in distance education from the same
university in 2014. Her research focusses on learning analytics, online learning
environment design, and related fields. She hopes to use learning analytics in explor-
ing the behaviour patterns of online learners in order to discern their learning prob-
lems and to offer better support for them.
xx About the Authors
Danny Y.-T. Liu is a molecular biologist by training, programmer by night,

researcher and academic developer by day, and educator at heart. A multiple national
teaching award winner, he works at the confluence of learning analytics, student
engagement, educational technology, and professional development and leadership
to enhance the student experience.
Min Liu is a Professor of Learning Technologies at the University of Texas at

Austin. Her teaching and research interests centre on educational uses of new
media and other emerging technologies, particularly the impact of such technolo-
gies on teaching and learning; and the design of new media enriched interactive
learning environments for learners at all age levels. She has published over 68
research articles in leading peer-reviewed educational technology journals and 13
peer-reviewed book chapters and presents regularly at national and international
technology conferences. She also serves on a number of editorial boards for
research journals in educational technology. Her current R&D projects include
studying the design and effectiveness of immersive, rich media environments on
learning and motivation, analytics in serious game environments, examining the
affordances and constraints of using mobile technologies in teaching and learning,
understanding MOOCs as an emerging online learning tool, and the use of Web 2.0
tools to facilitate instruction.
Dana-Kristin Mah is a researcher and consultant in the field of educational tech-

nologies and higher education. In her doctoral thesis, she concentrated on students’
first-year experience in higher education with a focus on academic competencies
and the potential of learning analytics and digital badges to enhance first-year stu-
dent retention. She is co-editor of the edited volume Foundations of Digital Badges
and Micro-credentials: Demonstrating and Recognizing Knowledge and
Competencies (Springer). Her previous roles include research and teaching assistant
in the Department Educational and Socialization Processes at the University of
Potsdam, Germany, and teaching assistant at the Centre for Scientific Continuing
Education and Cooperation at the Berlin Institute of Technology, Germany.
Ulrike Maier received her Diploma in Mathematics in 1990 and in 1994 her PhD
in mathematics from the University Dortmund, Germany. She was an assistant to
chairs of applied mathematics at the Universities Dortmund and Giessen, Germany,
from 1990 to 2002. From 2002 to 2004, she researched in the fields of medical tech-
nics and automotive production at the Fraunhofer Institute for “Algorithms and
Scientific Computing” (SCAI) at Sankt Augustin, Germany. Research for Zeiss
SMT AG, Oberkochen, Germany, followed from 2005 to 2007. From 2007 to 2008,
she worked as an assistant to chair of optoelectronics at the University of Mannheim,
Germany. She was a lecturer at the University of Applied Sciences at Heidenheim,
Germany, Spring 2008, and an assistant at the KIZ of the University of Ulm,
Germany, from 2008 to 2011. Since 2012, she is an academic assistant at the Study
Support Centre of Aalen University of Applied Sciences. At the SSC, she is respon-
sible for the professional supervision of students in the introductory phase,
About the Authors xxi
primarily in mathematics, and is the head of the research group of the SSC. Research
interests are in optimization, approximation theory, scientific computing, and learn-
ing processes.
Fred Paas is a Professor of Educational Psychology at Erasmus University

Rotterdam in the Netherlands and Professorial Fellow at the University of
Wollongong in Australia. Since 1990, he has been using the theoretical framework
of cognitive load theory to investigate the instructional control of cognitive load in
the training of complex cognitive tasks. He has (co)authored more than 250 SSCI-
listed journal articles, which have generated more than 28,000 citations. He is
Editor-in-Chief of the journal Educational Psychology Review and on the editorial
board of several other renowned journals, such as the Journal of Educational
Psychology. He is a fellow of the American Educational Research Association. In
2016, he was recognized as the world’s most productive author in the five best jour-
nals in the field of educational psychology for the period 2009–2014.
Connie Price is a senior curriculum consultant specializing in assessment design.

She has recently led key strategic projects in assessment quality assurance and qual-
ity improvement and has been involved in the review and renewal of assessment
policy at the university. She has a particular interest in digital and electronic assess-
ment modalities across a range of assessment types. These interests culminate in the
current project focussing on the monitoring and management of academic integrity
in online assessments.
Luisa Seiler studied pedagogy and business education at the University of

Koblenz-Landau. She completed her master’s degree in 2016 with her thesis “The
contribution of modern brain research to performance management”, in cooperation
with W.L. Gore & Associates. Luisa Seiler has been a research associate at Baden-
Wuerttemberg Cooperative State University since January 2017. Her current
research fields are digitalization and new media in the higher education context,
digital competencies, and human resources. Moreover, she is an external PhD stu-
dent at the University of Mannheim.
Lesley Sefcik is a lecturer and academic integrity advisor at Curtin University. She

provides university-wide teaching, advice, and academic research within the field of
academic integrity. Dr. Sefcik has an interdisciplinary education with a PhD in envi-
ronmental science (University of Michigan) focussing on plant physiological ecol-
ogy and global change, a Bachelor of Science in biology (University of Michigan)
and a Graduate Diploma in education (Murdoch University) majoring in science
and humanities and social science. She is a registered secondary teacher in Western
Australia has been awarded an outstanding teacher rating for the National
Professional Standards for Teachers in all seven domains. Dr. Sefcik’s professional
background is situated in assessment and quality learning within the domain of
learning and teaching. Past projects include the development of the External
Referencing of Standards (ERoS) system for external peer review of assessment.
xxii About the Authors
Current projects include the development and implementation of academic integrity

related programmes for students and staff at Curtin and research related to the
development and implementation of remote invigilation for online assessment.
Steve Steyn is a professional engineer and certified project manager with a decade

of experience worldwide in process control, automation, GIS, product development,
and project management industries. Steve taught at the North-West University in
South Africa for undergraduate students.
Andrew A. Tawfik, PhD is Assistant Professor of Instructional Design and

Technology at the University of Memphis. In addition, Dr. Tawfik also serves as the
director for the University of Memphis Instructional Design & Technology Studio,
where he oversees the design, development, and evaluation of digital instructional
resources in K–12, business, and medical contexts. His research interests include
problem-based learning, case-based reasoning, case library instructional design,
and computer-supported collaborative learning.
Tim van der Zee has MSc in psychology and is currently a PhD student at the
Leiden University Graduate School of Teaching (ICLON) in the Netherlands. In his
research, he focusses on understanding and improving the educational quality of
open online courses such as MOOCs (massive open online courses).
Lorenzo Vigentini is the academic lead in educational intelligence and analytics

in the portfolio of the Pro-Vice Chancellor Education at UNSW Sydney, steering
several initiatives at the intersection of educational analytics, institutional evalua-
tion, quality enhancement, student engagement, and educational technology tools
development. He holds a dual appointment as senior lecturer in the PVCE with
adjunct status with the School of Education and the School of Computing Science.
His methodological and technical background with expertise in psychology, learn-
ing and teaching, e-learning, and research methods puts him in a unique position to
bridge traditional educational theory, psychology, academic development, and the
new and evolving fields of educational data mining and learning/academic analyt-
ics, which is assuming a prominent role in providing essential evidence (both data
and tools) to inform strategy, QA and QE at various levels, empowering teachers
and managers to make the right choices to improve teaching practice and supporting
students experience and their future success.
Mark Warschauer is a Professor of Education and Informatics at the University

of California, Irvine. Dr. Warschauer is Director of the Digital Learning Lab (DLL)
at UC Irvine where he works on a range of research projects related to digital media
in education. In K–12 education, his team is developing and studying cloud-based
writing, examining new forms of automated writing assessment, exploring digital
scaffolding for reading, investigating one-to-one programmes with Chromebooks,
and analysing the use of interactive mobile robots for virtual inclusion. In higher
education, his team is looking at instructional practices in STEM lecture courses,
About the Authors xxiii
the impact of virtual learning on student achievement, the learning processes and
outcomes in massive open online courses (MOOCs), and the impact on students of
multitasking with digital media. The DLL team is also exploring new approaches to
data mining, machine learning, and learning analytics to analyse the learning and
educational data that result from use of new digital tools. Dr. Warschauer is author
and editor of a wide range of books. He is the founding editor of Language Learning
and Technology journal and has been appointed inaugural editor of AERA Open.
Wenting Zou is currently a doctoral student in Learning Technologies Programme

at the University of Texas at Austin. She is interested in using learning analytics to
understand learners’ behaviours across different e-learning platforms.
Jacqueline Wong is a PhD candidate in the Department of Psychology, Education

and Child Studies of Erasmus University Rotterdam. Her research focusses on moti-
vation and self-regulated learning in open online learning environments. She exam-
ines the influence of student characteristic and the effect of learning supports on
student success in massive open online courses (MOOCs).
Xin Pan is currently a second-year doctoral student in learning technologies at UT

Austin. Her interests in educational technology include MOOCs, game-based learn-
ing, enhancing science literacy with simulations and animations, and integrating
technologies into informal spaces such as museums.
Di Xu is an Assistant Professor of Educational Policy and Social Context at the

University of California, Irvine. Dr. Xu holds a PhD in economics and education
from Columbia University. Her research examines the impacts of educational pro-
grammes and policies on student academic performance, persistence, and degree
completion at the post-secondary education level, with a particular focus on stu-
dents from disadvantaged backgrounds.
Jane Yin-Kim Yau is a researcher in Learning Analytics and Mobile Learning at

the Chair for Learning, Design and Technology in the Business School at the
University of Mannheim, Germany. She is working on the BMBF-funded project
“Utilising learning analytics for study success” from 2017 to 2018. She completed a
PhD computer science (mobile learning) at the University of Warwick, UK, in 2010.
Yi Shi is a doctoral student in the Learning Technologies Programme at UT. With

a background in education, she is interested in how technologies could be used to
enhance the process of teaching and learning.
Zilong Pan is a doctoral student in the Learning Technologies Programme at the

University of Texas at Austin. He earned his master’s degree in middle school edu-
cation from the University of Georgia. He had taught in middle schools in Atlanta
area as a science and math teacher for 2 years. His research interests include learn-
ing analytics and integrating learning technology into STEM education.
xxiv About the Authors
Sebastian Zug is an Assistant Professor (junior professor) at the Faculty of

Computer Science of the OVGU since 2014. He holds the Chair of “Smart Embedded
Systems”. His research interests are focussed on fault tolerance mechanisms for
sensor systems in autonomous applications and outdoor robotics. Additionally, he
investigates the opportunities of web-based learning systems, especially remote
control applications, for academic teaching. One of the research projects of the
working group funded by the Federal Ministry of Education and Research (BMBF)
addresses this interdisciplinary topic. Sebastian Zug is currently involved in four
national projects and is a member of the German RoboCup Committee.
Part I
Theoretical and Technological
Perspectives Linking Learning
Analytics and Study Success
Chapter 1
Educational Theories and Learning
Analytics: From Data to Knowledge
The Whole Is Greater Than the Sum of Its Parts
Jacqueline Wong, Martine Baars, Björn B. de Koning, Tim van der Zee,

Dan Davis, Mohammad Khalil, Geert-Jan Houben, and Fred Paas
1 Introduction
Without theories, people could view research findings as disorganized collections of data,
because researchers and practitioners would have no overarching frameworks to which the
data could be linked.
Schunk (2012, p. 10)
At all levels of education, the widespread use of new technologies such as interac-
tive learning environments, learning management systems (LMS), intelligent tutor-
ing systems (ITS), and online learning provides access to large amounts of student
data (e.g. user interaction with online course content; Gašević, Dawson, & Siemens,
2015). Despite being a rich source of information, student data automatically col-
lected in online learning environments is typically not transformed into useful infor-
mation for teaching and learning (Greller & Drachsler, 2012) and is used poorly
across the educational domain (Dawson, Gašević, Siemens, & Joksimovic, 2014).
In starting to transform large amounts of student data into useful information for
J. Wong (*) · M. Baars · B. B. de Koning

Erasmus University Rotterdam, Rotterdam, The Netherlands
e-mail: [email protected]
T. van der Zee
Leiden University, Leiden, The Netherlands
D. Davis · G.-J. Houben
Delft University of Technology, Delft, The Netherlands
M. Khalil
Centre for the Science of Learning and Technology, University of Bergen, Bergen, Norway
F. Paas
Erasmus University Rotterdam, Rotterdam, The Netherlands
University of Wollongong, Wollongong, NSW, Australia
© Springer Nature Switzerland AG 2019 3

D. Ifenthaler et al. (eds.), Utilizing Learning Analytics to Support Study Success,
https://doi.org/10.1007/978-3-319-64792-0_1
4 J. Wong et al.
learning, educational researchers recently have taken an interest in learning analyt-

ics approaches (Knight & Buckingham Shum, 2017).
Although learning analytics is an evolving discipline, it draws on research, meth-
ods, and techniques from multiple established disciplines such as data mining,
information visualization, psychology, and educational sciences (Gašević et al.,
2015). Learning analytics is commonly defined as “the measurement, collection,
analysis, and reporting of data about the learners and their contexts for the purposes
of understanding and optimizing learning and the environment in which it occurs”
(Siemens & Long, 2011, p. 34). Trace data, also known as audit trails, log files, and
event traces, are captured in online environments as students study the learning
materials (Hadwin, Nesbit, Jamieson-Noel, Code, & Winne, 2007). By utilizing
learning analytics to examine the trace data, patterns related to learning processes
can be identified to deepen our understanding of how students learn and add to the
development of learning theories. In turn, this will help guide the design of instruc-
tional materials to support and enhance learning.
Given that understanding learning is a highly complex issue (Phillips, 2014),
many learning theories have been developed over the last century based on different
views of what learning is (Murphy & Knight, 2016). Learning theories are impor-
tant not only because they can help to explain the phenomenon of learning but also
because design principles for learning environments, materials, and tasks can be
derived from the theories (Ertmer & Newby, 1993). Moreover, learning theories can
help to convert information from learning analytics into actionable knowledge for
instructional and learning design.
Importantly, as expressed by Ifenthaler (2017), a synergistic relationship between
instructional design and learning analytics exists. On one hand, instructional design-
ers can better evaluate the learning environment, materials, and tasks by processing
data about the learners and their complex interactions within the learning environ-
ment using learning analytics approaches. On the other hand, learning analytics
require theories and principles on instructional design to guide the transformation of
the information obtained from the data into useful knowledge for instructional
design. Consistent with Ifenthaler’s (2017) view, this chapter emphasizes the impor-
tance of taking learning theories into account when employing learning analytics in
studies to support study success.
The aim of this chapter is to discuss how learning theories and learning analytics
could be integrated in educational research since the whole is greater than the sum
of its parts. We will first discuss the definition of learning and the role of learning
theories. Then, a qualitative analysis of studies employing learning analytics to
examine the current role of learning theories in research using learning analytics
will be presented. Section 4 discusses the studies reviewed and proposes an iterative
educational research loop to integrate both educational theories and learning
analytics.
1 Educational Theories and Learning Analytics: From Data to Knowledge 5
2 Understanding Learning
Building strong connections with the learning sciences was listed as one of the
future directions of learning analytics by Ferguson (2012). The author reasoned that
a good understanding of how learning occurs, how learning can be supported, and
how student characteristics influence learning are needed if the goal of learning
analytics is to understand and optimize learning. To understand the “how” of learn-
ing, one has to first define what learning is. Alexander, Schallert, and Reynolds
(2009) proposed that learning can be defined as “a multidimensional process that
results in a relatively enduring change in a person or persons, and consequently how
that person or persons will perceive the world and reciprocally respond to its affor-
dances physically, psychologically, and socially. The process of learning has as its
foundation the systemic, dynamic, and interactive relation between the nature of the
learner and the object of the learning as ecologically situated in a given time and
place as well as over time” (p. 186). This definition encapsulates the many perspec-
tives of learning that were derived from the evolution of learning theories.
2.1 Evolution of Learning Theories
Based on a recent review of papers published in Review of Educational Research

(RER) journal over the last century, Murphy and Knight (2016) found that learning
sciences have been guided by three predominant theoretical lenses: behavioural,
cognitive, and contextual. The authors used the word “lenses” to analogously refer
to the theories that researchers use. Just like how a certain lens may be more suitable
for taking pictures in one situation than another, one learning theory may be more
suitable for understanding learning in one environment than another. At the begin-
ning of the twentieth century, learning was viewed as a change in behaviour (for an
overview of learning theories, see Ormrod, 1999). Using the behavioural lens (e.g.
Skinner, 1977), researchers focused on the responses of individuals to the environ-
ment and the ways to condition the desired responses. Several theories, such as
classical conditioning and drive reduction theory, emerged from the behavioural
viewpoint. In the middle of the twentieth century, the cognitive lens (e.g. Ausubel,
1969) was used, viewing learning as a change in the mind of an individual. The
focus was on understanding the mental processes that influence the processing and
storing of information in the mind. Multiple theories, such as information process-
ing theory and cognitive constructivism, developed under the cognitive lens.
Although behavioural and cognitive lenses explained changes in one’s behaviour
and mind, researchers were missing theories to explain social factors that influence
learning that occurred in groups. The contextual lens arose to fill this gap. Under the
contextual lens (e.g. Vygotsky, 1978), learning was viewed as contextually bound
and a result of social interactions. Theories that developed from the contextual lens
included social constructivism and social learning theory.
Murphy and Knight (2016) concluded that the shift in theoretical lens
occurs when findings from new studies cannot be explained by the existing lens.
6 J. Wong et al.
However, a shift in theoretical lens does not invalidate the prior lens. Instead, each
theoretical lens offers researchers the filter to focus on different areas of learning.
More importantly, multiple theories can coexist and be simultaneously used to
guide instructional practice. Therefore, it is at the discretion of learning scientists
and learning analysts to recognize these nuanced perspectives of learning provided
by the different lenses and apply learning theories based on the learning materials,
learning conditions, learning tasks, and learner characteristics.
3 Role of Educational Theories in Learning Analytics
Given that learning theories evolved to accommodate new findings from studies,
one might question if there is a need for learning theories. There is no doubt that a
learning theory has to be built upon collective findings from studies (Alexander,
2006). Yet, without a theory to begin with, researchers will not know what to look
out for. This conundrum of not knowing what to look for is magnified in research
utilizing learning analytics since studies conducted in online learning environments
usually involve the collection of immense amounts of data. Therefore, a good theory
is needed to guide researchers (Alexander, 2006). Using the theoretical lens of a
learning theory, researchers will be better positioned to formulate their research
questions, make hypotheses about what learning outcome to expect, make decisions
on the research methods, and finally, make interpretations of the results derived
from learning analytics approaches (Murphy & Knight, 2016).
Since one of the aims of learning analytics is to advance educational research and
practice, it is of interest to take a look at how well learning theories are being referred
to or investigated in studies employing learning analytics to support study success.
Na and Tasir (2017) found mixed effects of the use of learning analytics interven-
tions to support students’ success. However, it is not clear whether the learning ana-
lytics interventions in the studies reviewed were based on specific learning theories
or whether any learning theories were mentioned in the studies. Gaining insight into
this is important to aid our understanding of how learning analytics can affect study
success. Therefore, the current study extends the Na and Tasir study by investigating
whether studies employing learning analytics to support study success take into
account learning theories and, if so, to what extent the learning theories are guiding
the studies. The main research question addressed in our review is as follows:
Which learning theories have been used in the studies examining learning analyt-
ics approaches to support study success?
3.1 Research Methodology
The review methodology consisted of four sequential steps qualifying it as a sys-

tematic qualitative review: (a) literature search based on keywords to identify rele-
vant papers, (b) assessment of search results to select a set of primary studies, (c)
categorising and integration of the results, and (d) reporting the findings (Gikandi,
Morrow, & Davis, 2011).
The aim of the first step was to identify published papers examining study suc-
cess using learning analytics. Given that learning analytics has been applied to
examine success in different domains and at various levels of education, broad
search terms (i.e. study success, student success, and achievement) were used to
capture all forms of success and achievement related to study and student. The
search string “learning analytics” AND (“stud* success” OR “achievement”) was
used to search for papers indexed in the databases of Scopus (http://www.scopus.
com) and Web of Science (http://www.webofknowledge.com/wos) in December
2017. These two databases were chosen because of their multidisciplinary indexing
of articles across journals and conferences. We only included papers published in
journals and conferences over the last 7 years starting from 2011 when the first
learning analytics and knowledge conference proceeding was published. After
removing duplicates, 164 papers that were published in 46 journals and 33 confer-
ence proceedings remained.
The second step was to select a set of primary studies. Given the aim of the study
was to qualitatively review the role of learning theories in studies employing learn-
ing analytics, impact factors were used to identify papers that were published in top
five journals and conferences. We ranked the scientific influence of the 46 journals
based on impact factors obtained from Scimago Journal and Country Rank (SJR;
SCImago, 2007) and Journal Citation Reports (JCR). The two impact factors were
taken into account as SJR is built on Scopus database, while JCR is built on Web of
Science database. We ranked the conferences using H-index obtained from Google
Scholar Metrics since conferences were not ranked by SJR or JCR. Table 1.1 shows
Table 1.1 Number of papers selected based on five highest-ranked journals according to the
journal titles in alphabetical order
Number
Publications of papers SJR JCR
Journal titles
Computers & Education 6 2.61 3.82
International Journal of Computer-Supported Collaborative 6 1.47 3.47
Learning
Computers in Human Behaviour 1 1.60 3.44
Internet and Higher Education 4 2.83 4.24
Journal of Computer Assisted Learning 1 1.65 1.25
Soft Computing 1 .75 2.47
Conference titles H-index
Americas Conference on Information Systems (AMCIS) 1 22
ACM Conference on International Computing Education Research 2 19
(ICER)
Conference on User Modeling, Adaptation and Personalization 1 21
(UMAP)
IEEE Global Engineering Education Conference (EDUCON) 2 19
International Learning Analytics and Knowledge Conference (LAK) 2 32
8 J. Wong et al.
the distribution of papers published across the top five journals and conferences
according to the SJR, JCR, and H-index. This selection process resulted in a set of
27 papers published in 6 journals and 5 conferences.
The 27 papers went through a second selection process based on the study type
(i.e. experimental, correlational, student survey only, and conceptual/review). We
selected only empirical papers (i.e. experimental and correlational studies) for the
review, specifically papers that used learning analytics approaches to analyse trace
data obtained from the online learning environments. This allowed us to examine
whether the studies referred to learning theories when employing learning analytics
approaches to analyse the trace data. We refer to the definition of learning analytics
as “the measurement, collection, analysis, and reporting of data about the learners
and their contexts for the purposes of understanding and optimizing learning and the
environment in which it occurs” (Siemens & Long, 2011, p. 34). Therefore, we
selected studies that collected data about the learner in online learning environment.
During this selection process, papers that used student surveys only (Atif, Bilgin, &
Richards, 2015; Tan, Yang, Koh, & Jonathan, 2016; Zhuhadar, Yang, & Lytras,
2013), reviews (Tlili, Essalmi, Jemni, & Chen, 2016), and conceptual papers (Kim
& Moon, 2017; Wise & Schwarz, 2017; Yassine, Kadry, & Sicilia, 2016) were
removed. This resulted in a final set of 20 empirical papers involving the analysis of
trace data using learning analytics approaches.
In the third step, the 20 papers were read in detail and categorised according
to the learning theories mentioned in the papers. Information on the studied learning
environment, the learning analytics techniques/application applied, and the types of
data collected were extracted from each paper. Finally, the findings of the papers
were integrated and qualitatively reviewed based on the learning theories mentioned
in the papers to answer the research question.
3.2 Results and Discussion
Among the set of 20 papers, there were only two (quasi)experimental papers (i.e.
Rowe et al., 2017; Tabuenca, Kalz, Drachsler, & Specht, 2015) comparing different
treatment conditions. Tabuenca et al. (2015) compared the effects of delivering noti-
fications between a fixed and a random schedule to support self-regulated learning,
while Rowe et al. (2017) compared the use of in-game measures of implicit science
knowledge either as a bridge or as a supplement to teaching activities to enhance
learning. The rest of the 18 papers were correlational studies.
3.2.1 Learning Theories and Learning Analytics Applications
After categorising the papers, 16 studies were found to mention theories related to
learning, while the other four studies did not. Table 1.2 shows a summary of the
learning theories mentioned in the 16 studies, the learning environments in which
the studies were deployed, the learning analytics approaches used, and the types of
data that were collected. Most studies tended to be situated within self-regulated
learning (n = 6), followed by motivation (n = 2), and social constructivism (n = 2).
Another six individual studies used other concepts related to learning (i.e. learner
effort, feedback, deep learning, engagement, implicit knowledge, and a combina-
tion of concepts).
Self-Regulated Learning
Self-regulated learning (SRL) was the most employed theory related to learning in
the selected studies. Models of SRL characterize self-regulated learners as students
who actively use and adjust their learning strategies to achieve their learning goals
(Bos & Brand-Gruwel, 2016; Kizilcec et al., 2017). There were six studies (i.e. Bos
& Brand-Gruwel, 2016; Jovanović et al., 2017; Kizilcec et al., 2017; Siadaty et al.,
2016; Tabuenca et al., 2015; You, 2016) which examined the use of learning analyt-
ics albeit in different learning environments (e.g. MOOCs and LMS). You (2016)
used hierarchical regression analyses to identify events from data generated in
learning management systems (LMS) to predict course achievement in e-learning
courses. The results showed that students who accessed the content videos within
the instructor-scheduled time and watched the full length of the video were the
strongest predictor of course achievement, followed by the number of late submis-
sions, the number of course logins, and whether the course information was
downloaded.
Instead of predictive modelling, Jovanović et al. (2017) employed an exploratory
learning sequence analysis to compare learning sequences of high performers and
low performers in a flipped classroom. Low performers mostly focused on summa-
tive assessments that counted towards their final course scores, while high perform-
ers engaged with all the activities (i.e. formative assessment, summative assessments,
reading materials, and videos) evenly. Using agglomerative hierarchical clustering
based on Ward’s method, the authors identified five student profiles (i.e. intensive,
highly strategic, strategic, selective, and highly selective) based on the activities that
students chose to engage in (e.g. focus on summative assessment or focus on course
video). While the learning analytics approach helped to detect and describe differ-
ences in students’ learning behaviour, it could not provide reasons as to why stu-
dents’ behaviour differed.
To be able to explain differences in students’ behaviours, Kizilcec et al. (2017)
correlated student behavioural data with student self-reports about their learning
approach. The authors examined the relationship between SRL survey data, student
interactions with course contents in MOOC, and personal goal attainment. The
results showed that students’ self-reported level of SRL was related to their inten-
tions in completing the course. Students who scored higher on goal setting and
strategic planning were more likely to attain their goals, while students who reported
more help-seeking were less likely to attain their goals. In general, students with
higher self-reported use of SRL strategies spent more time revisiting assessments.
Table 1.2 Learning theories identified from the selected papers
10
Learning theories Learning LA technique/ Performance-related

used Authors environment application Trace data collected measures
Self-regulated Bos and Brand- LMS, blended – Clustering Time spent viewing recorded Mid-course and final
learning (SRL) Gruwel (2016) course lectures, number of formative
– Multiple regression course assessment,
analysis assessment completed and score on self-reported inventory of
the formative assessment, time learning styles (ILS)
spent using the LMS, and number
of clicks in the LMS (e.g.
announcements, video files,
viewing grades)
Jovanović, Gašević, Flipped course – Learning sequence Number of correctly and incorrectly Midterm and final exam
Dawson, Pardo, and with learner analysis solved summative and formative scores
Mirriahi (2017) dashboard – Clustering assessment items, number of
solutions requested, number of
videos played, number of access to
content, dashboard, and schedule
Kizilcec, Pérez- MOOCs – Logistic regression Number of transitions from one Course goals (i.e. earning
Sanagustín, and models interaction state type (e.g. begin a a course certificate,
Maldonado (2017) – Transition graphs video to complete a video) and time complete all assessments,
spent on each type of learning and complete all lectures),
material, number of learning self-reported self-
materials interacted regulation of learning
Siadaty, Gašević, and Learn-B – Trace-based Number of actions performed by Perceived usefulness of
Hatala (2016) environment methodology students in the learning the features provided in
environment (e.g. clicking on the learning environment
different competencies, choose an
available learning path, rate a
learning path)
Tabuenca et al. Online course – SQL queries to Students log their study time on the Course grades, self-
(2015) with support from examine the mobile application which in turn reported self-regulation of
visualizes the summary of their learning
J. Wong et al.
mobile distribution of
application study time recording that shows the time spent
per assignment
You (2016) LMS, e-learning – Hierarchical Time spent viewing the Midterm and final exam
regression instructional videos, number of scores
course logins, number of late
submission, students’ reply to
instructor’s post, fulfilment of
attendance, number of posting in
the discussion board
Motivation Lonn, Aguilar, and Summer bridge – Multiple linear An early warning system that Course grades, pre- and
(achievement goal) Teasley (2015) programme regression assigned students one of the three post-measures of
statuses (i.e. encourage, explore, self-reported achievement
engage) based on the points goals (i.e. mastery and
students earned on their performance approach
coursework, difference between the and performance-
course average, and number of avoidance orientation)
logins
Motivation (mastery, Barba, Kennedy, and MOOC – Structural equation Number of clicks on videos and Final grade
value beliefs, Ainley (2016) modelling number of quiz attempts
individual interest,
and situational
interest)
Socio-constructivism Joksimović, Gašević, LMS – Hierarchical linear Number of time spent on four types Final course grades
(interaction types: Loughin, Kovanović, mixed models of interaction (i.e. student-student,
student-content, and Hatala (2015) using restricted student-content, student-teacher,
student-instructor, maximum student-system)
student-student) likelihood (REML)
1 Educational Theories and Learning Analytics: From Data to Knowledge
estimate
Social learning theory Carter and Social – Chi-squared test Number of interaction types (i.e. Average grade for
Hundhausen (2016) programming post, reply, receive a suggestion), programming assignment
environment topic of post, progress in the course and final course grade
Learner effort Zhao et al. (2017) MOOC – k-means clustering Time spent watching videos and Eligibility to earn a
(distributed practice) quiz score course certificate
11
(continued)
Table 1.2 (continued)
12
Learning theories Learning LA technique/ Performance-related

used Authors environment application Trace data collected measures
Feedback Sedrakyan, Snoeck, Conceptual – Process model Event log of students’ group work Scores on the group
(process-oriented) and de Weerdt (2014) modelling discovery and during the modelling process (i.e. project’s final solution
environment dotted chart create, edit, delete, redo, and copy)
(JMermaid) analysis
Deep learning Romero-Zaldivar, Virtual appliance – Multiple regression Time spent in the learning Final grades
Pardo, Burgos, and – Prediction environment, number of times
Kloos (2012) an action was performed (i.e. write
a command, open a webpage, open
a file with an editor, and use the C
compiler, memory profiler, and C
debugger), time spent performing
each action
Engagement Junco and Clem Digital textbooks – Hierarchical linear Number of reading days, number of Final course grades
(2015) regression reading sessions, time spent
reading, number of pages read,
number of highlights, number of
bookmarks, number of notes
Implicit knowledge Rowe et al. (2017) Computer game – Approach map for Implicit knowledge measured by Pre-/post-assessment
network clustering in-game behaviour involving improvement
specific strategic moves
Combination of Kim, Park, Yoon, and LMS in blended – Random forest Time spent on LMS, number of Final course grades
concepts (active Jo (2016) course technique to create LMS visits, number of discussion
participation, a prediction model board visits, number of posts, post
engagement, length, interval between LMS visits,
consistent effort and interval between discussion board
awareness, visits, number of replies received by
interaction) a student, number of replies
generated by a student
J. Wong et al.
Based on the results, the authors suggested MOOC instructors to guide students in
goal setting and strategic planning activities.
Instead of analysing temporal learning sequences, Bos and Brand-Gruwel (2016)
chose a more direct method of counting the number of times an activity was done
and the time spent on the activities in the learning environment. Similar to Kizilcec
et al.’s (2017) study, Bos and Brand-Gruwel (2016) combined SRL survey data with
data generated in a LMS platform. In the study, students were first clustered based
on their scores on the administered SRL surveys. The analysis resulted in three
clusters: (1) students who reported lack of regulation when external regulation is
absent, (2) students who reported use of self-regulation strategies when external
regulation is absent, and (3) students without a clear regulation strategy. The results
showed that although students in the three clusters used the online resources to a
similar extent (e.g. number of videos watched), they benefited differently from the
use of the same resources. Frequencies of login and time spent in the LMS alone
were found to be poor predictors of students’ performance. This is not surprising
given that the duration measured may not be the actual time students spent process-
ing information on the page in an online environment.
Two studies were found to examine interventions that support SRL. Siadaty et al.
(2016) examined the relationship between students’ perceived usefulness of the
interventions and actual use of SRL interventions. Seven SRL scaffolds were
embedded in a technologically enhanced learning environment: (1) usage informa-
tion, (2) social context of the workplace, (3) progress towards goal attainment, (4)
peer-recommended learning goal, (5) system-recommended competencies, (6)
system-recommended learning path, and (7) learning resources students own or
have shared with the organization. The authors predefined activities in the online
environment to measure SRL processes. For example, rating a learning path in the
online environment is a measurement of self-evaluation as a SRL process. The anal-
ysis of students’ activities in the online environment showed that (1) frequencies of
planning activities were related to looking at usage information, social context of
workplace, and system-recommended competencies and learning path, (2) frequen-
cies related to performance phase were related to information about social context
of the workplace and learning resources they own or have shared with the organiza-
tion, and (3) frequencies related to reflection phase were related to competences of
goals. The findings suggested that providing information on social context of the
workplace had the highest impact on processes of SRL. The authors concluded that
recommender system technology should be integrated in modern workplace envi-
ronments to support SRL. Although this study showed that recommender system
technology enhances SRL on the whole, it is not clear which factors in particular
(e.g. system-recommended competencies or system-recommended learning path)
influenced SRL. Moreover, a recommender system might increase students’ reliance
on the recommendations instead of their own regulation of learning.
In another experimental intervention study by Tabuenca et al. (2015), a within-
subjects design was used to examine the effect of a mobile tool for tracking and
monitoring study time on SRL. At different time points in the study, students
received notifications containing tips for time management that were either generic
14 J. Wong et al.
or based on learning analytics at random time or on a fixed schedule. Students

reported an increase in perceptions of time management and planning skills after
the notification intervention. Students specifically preferred notifications sent early
in the day with learning analytics information about their personal time manage-
ment and behaviour. Activities in the time logs showed that students were more
active at certain time periods and on certain days, and there were more records of
study time whenever notifications were sent. However, students who had more time
logs did not score higher in the final exam than students who had less time logs.
The six discussed studies exemplify the complexity of examining SRL in an
online environment. SRL processes consist of a broad range of learning strategies
such as time management, goal setting, and planning. The studies used different
learning analytics approaches to examine the trace data. Trace data can be examined
by aggregating an action in terms of frequencies and time spent on the online mate-
rials (e.g. Bos & Brand-Gruwel, 2016), action in context such as submitting an
assignment on time (e.g. You, 2016), transitions of activities (e.g. Kizilcec et al.,
2017), and learning sequences (e.g. Jovanović et al., 2017). The learning analytics
approaches provide insights into what students do in the online environment that
might relate to SRL. However, trace data alone are insufficient to explain students’
behaviour. Among the selected studies, four studies attempted to shed more light on
this by relating trace data to self-report data. The combination of trace data and self-
reports enables a deeper understanding on the relationship between SRL and stu-
dents’ behaviour. For example, students who reported higher levels of SRL also
spent more time revisiting assessments (Kizilcec et al., 2017). It should be noted
that these studies involved primarily correlational analyses, so causality cannot be
inferred from these studies. Therefore, there is a need for more experimental studies
such as the Tabuenca et al.’s (2015) study. Together, the selected studies suggest that
SRL is a promising area in which learning theories and learning analytics converge.
The fact that SRL turned out to be the most investigated learning theory in learning
analytics research is understandable given that SRL has been shown to be crucial to
academic success in online learning environments (Broadbent & Poon, 2015).
Motivation
Two studies (i.e. Barba et al., 2016; Lonn et al., 2015) examined motivation, each
with a different theoretical approach. Barba et al. (2016) examined the impact of
general motivation (i.e. individual interest, mastery approach, utility value beliefs)
and state-level motivation (i.e. situational interest). Motivation in this study was
defined as systems of beliefs that can be activated by contextual and personal fac-
tors. Using structural equation modelling, they investigated the relationship between
motivation, participation, and study success in MOOCs. The different types of moti-
vation were measured by surveys, whereas participation in MOOC activities was
measured by the number of videos viewed and the number of quizzes attempted.
The results showed that students who reported a mastery approach towards learning
attempted more quizzes. Students’ report of higher situational interest was related
to larger number of videos watched. The strongest predictor of final grades in the
MOOCs was the number of quizzes attempted followed by situational interest.
These results suggest that it is important for MOOC designers to focus on support-
ing situational interest.
The study by Lonn et al. (2015) focused on achievement goal theory to measure
the effects of a learning analytics intervention in a summer bridge programme.
Achievement goal theory was used to conceptualize students’ two types of motiva-
tion orientation: mastery goals focus on the development of personal competencies,
while performance goals focus on showing competence compared to others. The
intervention in Lonn et al.’s (2015) study consisted of an early alert system that
tracked students’ progress to identify whether they were at risk. Student advisors in
the course could then look at the information provided by the early alert system and
act accordingly. Results of the study showed that the mastery approach decreased
over time, suggesting that the learning analytics intervention is negatively c orrelated
to mastery approach. Therefore, the study suggested that this learning analytics
intervention should be implemented with caution as it may have a negative influ-
ence on student motivation.
Both discussed studies used surveys to measure motivation instead of predefin-
ing student activities in the log data as proxies of motivation (as was, e.g. done in
the SRL study by Siadaty et al., 2016). This could be due to the fact that motivation
is a cognitive process related to goal-directed behaviour (Schunk, 2012). The two
studies exemplify the important relationship between learning theories and learning
analytics. Barba et al. (2016) linked student motivation to participation, providing
insights to how motivation can be manifested in learning behaviours. This suggests
that learning analytics can help to quantify learning behaviours to deepen our under-
standing of motivation—what behaviours are related to motivation. Lonn et al.’s
(2015) study showed that learning analytics interventions can affect motivation.
This suggests that learning theories can help guide the implementation of learning
analytics interventions—how can motivation be supported to enhance study
success.
Social Constructivism
Two studies (i.e. Carter & Hundhausen, 2016; Joksimović et al., 2015) were catego-
rised under the theoretical framework of social constructivism. As discussed in Sect.
2, social constructivism can be viewed from a contextual lens. Under this view,
learning does not occur only within the learner but is contextualized and dependent
on the environment. These studies examined the interactions in online learning
environments and related the interactions to theory of social constructivism. Carter
and Hundhausen (2016) examined peer interactions using trace data generated in a
programming environment where students could pose and answer questions. The
results showed that students who asked a question, received a suggestion, and
acknowledge the suggestion were more likely to make progress in the course and
achieve better final grades.
16 J. Wong et al.
Joksimović et al. (2015) not only examined student-student interaction but also
interaction between student and instructor, student and content, and student and
system in an online course. The analytical approach involved identifying the inter-
actions, classifying them into interaction types, calculating the frequency and time
spent on each interaction type, and statistically analysing the relationship between
interaction types and final grades. The results showed that student-system interac-
tions were positively related to final grades, while student-content interactions were
negatively related to final grades. Also, student-instructor interactions were nega-
tively correlated to final grades in core courses only. Based on these results, the
authors suggested that the different courses (i.e. core, elective, and foundational
courses) require different forms of interactions to support the learning process.
The discussed studies demonstrate that using learning analytics enables research-
ers to examine the effect of actual interactions instead of relying on only perceived
interactions. The results from the two studies showed that interactions such as
student-student interactions (Carter & Hundhausen, 2016) or student-system inter-
actions (Joksimović et al., 2015) can differentially affect grades. Future studies can
build on these two studies to further compare different properties of interactions
(e.g. asynchronous, synchronous, virtual, augmented). In addition, learning analyt-
ics can also be used to help students monitor their interactions. To conclude, there is
a reciprocal relationship between learning analytics and social constructivism.
Learning analytics provide evidence for learning from a social constructivist
perspective, while social constructivism helps to make sense of interaction data
provided by learning analytics.
Studies Using Specific Learning Concepts
In this section, other specific learning concepts mentioned in individual papers are
discussed. What stands out is that the extent to which the learning theories were
discussed in the studies as well as the moment at which they were introduced varied.
Most studies introduced the learning theories at the beginning but failed to link the
patterns or clusters obtained back to the learning theories. In some studies, certain
concepts related to learning were mentioned although no clear learning theories
were stated.
Zhao et al. (2017) investigated the link between assessment and learner effort
within a MOOC. Educational researchers suggest that learner effort should be dis-
tributed evenly across topics and course weeks. This appears to be related to the
concept of distributed practice (Dunlosky & Rawson, 2015). Results of the study
showed that MOOC students behaved differently after meeting the minimum pass-
ing requirement. Some students reduced their engagement with videos and quizzes
after passing, suggesting that students who passed did not necessarily have complete
mastery of all course content. The authors concluded that differences in post-passing
behaviours may be related to students’ motivation for taking the course. However,
student motivation was not actually measured in this study.
The role of feedback is mentioned in Sedrakyan et al.’s (2014) study. Feedback

can be linked to several learning theories depending on the focus of the feedback
(Thurlings, Vermeulen, Bastiaens, & Stijnen, 2013). Feedback is also viewed as an
important component of self-regulated learning (Butler & Winne, 1995). Sedrakyan
et al. (2014) examined whether quality of work can be predicted by differences in
students’ learning patterns. Based on a three-dimensional analysis (i.e. hierarchical,
modelling, and time trend), the results showed that the quality of work can be pre-
dicted by students’ learning pattern. This suggested that instructors can identify
poor-performing students and provide process-oriented feedback during the task to
enhance their quality of work. The potential of feedback to support learning is
proposed but not investigated in the study.
Romero-Zaldivar et al. (2012) employed the concept of deep learning (Webb,
1997) to evaluate the effectiveness of a virtual appliance where students interact
with the tools from a pre-installed application on the computer. Based on the
assumption of deep learning, learning is enhanced when students have high level of
interactions with the learning tools. Predictive modelling based on the frequency
and time spent with the tools in the learning environment showed that students’ final
grades can be predicted by the use of two out of the six tools available. However, the
authors did not relate the activities back to the concept of deep learning.
Likewise, predictive modelling was used in Junco and Clem’s (2015) study in
which theory of engagement was mentioned. The authors gave a brief background
on the theory on engagement by Astin (1984) which suggested that amount of learn-
ing is related to the amount of time and effort that students invest. Course outcomes
were predicted based on the usage data generated from a digital textbook. The
results showed that time spent reading was significantly related to course grades.
Also, students in the top tenth percentile used more highlights than students in the
lower 90th percentile. The study did not further examine the texts that were high-
lighted, as such, it is not clear how students were using the highlights to support
their reading.
Rowe et al. (2017) examined the assessment of implicit science knowledge in
digital games. Implicit knowledge is defined as what learners are able to do given
their existing understanding. In-game measures of implicit learning were first devel-
oped using educational data mining technique. The digital games were then either
used as a bridge for science class or as an extra activity outside of class or not used
at all in an experimental study. Using hierarchical linear models, the results showed
that the in-game measures of implicit knowledge correlated to external measures of
learning (i.e. post-assessment). Moreover, students did better in the course when
teachers use information about students’ implicit knowledge for explicit teaching.
Kim et al. (2016) constructed proxy variables in an asynchronous online discus-
sion environment to measure various concepts related to learning: active participa-
tion in the course, engagement with discussion topics, consistent effort and
awareness, and interaction. Psychological and behavioural characteristics of
high-performing students were then identified for each concept. For instance,
psychological and behavioural characteristics of consistent effort and awareness
were responsibility, punctuality, time management, and intrinsic motivation.
18 J. Wong et al.
These characteristics were further operationalized by proxy variables that can be

measured by the log file data such as interval regularity of visit to the online
environment, total time spent, number of LMS visits, number of discussion board
visits, and number of posts. To evaluate how well the proxy variables were able to
predict good and poor performers, the authors used random forest technique to
develop the prediction model. The results indicated that, using the proxy variables,
the prediction model was highly accurate. The authors suggested that for whole-
class discussions, students can be encouraged to reply to others and be supported
to work towards more in-depth discussion. For team-based discussion, the authors
suggested employing support for cognitive engagement at the beginning and sus-
tain engagement throughout the course.
The studies mentioned above suggest that learning analytics have the potential to
provide information on various learning-related concepts. Learning analytics add
value to educational research through the collection of different sources of data (e.g.
trace data) and measuring and analysing the data in ways that can be related to learn-
ing theories (e.g. predictive models and clustering). However, for learning analytics
to achieve the potential of providing deeper insights to learning, it is important to first
clearly determine which learning theories are being investigated so that decisions can
be made on which data to be collected and which analytical method to be used.
3.2.2 Absence of Learning Theories
Out of the 20 empirical studies that used correlational and experimental design, 16
studies were found to mention certain learning theories or concepts related to learn-
ing. The four studies that did not mention any learning theories were mainly focused
on using exploratory approaches to identify student behaviours predictive of aca-
demic achievement. Studies by Brooks, Erickson, Greer, and Gutwin (2014) and
Liu and d’Aquin (2017) used clustering methods to identify groups of learners that
were most likely to be successful. The third study by Carter, Hundhausen, and
Adesope (2015) argued that theories in learning research lacked the ability to pre-
dict “student performance that are dynamic, robust, and continuously updated
throughout a course”. Therefore, they proposed a normalized programming state
model that explained how removing compilation errors from a programme is related
to better achievement. Finally, Marbouti, Diefes-Dux, and Madhavan (2016)
compared seven prediction methods to evaluate the models’ accuracy in identifying
at-risk students: (1) Logistic Regression, (2) Support Vector Machine, (3) Decision
Tree, (4) Multi-Layer Perceptron, (5) Naive Bayes Classifier, (6) K-Nearest
Neighbour, and (7) ensemble model. The accuracy of the models depends on the
performance data collected which can be affected by quality and reliability of the
grading. This suggests that there is no one prediction method that is the most accu-
rate. Together, while the studies using various learning analytics methodologies
without mentioning learning theories do provide insights into factors influencing
student success, we argue that more direct links with learning theories would help
to advance the conversation from “what are the factors that influence learning?” to
“how and why do these factors influence learning?”.
4 Conclusion and Suggestions for Future Research
The aim of the current review was to investigate which theories have been used in
studies employing learning analytics to support study success. We searched for
studies in two major databases and selected 20 empirical papers for the final review.
Based on the studies reviewed, self-regulated learning (SRL) appears to be widely
referenced in studies employing learning analytics (i.e. Bos & Brand-Gruwel, 2016;
Jovanović et al., 2017; Kizilcec et al., 2017; Siadaty et al., 2016; Tabuenca et al.,
2015; You, 2016). There are also two studies related to theories about motivation
(i.e. Barba et al., 2016; Lonn et al., 2015) and two studies related to theories on
social constructivism (i.e. Carter & Hundhausen, 2016; Joksimović et al., 2015).
There are several single studies on different concepts related to learning such as
learner effort (i.e. Zhao et al., 2017), feedback (i.e. Sedrakyan et al., 2014), deep
learning (i.e. Romero-Zaldivar et al., 2012), engagement (i.e. Junco & Clem, 2015),
and implicit knowledge (i.e. Rowe et al., 2017). Kim et al.’s (2016) study is the only
exception that examined multiple concepts related to learning (i.e. active participa-
tion, engagement, consistent effort and awareness, interaction).
All of these studies are examples of how learning theories are used in studies that
employed learning analytics to examine student behaviours in online learning envi-
ronments. We observed that, at present, learning theories have been used in studies
employing learning analytics in two ways. First, learning theories help to guide
decisions on the types of data to be collected and the learning analytics approaches
to take. From the studies, it is noted that similar data points (e.g. time spent on an
activity) can be used as proxies related to different learning theories (e.g. SRL and
engagement). Therefore, learning theories play an important role in explaining the
concept of learning that is being measured. For example, researchers examining
SRL may focus on learning sequences (e.g. Jovanović et al., 2017), while research-
ers taking the perspectives of socio-constructivism may focus on students’ interac-
tions with instructors and other students. Second, learning theories help researchers
to explain why students might behave in certain ways and why behaving in certain
ways might lead to study success. For example, students who are better at SRL are
more inclined to revisit assessments and, hence, more likely to be successful learners
(Kizilcec et al., 2017).
Although this chapter has identified several learning theories mentioned in stud-
ies employing learning analytics approaches to support study success, a trend that
we observed is that learning theories are often briefly mentioned or introduced at the
beginning of the articles but rarely circled back to contextualize the results with the
learning theory mentioned (e.g. Romero-Zaldivar et al., 2012). While the first part
(introducing the theory) is certainly a step in the right direction, we contend that a
robust, thorough employment of learning theory in learning analytics should use
the results obtained from the various analyses to make direct inferences about the
applicability of the theory on the learning behaviour observed (and also, perhaps,
the method applied, as learning analytics borrows from a very wide variety of
methodologies). As learning analytics is a young, blossoming, interdisciplinary
field, it is comprised of researchers from a plethora of other fields, each bringing
20 J. Wong et al.
with them various levels of expertise in different topics. And, as is often the case in
interdisciplinary research, knowledge from some fields will inevitably be more
prominent than others. For example, a large part of learning analytics research
comes from computer science departments (Dawson et al., 2014). To move forward
within the learning analytics field, it is imperative that learning analytics researchers,
regardless of their base discipline, go beyond a surface-level understanding of the
learning theory or theories they are employing. Instead of having it merely as a
framing at the beginning of a paper, the learning theories should be integral to the
research narrative and provide explanations at every stage about how the theory
informed each decision along the way.
Learning theories play an important role in transforming results obtained from
learning analytics into insights about learning. While learning analytics can help to
identify patterns of student behaviours and add new understanding to the field of
educational research, it alone does not provide explanations for underlying mecha-
nism. The analysis of trace data in Jovanović et al.’s (2017) study helped to detect
series of student actions corresponding to the unfolding of learning strategies used by
the students, yet the results fall short in explaining what underlying factors could have
accounted for the differences in the use of learning strategies between different groups
of students. In accordance with learning theory related to self-regulated learning in
Zimmerman’s model (Zimmerman & Campillo, 2003), the use of learning strategies
is preceded by self-motivational beliefs and processes of task analysis. By adopting
Zimmerman’s model in their study, Jovanović et al.’s (2017) could examine whether
motivational beliefs influence students’ use of learning strategies manifested in the
different series of student actions. When using learning theories, researchers should
recognize that a theory may have a number of constructs, for instance motivational
beliefs can include self-efficacy beliefs, goal orientation, and task interest. Therefore,
discussions among researchers are needed to discern learning theories that may align
better with learning analytics. The potential of learning analytics can only be realized
when the nuances of learning theories are aligned with the nuances of the data.
Another trend that we observed was the considerable overlap in the analytical
techniques found in several studies. For instance, regression was mostly used as the
analytical method in the first stage followed by clustering in the second stage (Bos &
Brand-Gruwel, 2016; You, 2016; Lonn et al., 2015; Romero-Zaldivar et al., 2012; and
Junco & Clem, 2015). There were also studies that explore novel analytics approaches
such as trace-based methodology (Siadaty et al., 2016) and process model discovery
(Sedrakyan et al., 2014). The multiple analytics approaches used in the studies dem-
onstrate the ability of learning analytics to deep dive into rich data sources of log
files, discussion forums, time spent on tasks, and number of interactions to extrapo-
late learning as a holistic and social process based on students’ behaviours. However,
as noted by Gašević et al. (2015), the interpretation of students’ behaviours can
change depending on the understanding of the students’ internal conditions (e.g. cog-
nitive load, self-efficacy, achievement goal orientation, and interest) as well as exter-
nal conditions (e.g. instructional design and previous experience with using the tool).
Therefore, future studies should include multiple sources of data that can be derived
from learning theories (e.g. prior knowledge, self-report of motivation) to supple-
ment the analysis of student data generated in the online environments.
Learning Learning
Analytics Analytics
Data collection Data collection
Learning Learning
Theory 1.0 Theory 1 .n
Learning environment 1.0 Learning environment 1 .n
Fig. 1.1 Propose iterative loop in which learning theory is integral to study employing learning
analytics
We propose an iterative loop as illustrated in Fig. 1.1 to guide future educational

research employing learning analytics. The iterative loop starts with a theory of
learning (learning theory 1.0) that is used to examine how students learn in a learn-
ing environment. This is followed by theory-guided data collection so that a pre-
defined set of data is collected. Subsequently, theory-guided selection of learning
analytics methods is used to analyse the data. The analysis based on learning analyt-
ics can either provide evidence to support the hypotheses derived from learning
theory 1.0 or suggest how the theory can be developed (learning theory 1.n). The
process is iterative until the findings fit a theory. Rowe et al.’s (2017) study is an
example of a study which already fits well with what we proposed. Based on theory
of implicit knowledge, data were collected in a digital game environment to detect
student actions related to implicit knowledge based on learning analytics approaches.
Hypotheses were derived to examine whether students whose teachers used the
digital game to assess implicit knowledge as a bridge in class would perform better
than students whose teachers use the digital game as a supplementary activity and
students whose teachers did not use the digital game at all. New data are collected
in the digital game environment along with course grade to understand how assess-
ing implicit knowledge can support the teacher and ultimately enhance learning.
Beside the iterative loop, we also suggest three ways in which learning theories
can and should be used. First, learning theories can guide decisions on which
research questions to investigate or not to investigate. By keeping abreast of the
development of learning theories, future studies employing learning analytics can
focus on research questions that are not yet answered instead of running the risk
of claiming new discoveries that are perhaps long-established findings. For
example, in digital learning environments, it is typically easy to collect data about
students’ levels of activity, which is commonly found to be a great predictor of
study success (e.g. You, 2016). This mirrors the finding that in higher education,
class attendance is one of the strongest predictors of study success (Credé, Roch,
& Kieszczynka, 2010).
22 J. Wong et al.
Second, learning theories can guide the operationalization of research questions

into testable hypotheses, which is a critical step in designing an empirical test.
Knowledge from educational research helps to sidestep collection of problematic or
inappropriate variables. For example, researchers might be tempted to rely on stu-
dents’ evaluations of online courses and educational technologies to infer about
better or more effective approaches. However, student evaluations of courses and/or
teachers are only minimally related to learning outcomes and should not be used as
a proxy of learning (Clayson, 2009).
Finally, learning theories can guide the design and evaluation of tools and inter-
ventions. In the learning analytics literature, dashboards and other educational tech-
nologies are a popular subject of research. Learning theories provide highly relevant
frameworks to guide the process of creating as well as evaluating dashboards and
other educational technologies. For example, the added, or possibly detrimental,
value of visualizations in dashboards can and should be empirically assessed, for
example, by using cognitive load theory (Sweller, 2011) and the cognitive-affective
theory of multimedia learning (Mayer, 2011). Similarly, these large fields of research
are invaluable to design and create dashboards and other tools based on decades of
relevant empirical research.
In conclusion, the current study shows that learning theories are often mentioned
without much depth in the studies employing learning analytics. While learning
analyst may be proficient with analytical approaches, they may be less familiar with
the nuances of learning. Similarly, learning scientist may be apt at recognizing the
nuances of learning but not equipped with skills to perform the analytics using trace
data. Therefore, the study of learning can benefit from the joint effort of learning
scientists and learning analysts in conducting research that integrate learning theo-
ries and learning analytics. This will help to achieve an understanding of learning of
which the whole is greater than the sum of its parts.
Acknowledgements This research is supported by Leiden-Delft-Erasmus Centre for Education

and Learning.
References
Alexander, P. A. (2006). Evolution of a learning theory: A case study. Educational Psychologist,
41(4), 257–264.
Alexander, P. A., Schallert, D. L., & Reynolds, R. E. (2009). What is learning anyway? A topo-
graphical perspective considered. Educational Psychologist, 44(3), 176–192.
Astin, A. W. (1984). Student involvement: A developmental theory for higher education. Journal
of College Student Personnel, 25(4), 297–308.
Atif, A., Bilgin, A., & Richards, D. (2015). Student preferences and attitudes to the use of early
alerts. In Paper Presented at the Twenty-first Americas Conference on Information Systems.
Ausubel, D. P. (1969). A cognitive theory of school learning. Psychology in the Schools, 6(4),
331–335.
Barba, P. D., Kennedy, G. E., & Ainley, M. D. (2016). The role of students’ motivation and partici-
pation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32(3),
218–231.
Bos, N., & Brand-Gruwel, S. (2016). Student differences in regulation strategies and their use of
learning resources: Implications for educational design. In Proceedings of the 6th International
Conference on Learning Analytics & Knowledge (pp. 344–353).
Brooks, C., Erickson, G., Greer, J., & Gutwin, C. (2014). Modelling and quantifying the behaviours
of students in lecture capture environments. Computers & Education, 75, 282–292.
Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in
online higher education learning environments: A systematic review. The Internet and Higher
Education, 27, 1–13.
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis.
Review of Educational Research, 65(3), 245–281.
Carter, A. S., & Hundhausen, C. D. (2016) With a little help from my friends: An empirical study
of the interplay of students’ social activities, programming activities, and course success. In
Proceedings of the 2016 ACM Conference on International Computing Education Research
(pp. 201–209).
Carter, A. S., Hundhausen, C. D., & Adesope, O. (2015). The normalized programming state
model: Predicting student performance in computing courses based on programming behav-
ior. In Proceedings of the 11th Annual International Conference on International Computing
Education Research (pp. 141–150).
Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn?
A meta-analysis and review of the literature. Journal of Marketing Education, 31(1), 16–30.
Credé, M., Roch, S. G., & Kieszczynka, U. M. (2010). Class attendance in college: A meta-analytic
review of the relationship of class attendance with grades and student characteristics. Review of
Educational Research, 80(2), 272–295.
Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S. (2014). Current state and future
trends: A citation network analysis of the learning analytics field. In Proceedings of the 4th
International Conference on Learning Analytics and Knowledge (pp. 231–240).
Dunlosky, J., & Rawson, K. A. (2015). Practice tests, spaced practice, and successive relearn-
ing: Tips for classroom use and for guiding students’ learning. Scholarship of Teaching and
Learning in Psychology, 1(1), 72.
Ertmer, P. A., & Newby, T. J. (1993). Behaviorism, cognitivism, constructivism: Comparing
critical features from an instructional design perspective. Performance Improvement Quarterly,
6(4), 50–72.
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International
Journal of Technology Enhanced Learning, 4(5–6), 304–317.
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about
learning. TechTrends, 59(1), 64–71.
Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education:
A review of the literature. Computers & Education, 57(4), 2333–2351.
Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for
learning analytics. Journal of Educational Technology & Society, 15(3), 42–57.
Hadwin, A. F., Nesbit, J. C., Jamieson-Noel, D., Code, J., & Winne, P. H. (2007). Examining trace
data to explore self-regulated learning. Metacognition and Learning, 2(2–3), 107–124.
Ifenthaler, D. (2017). Learning analytics design. In L. Lin & M. Spector (Eds.), The sciences of learn-
ing and instructional design: Constructive articulation between communities (pp. 202–211).
New York, NY: Routledge.
Joksimović, S., Gašević, D., Loughin, T. M., Kovanović, V., & Hatala, M. (2015). Learning at
distance: Effects of interaction traces on academic achievement. Computers & Education, 87,
204–217.
Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to
unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74–85.
Junco, R., & Clem, C. (2015). Predicting course outcomes with digital textbook usage data. The
Internet and Higher Education, 27, 54–63.
Kim, D., Park, Y., Yoon, M., & Jo, I. H. (2016). Toward evidence-based learning analytics: Using
proxy variables to improve asynchronous online discussion environments. The Internet and
Higher Education, 30, 30–43.
24 J. Wong et al.
Kim, K., & Moon, N. (2017). A model for collecting and analyzing action data in a learning process
based on activity theory. Soft Computing, 22, 6671–6681.
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies
predict learner behavior and goal attainment in Massive Open Online Courses. Computers &
Knight, S., & Buckingham Shum, S. (2017). Theory and learning analytics. In C. Lang, G. Siemens,
A. F. Wise, & D. Gašević (Eds.), The handbook of learning analytics (pp. 17–22). Alberta,
Canada: Society for Learning Analytics Research (SoLAR).
Liu, S., & d’Aquin, M. (2017). Unsupervised learning for understanding student achievement in
a distance learning setting. In Global Engineering Education Conference (EDUCON), 2017
IEEE (pp. 1373–1377).
Lonn, S., Aguilar, S. J., & Teasley, S. D. (2015). Investigating student motivation in the context
of a learning analytics intervention during a summer bridge program. Computers in Human
Behavior, 47, 90–97.
Marbouti, F., Diefes-Dux, H. A., & Madhavan, K. (2016). Models for early prediction of at-risk
students in a course using standards-based grading. Computers & Education, 103, 1–15.
Mayer, R. E. (2011). Applying the science of learning. Upper Saddle River, NJ: Pearson.
Murphy, P. K., & Knight, S. L. (2016). Exploring a century of advancements in the science of
learning. Review of Research in Education, 40(1), 402–456.
Na, K. S., & Tasir, Z. (2017). A systematic review of learning analytics intervention contributing
to student success in online learning. In International Conference on Learning and Teaching in
Computing and Engineering (LaTICE) 2017 (pp. 62–68).
Ormrod, J. E. (1999). Human learning (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
Phillips, D. C. (2014). Research in the hard sciences, and in very hard “softer” domains.
Educational Researcher, 43(1), 9–11.
Romero-Zaldivar, V. A., Pardo, A., Burgos, D., & Kloos, C. D. (2012). Monitoring student progress
using virtual appliances: A case study. Computers & Education, 58(4), 1058–1067.
Rowe, E., Asbell-Clarke, J., Baker, R. S., Eagle, M., Hicks, A. G., Barnes, T. M., … Edwards, T.
(2017). Assessing implicit science learning in digital games. Computers in Human Behavior,
76, 617–630.
Schunk, D. H. (2012). Learning theories: An educational perspective (6th. ed.). Upper Saddle
River, NJ: Merrill/Prentice Hall.
SCImago. (2007). SJR—SCImago Journal & Country Rank. Retrieved July 21, 2015, from http://
www.scimagojr.com.
Sedrakyan, G., Snoeck, M., & de Weerdt, J. (2014). Process mining analysis of conceptual model-
ing behavior of novices–empirical study using JMermaid modeling and experimental logging
environment. Computers in Human Behavior, 41, 486–503.
Siadaty, M., Gašević, D., & Hatala, M. (2016). Associations between technological scaffolding
and micro-level processes of self-regulated learning: A workplace study. Computers in Human
Behavior, 55, 1007–1019.
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. Educause
Review, 48(5), 31–40.
Skinner, B. F. (1977). Why I am not a cognitive psychologist. Behavior, 5(2), 1–10.
Sweller, J. (2011). Cognitive load theory. In Psychology of learning and motivation (Vol. 55,
pp. 37–76). San Diego, CA: Academic Press.
Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015). Time will tell: The role of mobile
learning analytics in self-regulated learning. Computers & Education, 89, 53–74.
Tan, J. P. L., Yang, S., Koh, E., & Jonathan, C. (2016). Fostering twenty-first century literacies
through a collaborative critical reading and learning analytics environment: user-perceived
benefits and problematics. In Proceedings of the 6th International Conference on Learning
Analytics & Knowledge (pp. 430–434).
Thurlings, M., Vermeulen, M., Bastiaens, T., & Stijnen, S. (2013). Understanding feedback:
A learning theory perspective. Educational Research Review, 9, 1–15.
Tlili, A., Essalmi, F., Jemni, M., & Chen, N. S. (2016). Role of personality in computer based
learning. Computers in Human Behavior, 64, 805–813.
Vygotsky, L. (1978). Mind in society: The development of higher psychological processes.

Cambridge, MA: Harvard University Press.
Webb, G. (1997). Deconstructing deep and surface: Towards a critique of phenomenography.
Higher Education, 33(2), 195–212.
Wise, A. F., & Schwarz, B. B. (2017). Visions of CSCL: Eight provocations for the future of the
field. International Journal of Computer-Supported Collaborative Learning, 12(4), 423–467.
Yassine, S., Kadry, S., & Sicilia, M. A. (2016). A framework for learning analytics in moodle for
assessing course outcomes. In Global Engineering Education Conference (EDUCON), 2016
IEEE (pp. 261–266).
You, J. W. (2016). Identifying significant indicators using LMS data to predict course achievement
in online learning. The Internet and Higher Education, 29, 23–30.
Zhao, Y., Davis, D., Chen, G., Lofi, C., Hauff, C., & Houben, G. J. (2017). Certificate achievement
unlocked: how does MOOC learners’ behaviour change? In Adjunct Publication of the 25th
Conference on User Modeling, Adaptation and Personalization (pp. 83–88).
Zhuhadar, L., Yang, R., & Lytras, M. D. (2013). The impact of Social Multimedia Systems on
cyberlearners. Computers in Human Behavior, 29(2), 378–385.
Zimmerman, B. J., & Campillo, M. (2003). Motivating self-regulated problem solvers. In J. E.
Davidson & R. J. Sternberg (Eds.), The psychology of problem solving (pp. 233–262).
Cambridge, UK: Cambridge University Press.
Mohammad Khalil is a researcher at the centre for the science of learning and technology at
University of Bergen. Mohammad has worked as a postdoctoral candidate at Delft University of
Technology. He has a doctoral degree in computer science from Graz University of Technology.
His PhD dissertation was about learning analytics in Massive Open Online Courses (MOOCs). At
the moment, his research is strongly related to MOOCs, online learning, and learning analytics.
For publications as well as further research activities, visit his website: http://mohdkhalil.word-
press.com.
Chapter 2
Utilising Learning Analytics for Study
Success: Reflections on Current Empirical
Findings
Dirk Ifenthaler, Dana-Kristin Mah, and Jane Yin-Kim Yau
1 Introduction
Study success includes the successful completion of a first degree in higher educa-
tion to the largest extent and the successful completion of individual learning tasks
to the smallest extent (Sarrico, 2018). The essence here is to capture any positive
learning satisfaction, improvement or experience during learning. As some of the
more common and broader definitions of study success include terms such as reten-
tion, persistence and graduation rate, the opposing terms include withdrawal, drop-
out, noncompletion, attrition and failure (Mah, 2016).
Learning analytics (LA) show promise to enhance study success in higher educa-
tion (Pistilli & Arnold, 2010). For example, students often enter higher education
academically unprepared and with unrealistic perceptions and expectations of aca-
demic competencies for their studies. Both the inability to cope with academic
requirements and unrealistic perceptions and expectations of university life, in par-
ticular with regard to academic competencies, are important factors for leaving the
institution prior to degree completion (Mah, 2016). Yet Sclater and Mullan (2017)
reported on the difficulty to isolate the influence of the use of LA, as often they are
used in addition to wider initiatives to improve student retention and academic
achievement.
However, the success of LA in improving higher education students’ learning
has yet to be proven systematically and based on rigorous empirical findings.
D. Ifenthaler (*)
University of Mannheim, Mannheim, BW, Germany
Curtin University, Perth, WA, Australia
D.-K. Mah · J. Y.-K. Yau

https://doi.org/10.1007/978-3-319-64792-0_2
28 D. Ifenthaler et al.
Only a few works have tried to address this but limited evidence is shown (Suchithra,
Vaidhehi, & Iyer, 2015). This chapter aims to form a critical reflection on empirical
evidence demonstrating how LA have been successful in facilitating study success
in continuation and completion of students’ university courses.
2 C
urrent Empirical Findings on Learning Analytics
and Study Success
There have been a number of research efforts, some of which focussed on various
LA tools and some focussed on practices and policies relating to learning analytics
system adoption at school level, higher education and national level. Still, signifi-
cant evidence on the successful usage of LA for improving students’ learning in
higher education is lacking for large-scale adoption of LA (Buckingham Shum &
McKay, 2018).
An extensive systematic literature review of empirical evidence on the benefits of
LA as well as the related field of educational data mining (EDM) was conducted by
Papamitsiou and Economides (2014). They classified the findings from case studies
focussing on student behaviour modelling, prediction of performance, increase self-
reflection and self-awareness, prediction of dropout as well as retention. Their find-
ings suggest that large volumes of educational data are available and that pre-existing
algorithmic methods are applied. Further, LA enable the development of precise
learner models for guiding adaptive and personalised interventions. Additional
strengths of LA include the identification of critical instances of learning, learning
strategies, navigation behaviours and patterns of learning (Papamitsiou &
Economides, 2014). Another related systematic review on LA was conducted by
Kilis and Gülbahar (2016). They conclude from the reviewed studies that log data
of student’s behaviour needs to be enriched with additional information (e.g. actual
time spent for learning, semantic-rich information) for better supporting learning
processes. Hence, LA for supporting study success requires rich data about stu-
dents’ efforts and performance as well as detailed information about psychological,
behavioural and emotional states.
As further research is conducted in the field of LA, the overriding research ques-
tion of this chapter remains: Is it possible to identify a link between LA and related
prevention and intervention measures to increase study success in international
empirical studies?
2.1 Research Methodology
Our critical reflection on empirical evidence linking study success and LA was con-
ducted in 2017. Literature review contributions to LA were first analysed, followed
by individual experimental case studies containing research findings and empirical
conclusions as well as evidence. Search terms included “learning analytics” in
2 Utilising Learning Analytics for Study Success: Reflections on Current Empirical… 29
combination with “study success”, “retention”, “dropout”, “prevention”, “course

completion” and “attrition”. We searched international databases including Google
Scholar, ACM Digital Library, Web of Science, ScienceDirect, ERIC and
DBLP. Additionally, we searched articles published in journals such as Journal of
Learning Analytics, Computers in Human Behaviour, Computers & Education,
Australasian Journal of Educational Technology and British Journal of Educational
Technology. 6220 articles were located, and after duplicated papers were removed,
3163 were remaining. All of these abstracts of papers were screened and were
included in our critical reflection on empirical evidence according to our inclusion
criteria as follows: (a) were situated in the higher education context, (b) were pub-
lished between 2013 and 2017, (c) were published in English, (d) presented either
qualitative or quantitative analyses and findings and (e) were peer-reviewed. The
number of key studies identified was 374 (in the first round) then limited to 46 (due
to substantiality of empirical evidence); an elaboration of the identified empirical
evidence from the limited studies will form our upcoming work. In this paper, we
provide a general overview of the identified empirical evidence.
2.2 Results of the Critical Reflection
This section is divided into (1) positive evidence on the use of LA to support study
success, (2) insufficient evidence on the use of LA to support study success and (3)
link between LA and intervention measures to facilitate study success.
2.2.1 P
ositive Evidence on the Use of Learning Analytics to Support
Study Success
Some of the positive empirical evidence presented by Sclater and Mullan (2017)
include the following: At the University of Nebraska-Lincoln after LA was adopted,
their 4-year graduation rate increased by 3.8% in 4 years. At Columbus State
University College, Georgia, course completion rates rose 4.2%. Similarly, at the
University of New England, South Wales, the dropout rate decreased from 18% to
12%. Control group studies yield the following results: there was a significant
improvement in final grade (6%) at Marist College; at Strayer University, Virginia,
the identified at-risk students were given intervention and resulted in 5% increase in
attendance, 12% increase in passing and 8% decrease in dropout. At the University
of South Australia, 549 of 730 at-risk students were contacted; 66% passed with
average GPA of 4.29. Fifty-two percent of un-contacted at-risk students passed with
average GPA of 3.14. At Purdue University, Indiana, it was found that using the
university’s predictive analytics system (Course Signal), there were consistently
higher levels of Bs and Cs grades obtained than Ds and Fs grades in two semesters
of courses. A 15% increase in recruitment and a 15% increase in retention as a result
was reported (Tickle, 2015).
We also identified positive evidence on the use of LA to support study success

through the use of assessment data, engagement indicators, online platform data and
the use of personalised feedback, as follows.
Predictive Analytics Using Assessment Data It was found on average that there was
a 95% probability if a student had not submitted their assignment and that they will
not finish the course (Hlosta, Zdrahal, & Zendulka, 2017). Here, assessment descrip-
tion referred to (1) students’ demographic information (e.g. age, gender, etc.), (2)
students’ interactions with the VLE system, (3) information about students’ date of
registration and (4) a flag indicating student assignment submission. This informa-
tion is used to extract learning patterns from the students where their progress of the
course can be predicted. The assessment of the first assignment provides a critical
indicator for the remainder of the course. The conducted experiments showed this
method can successfully predict at-risk students.
Predictive Analytics Using Engagement Indicators Information about students’

behaviour that is made available during the course can be used to predict the
decrease of engagement indicators at the end of a learning sequence. Three main
tasks that students conducted in a MOOC environment were able to yield good
results in the prediction if there would be a decrease in engagement in the course as
signalled by engagement indicators (Bote-Lorenzo & Gomez-Sanchez, 2017). The
authors found that three engagement indicators derived from tasks being carried out
in a MOOC were very successful in predicting study success—watching lectures,
solving finger exercises and submitting assignments. It was suggested that their
predictive method would be useful to detect disengaging students in digital learning
environments.
Predictive Analytics Using Digital Platform Data Self-report and digital learning
system information (i.e. trace data) can be used to identify students at risk and in
need of support as demonstrated by a study conducted by Manai, Yamada, and
Thorn (2016). For example, some self-report survey items measure non-cognitive
factors such as indicative predictors of student outcomes allowing one to inform
actionable insights with only a few items’ data. Certain formulas were used in their
study such as (1) if students showing higher levels of fixed mindset and to be at risk,
a growth mindset is promoted to them by engaging them in growth mindset activi-
ties and also giving feedback to students that establishes high standards and assur-
ing that the student is capable of meeting them; (2) if students showing higher levels
of belonging uncertainty, group activities that facilitate building a learning commu-
nity for all students in the classroom are provided; and (3) if students showing low
levels of math conceptual knowledge, scaffolding for students is provided during
the use of the online learning platform. Similarly, Robinson, Yeomans, Reich,
Hulleman, and Gehlbach (2016) utilised natural language processing, and their
experiment showed promising predictions from unstructured text which students
would successfully complete an online course.
Personalised Feedback Leading to Learning Gains Feedback can be tailored based

on the student’s affective state in the intelligent support system. The affective state
is derived from speech and interaction, which is then used to determine the type of
appropriate feedback and its presentation (interruptive or non-interruptive)
(Grawemeyer et al., 2016). Their results showed that students using the environment
were less bored and less off-task showing that students had higher learning gains
and there is a potential and positive impact affect-aware intelligent support.
2.2.2 Insufficient Evidence on the Use of LA to Support Study Success
The most recent review of learning analytics published in 2017 from Sclater and
Mullan (2017) described the use of LA to be most concentrated in the United States,
Australia and England; most institutional initiatives on LA are at an early stage and
lacking sufficient time to find concrete empirical evidence of their effectiveness
(Ifenthaler, 2017a). However, some of the most successful projects were in the US
for-profit sector, and these findings are unpublished. In the review conducted by
Ferguson et al. (2016), the state of the art in the implementation of LA for education
and training in Europe, United States and Australia was presented which is still
scarce. Specifically, it was noticed that there are relatively scarce information on
whether LA improves teaching and learners’ support at universities, and problems
with the evidence include lack of geographical spread, gaps in our knowledge
(informal learning, workplace learning, ethical practice, lack of negative evidence),
little evaluation of commercially available tools and lack of attention to the learning
analytics cycle (Ferguson & Clow, 2017).
Threats deriving from LA include ethical issues, data privacy and danger of over-
analysis, which do not bring any benefits and overconsumption of resources (Slade
& Prinsloo, 2013). Accordingly, several principles for privacy and ethics in LA have
been proposed. They highlight the active role of students in their learning process,
the temporary character of data, the incompleteness of data on which learning ana-
lytics are executed, the transparency regarding data use as well as the purpose, anal-
yses, access, control and ownership of the data (Ifenthaler & Schumacher, 2016;
West, Huijser, & Heath, 2016). In order to overcome concerns over privacy issues
while adopting LA, an eight-point checklist based on expert workshops has been
developed that can be applied by teachers, researchers, policymakers and institu-
tional managers to facilitate a trusted implementation of LA (Drachsler & Greller,
2016). The DELICATE checklist focusses on Determination, Explain, Legitimate,
Involve, Consent, Anonymise, Technical aspects and External partners. However,
empirical evidence towards student perceptions of privacy principles related to
learning analytics is still in its infancy and requires further investigation and best
practice examples (Ifenthaler & Tracey, 2016).
Ferguson et al. (2016) documented a number of tools that have been imple-
mented for education and training and raised a number of important points—(a)
most LA tools are provided on the supply side from education institutions and not
on the demand side required by students and learners; (b) data visualisation tools are
available, however do not provide much help in advising steps that learners should
take in order to advance their studies/increase study success; and (c) especially evi-
dence is lacking on formal validation and evaluation of LA tools of the impact and
success, although national policies in some European countries such as Denmark,
the Netherlands and Norway and universities such as Nottingham Trent University,
Open University UK and Dublin City University have commenced to create an
infrastructure to support and enable policies of utilisation of LA or implementation/
incorporation of LA systems. Hence, the evidence on successful implementation
and institution-wide practice is still limited (Buckingham Shum & McKay, 2018).
Current policies for learning and teaching practices include developing LA that are
supported through pedagogical models and accepted assessment and feedback prac-
tices. It is further suggested that policies for quality assessment and assurance prac-
tices include the development of robust quality assurance processes to ensure the
validity and reliability of LA tools as well as developing evaluation benchmarks for
LA tools (Ferguson et al., 2016).
2.2.3 L
ink between Learning Analytics and Intervention Measures
to Facilitate Study Success
Different LA methods are used to predict student dropout such as predictive models
and student engagement with the virtual learning environment (VLE) (more reliable
indicator than gender, race and income) (Carvalho da Silva, Hobbs, & Graf, 2014;
Ifenthaler & Widanapathirana, 2014). Some of the significant predictors of dropout
used in these methods can be indicated and include the following: posting behaviour
in forums, social network behaviour (Yang, Sinha, Adamson, & Rose, 2013), per-
centage of activities delivered, average grades, percentage of resources viewed and
attendance (85% accuracy of at-risk student identification) (Carvalho da Silva et al.,
2014). Similarly, different factors are used at Nottingham Trent University to signal
student engagement: library use, card swipes into buildings, VLE use and electronic
submission of coursework, analyses the progression and attainment in particular
groups (Tickle, 2015). An example technique is as follows: if there is no student
engagement for 2 weeks, tutors will get an automatic email notification, and they
are encouraged to open up a dialogue with the at-risk student. Their LA system
intends to help increase not only study retention but also to increase study perfor-
mance. Prevention measures include pedagogical monitoring. The timeliness of the
institution or university’s intervention is very important including noticing signs of
trouble and responding immediately to these (Tickle, 2015). A question concerning
ethics may be “do students want an algorithm applied to their data to show they are
at risk of dropping out?” causing intervention from respective tutors to take place
(West et al., 2016).
LA are often discussed and linked with regard to self-regulated learning. Self-
regulated learning can be seen as a cyclical process, starting with a forethought
phase including task analysis, goal setting, planning and motivational aspects
(Ifenthaler, 2012). The actual learning occurs in the performance phase, i.e. focus-
sing, applying task strategies, self-instruction and self-monitoring. The last phase
contains self-reflection, as learners evaluate their outcomes versus their prior set
goals. To close the loop, results from the third phase will influence future learning
activities (Zimmerman, 2002). Current findings show that self-regulated learning
capabilities, especially revision, coherence, concentration and goal setting, are
related to students’ expected support of LA systems (Gašević, Dawson, & Siemens,
2015; Schumacher & Ifenthaler, 2018b). For example, LA facilitate students through
adaptive and personalised recommendations to better plan their learning towards
specific goals (McLoughlin & Lee, 2010; Schumacher & Ifenthaler, 2018a). Other
findings show that many LA systems focus on visualisations and outline descriptive
information, such as time spent online, the progress towards the completion of a
course and comparisons with other students (Verbert, Manouselis, Drachsler, &
Duval, 2012). Such LA features help in terms of monitoring. However, to plan
upcoming learning activities or to adapt current strategies, further recommendations
based on dispositions of students, previous learning behaviour, self-assessment
results and learning goals are important (McLoughlin & Lee, 2010; Schumacher &
Ifenthaler, 2018b). In sum, students may benefit from LA through personalised and
adaptive support of their learning journey; however, further longitudinal and large-
scale evidence is required to demonstrate the effectiveness of LA.
3 Conclusion
This critical reflection of current empirical findings indicates that a wider adoption
of LA systems is needed as well as work towards standards for LA which can be
integrated into any learning environment providing reliable at-risk student predic-
tion as well as personalised prevention and intervention strategies for supporting
study success. In particular, personalised learning environments are increasingly
demanded and valued in higher education institutions to create a tailored learning
package optimised for each individual learner based on their personal profile which
could contain information such as their geo-social demographic backgrounds, their
previous qualifications, how they engaged in the recruitment journey, their learning
activities and strategies, affective states and individual dispositions, as well as track-
ing information on their searches and interactions with digital learning platforms
(Ifenthaler, 2015). Still, more work on ethical and privacy guidelines supporting LA
is required to support the implementation at higher education institutions (Ifenthaler
& Tracey, 2016), and there are still many open questions how LA can support learn-
ing, teaching as well as the design of learning environments (Ifenthaler, 2017b;
Ifenthaler, Gibson, & Dobozy, 2018). Another field requiring rigorous empirical
research and precise theoretical foundations is the link between data analytics and
assessment (Ifenthaler, Greiff, & Gibson, 2018). Further, as LA are of growing
interest for higher education institutions, it is important to understand students’
expectations of LA features (Schumacher & Ifenthaler, 2018a) to be able to align
them with learning theory and technical possibilities before implementing them
(Marzouk et al., 2016). As higher education institutions are moving towards adop-
tion of LA systems, change management strategies and questions of capabilities are
key for successful implementations (Ifenthaler, 2017a). The preliminary findings
obtained in this critical reflection suggest that there are a considerable number of
sophisticated LA tools which utilise effective techniques in predicting study success
and at-risk students of dropping out.
Limitations of this study include the difficulty in comparing results of different
studies as various techniques and algorithms, research questions and aims were
used. Although much empirical evidence is documented in these papers, many stud-
ies are still works-in-progress, experimental studies and at very small scale. The
papers discuss how LA can work to predict study success, and the steps following
this to the discussions with the students and the approaches that teachers can take to
address to at-risk students are under-documented. The questions raised concerning
this are, for example: (a) Will students be able to respond positively and proactively
when informed that their learning progress is hindered or inactivated? (b) Will
instructors be able to influence the at-risk students positively so that they will re-
engage with the studies? (c) In addition, ethical dimensions regarding descriptive,
predictive and prescriptive learning analytics need to be addressed with further
empirical studies and linked to study success indicators.
However, evidence on a large scale to support the effectiveness of LA actually
retaining students onto courses are still lacking, and we are currently examining the
remainder of the key studies thoroughly to obtain a clearer and more exact picture
of how much empirical evidence there is that LA can support study success. Methods
and advice also can be used as a guide in helping students to stay on the course after
they have been identified as at-risk students. One suggestion is to leverage existing
learning theory by clearly designing studies with clear theoretical frameworks and
connect LA research with decades of previous research in education. Further docu-
mented evidence on LA include that LA cannot be used as a one-size-fits-all
approach, i.e. requiring personalisation, customisation and adaption (Gašević,
Dawson, Rogers, & Gašević, 2016; Ifenthaler, 2015).
Our future work also includes locating learning theories onto LA (which is cur-
rently lacking)—there is missing literature on variables as key indicators of interac-
tion and study success in digital learning environments. Hence, while the field of
learning analytics produces ever more diverse perspectives, solutions and defini-
tions, we expect analytics for learning to form a novel approach for guiding the
implementation of data- and analytics-driven educational support systems based on
thorough educational and psychological models of learning as well as producing
rigorous empirical research with a specific focus on the processes of learning and
the complex interactions and idiosyncrasies within learning environments.
Acknowledgements The authors acknowledge the financial support by the Federal Ministry of
Education and Research of Germany (BMBF, project number 16DHL1038).
References
Bote-Lorenzo, M., & Gomez-Sanchez, E. (2017). Predicting the decrease of engagement indica-
tors in a MOOC. In International Conference on Learning Analytics & Knowledge, Vancouver,
Canada.
Buckingham Shum, S., & McKay, T. A. (2018). Architecting for learning analytics. Innovating for
sustainable impact. Educause Review, 53(2), 25–37.
Carvalho da Silva, J., Hobbs, D., & Graf, S. (2014). Integrating an at-risk student model into learn-
ing management systems. In Nuevas Ideas en Informatica Educativa.
Drachsler, H., & Greller, W. (2016). Privacy and analytics - It's a DELICATE issue. A checklist for
trusted learning analytics. Paper presented at the Sixth International Conference on Learning
Analytics & Knowledge, Edinburgh, UK.
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., … Vuorikari, R.
(2016). Research evidence on the use of learning analytics – Implications for education policy.
In R. Vuorikari & J. Castano Munoz (Eds.). Joint Research Centre Science for Policy Report.
Ferguson, R. & Clow, D. (2017). Where is the evidence? A call to action for learning analytics. In
International Conference on Learning Analytics & Knowledge, Vancouver, Canada.
Gašević, D., Dawson, S., Rogers, T., & Gašević, D. (2016). Learning analytics should not promote
one size fits all: The effects of instructional conditions in predicting academic success. Internet
and Higher Education, 28, 68–84.
learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x
Grawemeyer, B., Mavrikis, M., Holmes, W., Gutierrez-Santos, S., Wiedmann, M., & Rummel, N.
(2016). Determination of off-task/on-task behaviours. In Proceedings of the Sixth International
Conference on Learning Analytics & Knowledge, Edinburgh, UK.
Hlosta, M., Zdrahal, Z., & Zendulka, J. (2017). Ouroboros: Early identification of at-risk students
without models based on legacy data. In Proceedings of the Seventh International Conference
on Learning Analytics & Knowledge, Vancouver, Canada.
Ifenthaler, D. (2012). Determining the effectiveness of prompts for self-regulated learning in
problem-solving scenarios. Journal of Educational Technology & Society, 15(1), 38–52.
Ifenthaler, D. (2015). Learning analytics. In J. M. Spector (Ed.), The SAGE encyclopedia of edu-
cational technology (Vol. 2, pp. 447–451). Thousand Oaks, CA: Sage.
Ifenthaler, D. (2017a). Are higher education institutions prepared for learning analytics?
TechTrends, 61(4), 366–371. https://doi.org/10.1007/s11528-016-0154-0
Ifenthaler, D. (2017b). Learning analytics design. In L. Lin & J. M. Spector (Eds.), The sciences
of learning and instructional design. Constructive articulation between communities (pp. 202–
211). New York, NY: Routledge.
Ifenthaler, D., Gibson, D. C., & Dobozy, E. (2018). Informing learning design through analyt-
ics: Applying network graph analysis. Australasian Journal of Educational Technology, 34(2),
117–132. https://doi.org/10.14742/ajet.3767
Ifenthaler, D., Greiff, S., & Gibson, D. C. (2018). Making use of data for assessments: Harnessing
analytics and data science. In J. Voogt, G. Knezek, R. Christensen, & K.-W. Lai (Eds.),
International handbook of IT in primary and secondary education (2nd ed., pp. 649–663).
New York, NY: Springer.
Ifenthaler, D., & Schumacher, C. (2016). Student perceptions of privacy principles for learning
analytics. Educational Technology Research and Development, 64(5), 923–938. https://doi.
org/10.1007/s11423-016-9477-y
Ifenthaler, D., & Tracey, M. W. (2016). Exploring the relationship of ethics and privacy in learn-
ing analytics and design: Implications for the field of educational technology. Educational
Technology Research and Development, 64(5), 877–880. https://doi.org/10.1007/s11423-016-
9480-3
Ifenthaler, D., & Widanapathirana, C. (2014). Development and validation of a learning analyt-
ics framework: Two case studies using support vector machines. Technology, Knowledge and
Learning, 19(1-2), 221–240. https://doi.org/10.1007/s10758-014-9226-4
Kilis, S., & Gülbahar, Y. (2016). Learning analytics in distance education: A systematic litera-
ture review. Paper presented at the 9th European Distance and E-learning Network (EDEN)
Research Workshop, Oldenburg, Germany.
Mah, D.-K. (2016). Learning analytics and digital badges: Potential impact on student reten-
tion in higher education. Technology, Knowledge and Learning, 21(3), 285–305. https://doi.
org/10.1007/s10758-016-9286-8
Manai, O., Yamada, H., & Thorn, C. (2016). Real-time indicators and targeted supports: Using
online platform data to accelerate student learning. In International Conference on Learning
Analytics & Knowledge, UK, 2016.
Marzouk, Z., Rakovic, M., Liaqat, A., Vytasek, J., Samadi, D., Stewart-Alonso, J., … Nesbit, J. C.
(2016). What if learning analytics were based on learning science? Australasian Journal of
Educational Technology, 32(6), 1–18. https://doi.org/10.14742/ajet.3058
McLoughlin, C., & Lee, M. J. W. (2010). Personalized and self regulated learning in the Web
2.0 era: International exemplars of innovative pedagogy using social software. Australasian
Journal of Educational Technology, 26(1), 28–43.
Papamitsiou, Z., & Economides, A. (2014). Learning analytics and educational data mining in
practice: A systematic literature review of empirical evidence. Educational Technology &
Society, 17(4), 49–64.
Pistilli, M. D., & Arnold, K. E. (2010). Purdue signals: Mining real-time academic data to enhance
student success. About Campus: Enriching the Student Learning Experience, 15(3), 22–24.
Robinson, C., Yeomans, M., Reich, J., Hulleman, C., & Gehlbach, H. (2016). Forecasting student
achievement in MOOCs with natural language processing. In International Conference on
Learning Analytics & Knowledge, UK.
Sarrico, C. S. (2018). Completion and retention in higher education. In J. C. Shin & P. Teixeira
(Eds.), Encyclopedia of international higher education systems and institutions. Dordrecht,
The Netherlands: Springer.
Schumacher, C., & Ifenthaler, D. (2018a). Features students really expect from learning analytics.
Computers in Human Behavior, 78, 397–407. https://doi.org/10.1016/j.chb.2017.06.030
Schumacher, C., & Ifenthaler, D. (2018b). The importance of students’ motivational dispositions
for designing learning analytics. Journal of Computing in Higher Education, 30, 599. https://
doi.org/10.1007/s12528-018-9188-y
Sclater, N., & Mullan, J. (2017). Learning analytics and student success – Assessing the evidence.
Bristol, UK: JISC.
Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American
Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366
Suchithra, R., Vaidhehi, V., & Iyer, N. E. (2015). Survey of learning analytics based on pur-
pose and techniques for improving student performance. International Journal of Computer
Applications, 111(1), 22–26.
Tickle, L. (2015). How universities are using data to stop students dropping out. Guardian. Retrieved
from https://www.theguardian.com/guardian-professional/2015/jun/30/how-universities-are-
using-data-to-stop-students-dropping-out
Verbert, K., Manouselis, N., Drachsler, H., & Duval, E. (2012). Dataset-driven research to support
learning and knowledge analytics. Educational Technology & Society, 15(3), 133–148.
West, D., Huijser, H., & Heath, D. (2016). Putting an ethical lens on learning analytics. Educational
Technology Research and Development, 64(5), 903–922. https://doi.org/10.1007/s11423-
016-9464-3
Yang, D., Sinha, T., Adamson, D., & Rose, C. (2013). “Turn on, tune in, drop out”: Anticipating
student dropouts in massive open online courses. In Neural Information Processing Systems.
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice,
41(2), 64–70.
Chapter 3
Supporting Stakeholders with Learning
Analytics to Increase Study Success
A Case Study
Miriam Hommel, Armin Egetenmeier, and Ulrike Maier
1 Introduction
More and more students are entering the universities with different types of univer-
sity entrance qualifications (UEQ). This implies a high degree of first-year diversity,
heterogeneous university preparation, and a wide range of skills, especially in math-
ematics. Heterogeneity has, therefore, become a ubiquitous issue and challenge in
higher education institutes (HEIs) (Bebermeier & Nussbeck, 2014; Reinmann, 2015).
It has been known for some time that underachievement is a major cause of drop-
out at HEIs in Germany (Heublein, Richter, Schmelzer, & Sommer, 2012, 2014;
Heublein & Wolter, 2011). Heublein et al. (2017a) recently published results from a
nationwide survey investigating motives and causes of dropout in German HEIs.
They found that especially in the study entry phase (first year) at Universities of
Applied Sciences (UAS) in the federal state of Baden-Wuerttemberg, almost 50%
of the students decide to quit university without a degree (Heublein et al., 2017b,
p. 281). Based on their research, Heublein et al. (2017a) formulated recommenda-
tions for various fields of action to lower dropout rates. Suggestions are, e.g., a
comprehensive range of support measures and the introduction of a control system
that enables HEIs to take preventive or intervention measures (Heublein et al.,
2017a, pp. 20–21).
Most universities already offer support such as tutorials or preparatory courses
(Biehler, Hochmuth, Fischer, & Wassong, 2011; Hoppenbrock, Biehler, Hochmuth,
& Rück, 2016). In order to strengthen these supporting structures with regard to the
challenges in the study entry phase, the German Federal Government has initiated a
nationwide funding program for HEIs (Deutsches Zentrum für Luft- und Raumfahrt
e. V. [DLR], 2014; DLR, 2015).
M. Hommel · A. Egetenmeier · U. Maier (*)

Study Support Center, Aalen University of Applied Sciences, Aalen, Germany

https://doi.org/10.1007/978-3-319-64792-0_3
38 M. Hommel et al.
With the proliferation of many different supportive structures, there is a growing

interest in analyzing their effectiveness. Due to the governmental funding, reporting
is mandatory. A detailed analysis of the impact of the supporting measures may also
be beneficial for similar projects as their outcomes are important for both institu-
tions and policy-makers. Investigations concerning preparatory courses (prep
courses) can be found in Derr, Hübl, and Podgayetskaya (2015) or Heiss and
Embacher (2016). A study on tutorials is presented in Bebermeier and Nussbeck
(2014). Even though a data-driven approach may bring new insights, mostly quali-
tative methods (e.g., surveys) are used for the analyses. For estimating the effective-
ness of the supportive measures more realistically, there is a necessity to analyze
corresponding data also quantitatively to prevent study-dropouts and to ensure high
teaching quality and study success.
Despite the fact that study success is a key aspect of many higher education stud-
ies, there is no universally accepted definition of “study success.” One reason may
be the different perspectives of “success” by students, lecturers, or society (Erdel,
2010, p. 12). In the absence of a consistent and clear definition of “study success,”
different characteristics are used to include as many relevant aspects as possible.
This begins with a simple but obvious definition, such as “achieving a degree”
(Dell’mour & Landler, 2002, p. 43), to a more advanced version that includes crite-
ria like “time to completion” or “final grade.” More subjective criteria such as
“study satisfaction,” “relevant competences,” or “employability” are also used as
characteristics for “study success” (Rindermann & Oubaid, 1999).
In educational policy in Germany, “study success” is often defined as the oppo-
site of “study failure” (Erdmann & Mauermeister, 2016). In this sense, avoiding
dropping out can be considered as “success” which explains the close link between
research on “study success” and “study failure” (study-dropout). Assuming that the
support measures of a university can influence the (individual) decision to continue
studying, a “student success” is also an “institutional success.”
In our study, we use a working definition of “study success” based on the aca-
demic achievement of students in the first year. This includes the impact of the sup-
port measures on their intended purpose (in our case the improvement of
mathematical skills of freshmen). Dropouts are often caused by performance prob-
lems, and insufficient math skills increase the risk of quitting (Heublein, Hutzsch,
Schreiber, Sommer, & Besuch, 2010, pp. 68–70). Therefore, currently our main
criterion to quantify “study success” is the passing of the first math exam as it marks
an important step toward graduation (compare Sect. 2.2.4). This offers the opportu-
nity to evaluate the effects of supportive actions on study success at an early stage.
1.1 Learning Analytics
In recent years, new technology-enriched methods and digital offerings have been
developed to improve teaching and increase students’ learning outcomes. By using
digital learning material, large amounts of data can be analyzed. In this
3 Supporting Stakeholders with Learning Analytics to Increase Study Success 39
environment, new fields of research such as Educational Data Mining (EDM) and
Learning Analytics (LA) were established. A major difference between both fields
is that while EDM focuses on automation by the computer, the goal of LA is to
empower stakeholders to make decisions based on data rather than relying only
on “gut instinct” or experiences (Long & Siemens, 2011, p. 32). Our contribution
focuses on LA.
Since the first Conference on Learning Analytics and Knowledge (LAK) in 2011,
a commonly accepted definition for LA is “the measurement, collection, analysis
and reporting of data about learners and their contexts, for purposes of understand-
ing and optimizing learning and the environments in which it occurs” (Long &
Siemens, 2011, p. 34). Brown (2011, p. 1) supplements this definition by “The pur-
pose of LA is to observe and understand learning behaviors in order to enable
appropriate interventions.”
1.2 Benefits and Issues of Learning Analytics
A benefit of LA is the ability to promote learners on electronic platforms through

help systems and personalized feedback in their personal learning habits. It offers
the opportunity to track individual learning outcomes and learning progresses of
students (regardless of group size). On the other hand, data can be grouped and
related as desired. Thus, LA can assist teachers in identifying students at risk and
intervene in the learning process in a timely manner with the help of feedback.
In addition, the observation of learning processes enables HEIs to further
develop and improve the quality of teaching in the long term. This can lead to cur-
riculum developments and the introduction of new supportive measures. The use of
data already available at universities can facilitate the evaluation of the effective-
ness of these measures. This can also influence the knowledge on education pro-
cesses and thus lead to an improvement of educational research. “Theoretically, LA
has the potential to dramatically influence existing education models and gain new
insights into what works and what does not work in teaching and learning”
(Siemens, 2012, p. 7).
In contrast to the hype of recent years, there has been a slowdown in the introduc-
tion and use of LA, in particular due to ethical, privacy, and legal concerns (Drachsler
& Greller, 2016). Especially ethical aspects like data protection often remain unad-
dressed by institutional review boards (Willis, Slade, & Prinsloo, 2016). Besides
these aspects, the data itself includes issues. It is still not clear what type of data really
matters to generate incontrovertible evidence about what makes successful learning
(Kuh, Kinzie, Buckley, Bridges, & Hayek, 2006; Macfadyen & Dawson, 2012).
Although technology-enhanced learning is becoming more relevant, most HEIs
(especially in Germany) are still using “traditional” (face-to-face) learning environ-
ments (Romero & Ventura, 2013, p. 16). Nevertheless, data collection mostly takes
place in centralized educational systems like learning management systems (LMS)
or web-based systems (Chatti et al., 2014; Papamitsiou & Economides, 2014).
40 M. Hommel et al.
On closer examination of the study success, “appropriate intervention” is an inte-

gral part of Learning Analytics. In an effective LA cycle, a key step is “‘closing the
loop’ by feeding back this product [e.g., analytics] to learners through one or more
interventions” (Clow, 2012, p. 134). Since the effect of feedback depends on indi-
vidual factors such as prior knowledge or self-efficacy (Kluger & DeNisi, 1996;
Narciss, 2013), the interpretations of LA results require some level of expertise.
Possible misinterpretations may lead to unintended behavior of the students (Corrin
& Barba, 2014). In order to avoid such behavior, it is useful to include pedagogical
considerations (Tempelaar, Rienties, & Nguyen, 2017; Wise, 2014) that can also
increase the quality of the LA application.
Overviews of benefits and issues of LA are given, for example, in Long and
Siemens (2011); Papamitsiou and Economides (2014); Avella, Kebritchi, Nunn, and
Kanai (2016); and Leitner, Khalil, and Ebner (2017).
1.3 Stakeholders of Learning Analytics
The obvious group of users of an LA framework are students and teaching staff.
Brown (2011, p. 1) stresses: “The reports that an LA application generates can be
very helpful for instructors (about student activities and progress), for students
(feedback on their progress), and for administrators (e.g., aggregations of course
and degree completion data).” Thus, to unfold the full potential of an LA frame-
work, it is necessary and lucid to address several stakeholders. Taking into account
politics/government as a superordinate extension, four stakeholder levels can be
identified (Greller & Drachsler, 2012). Embedded in an institutional framework,
these groups can be distributed hierarchically to mega-, macro-, meso-, and
micro-level.
Ifenthaler (2015) presents a matrix that shows how all levels of stakeholders
benefit from LA. Figure 3.1 shows an adaptation of a graphic by Ifenthaler (2015)
illustrating the actors and their influences within the education system and the pos-
sible recipients of an LA support. In contrast to the original figure, the learning
environment includes not only teacher and learner (as main recipients of informa-
tion) but also parts of the LA system. Research within an LA framework seems to
be a good way to self-evaluate (and constantly improve) provided support or reports.
Regarded as an additional group, researchers act across all LA levels and thus can
hardly be associated to a specific stakeholder level (cf. Greller & Drachsler, 2012;
Romero & Ventura, 2013).
In order to support students’ success, various approaches are pursued at HEIs.
This led to many models and tools in the field of LA addressing different stakehold-
ers. Most of the large-scale, systematic implementations of LA are located at
Colleges and HEIs in the United States of America, Canada, and Australia (Sclater,
Peasgood, & Mullan, 2016, A21). One of the best-known implementations in the
field is “Course Signals” (CS) of Purdue University (Arnold & Pistilli, 2012). Based
on a prediction model, the CS system visualizes the students’ chances of success for
Fig. 3.1 Learning Analytics associated with stakeholder levels (inspired by Ifenthaler, 2015)
teachers and learners via “status traffic lights.” Other tools like the “Student Activity
Meter” (SAM) (Govaerts, Duval, Verbert, & Pardo, 2012) support students’ self-
reflection and awareness of, for example, resource usage and expenditure of time.
This type of feedback has been developed for learners and teachers alike.
In Australia, the University of Sydney has developed the “Student Relationship
Engagement System” (SRES) and has deployed it at several Australian universities
(Vigentini et al., 2017). SRES is an educator-driven, web-based LA system support-
ing student success with personalized feedback. To provide students with appropri-
ate information, various types of data (e.g., attendance, grades, personal information)
can be collected, filtered, and analyzed in the system. This pedagogically oriented
system is specifically designed to meet the needs of the teaching staff.
As student advisors contribute to student success, they can also benefit from LA
implementations. For instance, the “Learning dashboard for Insights and Support
during Study Advice” (LISSA) (Charleer, Vande Moere, Klerkx, Verbert, & Laet,
2017) has been developed to provide insight into grade data of students to be coun-
seled. The tool is suitable as a starting point for advisory sessions based on facts
which can stimulate the dialogue between student and advisor.
A growing adoption of LA can also be found in some European countries due to
national education initiatives like JISC in the United Kingdom (cf. Sclater et al.,
2016) or SURFnet in the Netherlands (cf. Baas et al., 2015). These initiatives build
a nationwide collaborative network to address common existing and upcoming
issues in the field. This paves the way for further LA implementations. An overview
of case studies and, in particular, LA applications is presented in a paper by Wong
(2017) which also includes a list of LA implementations sorted by different goals
and uses (e.g., improving student retention, supporting decision-makers, or giving
timely feedback).
42 M. Hommel et al.
Documented case studies in Germany are rare, partly due to strict data protection
legislation. Another reason may be that HEIs in Germany are still in an early stage
of adopting LA, and thus, there are only few LA systems, e.g., to identify at-risk
students (Ifenthaler, Mah, & Yau, 2017). Examples are LeMo (Fortenbacher et al.,
2013) and eLAT (Dyckhoff, Zielke, Bültmann, Chatti, & Schröder, 2012) supporting
teaching staff, or the tool LAPS, formerly S-BEAT (Hinkelmann, Maucher, & Seidl,
2016), which was developed to support student counseling. It is noticeable that the
results mostly still address a small number of stakeholders. This is regrettable
because a limited access to the results may cause disadvantages for stakeholders and
decision-makers involved (Ifenthaler & Widanapathirana, 2014). Due to the differ-
ent interpretation of analytics results, informing more stakeholders (on different lev-
els) can provide various insights and, therefore, cause different interventions.
In the following sections, we describe how we provide information about the
learning process of the first-year students and their success to stakeholders of all LA
levels (compare Fig. 3.1). Section 2 first introduces the Study Support Center (SSC)
of Aalen UAS. Subsequently, its evidence-oriented research (as a basis for LA) with
study design, research questions, and research method is described. Afterward,
some results of our analyses are exemplified. Section 3 presents how our results
affect different levels of stakeholders. Our contribution ends with a short summary
and conclusion (Sect. 4).
2 E
vidence-Oriented Accompanying Research at Aalen
University of Applied Sciences
2.1 Study Support Center
As introduced in Sect. 1, high dropout rates as a result of missing basic mathematical

prerequisites are a challenge at many universities. At Aalen UAS, a university offer-
ing degrees in the fields of technology and economics, the SSC was set up as part of
the quality pact teaching project “AkaMikon” (see Acknowledgments) in 2011 to
support students of all study courses in the introductory phase. The overall objective
of the SSC is to reduce the number of subject-related study-dropouts. With mathe-
matical prep courses, lecture-accompanying tutorials, as well as measures to level the
heterogeneity in the initial mathematical knowledge (e.g., by pilot projects for con-
tinuous learning and extracurricular offers at selected schools), the SSC team pursues
the goal to facilitate the study entry for the new students (Nagengast, Hommel, &
Löffler, 2013; Nagengast, Hommel, Maier, Egetenmeier, & Löffler, 2017). In addi-
tion to these support measures in traditional classroom teaching, digital learning
environments, e.g., a mathematics online course, are being developed to complement
traditional supportive measures (Krieg, Egetenmeier, Maier, & Löffler, 2017).
Figure 3.2 illustrates the time frame of the SSC support. A three-stage test con-
cept (pretest at the first prep course day, posttest at the end of the prep course, fol-
low-up test 4–6 weeks after the end of the prep course) forms the basis for our
research on the impact of the prep course.
Fig. 3.2 Time frame of SSC support measures
One goal of the SSC is to ensure high teaching quality, which for us means that
students are able to follow the lecture and that they are motivated (ideally intrinsi-
cally) to work independently and to internalize the content. Consequently, students
should not only learn for the exam but also be able to recall the content of the lecture
even at a later point in time.
Concerning the SSC support, a prerequisite for high teaching quality is a scien-
tific background (mostly mathematics) of its members, which in particular ensures
their professional competence. In addition, each team member has completed train-
ing in the field of didactics and gained experience in teaching underperforming
students for several semesters.
The quality of lessons is usually subjective and difficult to quantify. Therefore, it
seems more appropriate to agree on effective teaching methods. The SSC prefers
teaching methods focusing on practicing in order to encourage the students to
actively engage with the mathematical content. Hence, activating exercises are inte-
grated into the lecture part of the prep course to keep the students’ attention high
and to motivate them to practice and apply the course contents directly as suggested
by Heublein et al. (2017a, 2017b). The prep course groups usually have a size of
50–70 students and are arranged according to study courses to encourage fellow
students to get to know each other and to form study groups from the beginning.
After the lecture part of the prep course, tutorials are offered where each group is
conducted by specially trained students of higher semesters from the respective
study course. This and the comparatively small group size lowers the threshold to
ask questions about mathematical contents and studying. Thus, the prep course also
includes social components.
In lecture-accompanying tutorials, the members of the SSC teach in various fields
of study (engineering, computer sciences, business administration, health manage-
ment). Subjects are (economics) mathematics as well as physics and engineering
mechanics. Again, the focus lies on the active practice of the mathematical content.
2.2 Study Design
Since evaluation of study and teaching is an important instrument for ensuring (didac-
tic) quality, the SSC has been carrying out a scientific accompanying research since
the beginning. The aim of this research is to especially monitor the effectiveness of
44 M. Hommel et al.
the supportive measures in order to adapt them efficiently. Furthermore, the analysis
of the learning outcomes and progresses should provide insights into the learning
behavior and performance of first-year students. A long-term study should also dis-
close trends. As the results of this research can support different stakeholders in
decision-making (learners, teachers, faculty and course facilitators, institution, gov-
ernance), they are used as a basis for the LA framework of Aalen UAS (see Sect. 3).
The following subsections describe the data captured and analyzed by the SSC
regularly since 2013, some research questions as well as the research method and
some exemplary evaluation results.
2.2.1 Database and Data Protection
In order to analyze the students’ learning behavior and performance as well as to

check the effectiveness of the supporting measures, the following data is collected
each semester.
• In the prep course and in the semester-accompanying tutorials, attendance lists
are kept to record the participation frequency of the students in the measures.
• At various times, paper and pencil tests (pretest, posttest, follow-up test) of the
mathematical foundations that are covered by the prep course are carried out to
capture the specific level of knowledge of the students (compare time frame in
Fig. 3.2).
• Prior to the tests, a self-assessment of the new students on the course topics is
queried in order to examine how students assess their own basic mathematical
knowledge. The self-assessments are compared with the actual test results.
The collected data (about 4000–4500 test sheets per year) is entered manually
into standardized Excel spreadsheets. Therefore, the evaluations are far from being
“real time.” To gain further insight, the data is supplemented by electronically avail-
able data records from the student information system (SIS, known in Germany
generally as the HIS management system of the university administration) and
linked to it. The SIS contains, e.g., sociodemographic data such as the type and
grade of the higher education entrance qualification and examination results. Unlike
common data collection on sociodemographic data using questionnaires, SIS pro-
vides a consistent and reliable source of information.
The evaluation of personal data requires a data protection safeguarding. For the
database of the project described above, this safeguarding has already taken place.
This means, especially, that the individual linkage of the collected data to SIS data
is permitted (Egetenmeier et al., 2016). For the online course in mathematics, the
process of data protection safeguarding is still in process.
The students’ participation in the support measures and the prep course tests is
optional. Note: Data is only evaluated for those students who have signed a data
privacy statement developed for the SSC (SSC DPS), i.e., only the data of part of the
students in the study entry phase are analyzed. That accounts for about 50–60% of
the first-year students (representing a statistically relevant subgroup).
2.2.2 Research Questions
A better understanding of learning behavior, learning progress, and the impact of

support measures can lead to appropriate interventions to improve learning as well
as learning environments if the insights are fed back to the different stakeholders.
The identification of groups of students who are particularly at risk of dropping out
and may require special support is also of great interest to stakeholders of all levels.
Moreover, understanding long-term trends can help stakeholders of the meso- and
macro-level to develop curricula and supportive measures.
To gain such insights on first-year students of all faculties in the field of mathe-
matics, the following research questions were examined:
1. With which level of knowledge of the individual basic mathematical topics do
students start their degrees, and how is their self-assessment on these topics?
2. Is there a way to directly measure learning outcome of the “prep course” and
does the attendance of the course change the students’ self-assessment?
3. Do the measures “prep course” and “lecture-accompanying tutorials” have an
influence on the success in the exams of basic subjects in the study entry phase?
4. Students of which sociodemographic background use the SSC measures

“prep course” and “tutorials” and with what frequency do the respective groups
use the measures?
5. Which participants show special deficits in the initial mathematical knowledge
that is indispensable for the chosen subject?
6. Which groups of students are at particular risk of changing the course or even
dropping out? Is it possible to identify characteristic features in the personal data
(e.g., UEQ, test results in the prep course, but also ECTS credits, non-taking of
examinations, failed attempts) that already at an early-stage point toward a prob-
lematic study process?
7. What effects do SSC measures have on the study process or the long-term suc-
cess of first-year students?
8. Is it possible to disclose trends or developments from the collected and analyzed
data of the first-year students?
2.2.3 Research Method
The following list explains how the data described in Sect. 2.2.1 is used in order to
answer the questions in Sect. 2.2.2. The numbering coincides with the order of the
research questions.
1. An unannounced pretest together with a self-assessment to the content of the
prep course is taken directly before starting the prep course. The results indicate
the level of initial math knowledge of the freshmen as well as their
self-assessment.
2. The comparison of the post- and follow-up test results with the pretest

results determines the learning progress achieved by the measure “prep course.”
46 M. Hommel et al.
The learning progress of basic mathematical knowledge through the prep course

can thus be made visible. The prep course follow-up tests are written in selected
lectures during the semester to provide a comparison group of students who did
not attend the prep course.
3. Based on the individual marks in math exams (from the SIS), it can be examined
whether a regular participation in the prep course or in the lecture-accompanying
tutorials favors the examination success.
4. Analyzing the data from the SIS sheds light on how the groups of participants are
composed (e.g., UEQ type and final school grade – FSG) and whether there are
differences in the use of support.
5. Group-specific analyses (e.g., according to sociodemographic characteristics)
help to identify students with a particular need for support.
6. In addition to the answers given to questions 3–5, an in-depth analysis of the test
results is planned. Postponing exams, multiple exam attempts, or multiple
changes of study courses may indicate that a student is at risk of quitting his or
her studies.
7. The analysis and comparison of individual courses of SSC participants can pro-
vide an answer to question 7. The SSC is allowed to conduct these investigations
based on the data privacy statement.
8. Long-term studies aim to identify trends and developments. For this, data has
been collected since summer semester 2013.
The algorithm used to analyze the data consists of several steps (pseudonymizing,
merging, filtering/clustering, evaluating, visualizing). Figure 3.3 illustrates the dif-
ferent steps.
In order to comply with data protection, the data is prepared in a pseudonymizing
step, i.e., in each file, the typical student matriculation number is replaced by an
automatically generated pseudo number (Egetenmeier et al., 2016). All evaluations
are carried out with the modified data. Since the matriculation number at a univer-
sity is assigned uniquely, the students’ individual learning process can be followed
throughout the semesters. In accordance with data protection, decoding of the
pseudo-numbers and, thus, individual identification of students is very difficult.
Subsequently, the data obtained from the support measures are linked with
sociodemographic data in a merging step using the pseudo numbers as key. Filtering
Fig. 3.3 Data preprocessing and analyzing steps

the relevant data for a particular rating, i.e., selecting only the data that is really
needed for the particular evaluation (e.g., pseudo number + participation in the
prep course + mark in the first math exam), keeps the data analysis short and func-
tional. In the clustering step, some data (e.g., the type of UEQ or the attribute of
having passed or failed an exam) are used to group the dataset into a few relevant
groups. This allows the identification of group-specific features in the data. The
evaluating step is the main step of the analysis. Here, methods of descriptive statis-
tics are used. Finally, in a visualization step, the results of the evaluations are dis-
played according to the target group (stakeholder).
The above steps of the data analysis are realized using separate MATLAB rou-
tines developed at SSC. It should be noted that each of the SSC’s data analyses must
consist of at least seven people to protect students’ privacy. This corresponds to the
evaluation statute of Aalen UAS.
2.2.4 Exemplary Analyses
This section contains some exemplary data analyses of the SSC answering several
of the research questions described above. Analyses answering research questions 1
and 2 are presented in Nagengast et al. (2013). It could be shown that the prep course
significantly improves the entry-level competences in mathematics (development
from pre- to posttest). Unfortunately, the improvement is not permanent. Question
5 is investigated in Nagengast et al. (2017). It turned out that the educational biog-
raphy has an important impact on the initial mathematical knowledge of first-year
students. Questions 6–8 are subject of future research.
This contribution concentrates on questions 3 and 4, i.e., the impact of SSC mea-
sures on math exams and in addition UEQ grouping and prep course participation
for first-year students who passed or failed the first math exam.
The influence of the SSC prep course and the tutorials on the first mathematics
exam of the degree has been evaluated since the beginning of the accompanying
research. Figures 3.4, 3.5, and 3.6 show results for the winter semester (WS) 2016.
Fig. 3.4 Impact of participation in prep course lectures on the success at the first math exam
48 M. Hommel et al.
Fig. 3.5 Impact of participation in prep course tutorials on the success at the first math exam
Fig. 3.6 Impact of participation in semester-accompanying tutorials on the success at the first
math exam
Other semesters show comparable results. Figures 3.4 and 3.5 present the influence
of the prep course participation on the success at the math exam. At the top of the
figures, the results of all participants at the first math exam are visualized (exam
participants (total)). In the middle, the results for those students who participated in
the prep course lecture (Fig. 3.4) and the prep course tutorial (Fig. 3.5) are shown.
The difference group, whose results are displayed at the bottom, includes all stu-
dents who did not participate in the respective support measure or did not sign the
SSC data privacy statement (SSC DPS). Figure 3.5 also distinguishes between those
attending more (50–100%) or less (1–49%) than half of the prep course tutorials.
It has turned out that a regular participation in the prep course lecture (Fig. 3.4)
and especially in the tutorials (Fig. 3.5) on more than half of the offered dates
significantly correlates with better exam results. Note the low rate of 21% for
failed exams in this group (Fig. 3.5).
Figure 3.6 displays the influence of a regular participation in lecture-
accompanying tutorials during the semester. As not every mathematics lecture in the
first semester can be supported by a tutorial held by the SSC for reasons of capacity,
the total number of exam participants (n = 553) is lower than in Figs. 3.4 and 3.5. In
Fig. 3.6 it contains only those students who had the possibility to attend a SSC tuto-
rial. In this case, the difference group consists of all students having written the first
math exam but never attended the SSC tutorial or did not sign the SSC DPS. It is
also possible that students of this group took part in alternative tutorials that are held
by students from higher semesters.
Figure 3.6 shows that a regular participation in semester-accompanying tutorials
also correlates with the chance to pass the exam significantly. Of course, there are
also students who do not make use of the measures and still perform well (see
results of the difference group), as they master their mathematical content well.
For getting deeper insights, the sociodemographic background of the students is
considered in the analyses (Nagengast et al., 2017). Figure 3.7 compares the distri-
bution of the educational biographies (UEQ-types) of all first-semester students
having signed the SSC DPS and the participants of the first math exam having
signed the SSC DPS. For an explanation of the different UEQ-types and the German
education system in general, see Eckhardt (2017).
Fig. 3.7 Proportion of students with different educational biographies (types of UEQ) for all first-
semester students (left-hand side) and all participants in the first math exam (right-hand side)
50 M. Hommel et al.
The diagrams in Fig. 3.7 show similar distributions of the educational biogra-

phies for all first-semester students (left-hand side) and for all exam participants
(right-hand side). Furthermore, both groups consist of a similar number of students:
The diagram on the left-hand side contains the educational biographies of 898 stu-
dents, whereas the educational biographies of 906 exam participants (72 of them
repeated the exam) were analyzed in the diagram on the right-hand side.
For further analysis of the relationship between the educational biography and
the exam results, the group of participants in the first math exam (right-hand side
of Fig. 3.7) is divided into students who passed the exam and those who failed it.
Figure 3.8 shows that the group of students passing the exam contains a higher
proportion of students with a general higher education entrance qualification
gained at the “Gymnasium” (50%) or “Fachgymnasium” (12%) than the group of
failing students (29% and 13%, respectively). This shows that it might be harder
for students with specific educational biographies (e.g., “Berufsfachschule” or
“Kolleg”) to pass the exam and, therefore, specific support measures could be
useful.
Table 3.1 shows for different groups the proportions of students having signed
the SSC DPS in comparison to all students of the respective group. In the WS 2016,
85% of the students having passed the first mathematics exam have signed the SSC
DPS. This means that 85% of the exam passers participated in the measures of the
Fig. 3.8 Proportion of students with different educational biographies (types of UEQ) for the
participants in the first mathematics exam having passed it (left-hand side) and having failed it
(right-hand side)
Table 3.1 Overview over the numbers of first-semester students and participants at the first math
exam and the respective number with signed SSC data privacy statement (SSC DPS) for the WS 2016
Number with Relative proportion of signed
Total number signed SSC DPS SSC DPS to total number
First-semester students 1326 898 68%
Exam participants 1112 906 82%
Passed exam 739 630 85%
Failed exam 373 276 74%
Fig. 3.9 Participation behavior in the prep course of students who passed the first math exam
(left-hand side) and those who failed it (right-hand side)
SSC, e.g., by visiting the prep course or a semester-accompanying tutorial. In con-

trary, only 74% of the students who failed the exam took part in the SSC measures.
Thus, the SSC reaches a larger share of students who pass the exam than those who
fail. This also shows that participation in the SSC measures increases the chances of
success in the first mathematics exam.
For a better understanding of bad exam results, further analyses can be helpful.
Therefore, the participation behavior in the prep course (lecture as well as tutorial)
was analyzed for the passing and the failing group. Figure 3.9 shows the relative
participation numbers for each of the 15 prep course days for both groups, separated
for lecture and tutorial. It should be mentioned that the third week of the prep course
consists of tutorials only.
Figure 3.9 shows that students who passed the exam had a higher attendance rate
at the lecture and especially at the tutorials than students who failed. This implicates
that the revision of the mathematical basics and particularly the active practicing
during the tutorials help to get better exam results. Additionally, it should be noted
that the second prep course week contains also topics relevant to the first math exam
which could also lead to better results for the participants.
52 M. Hommel et al.
The exemplary results presented give an idea of how LA can help to understand
learning outcomes. At Aalen UAS such results are used to inform various stakehold-
ers through a feedback system about first-year students, their in-depth knowledge of
mathematics, and possible developments. The following section describes the levels
to which this feedback system is directed.
3 B
enefits of Learning Analytics at Aalen UAS for Different
Stakeholders
This section presents the LA framework of Aalen UAS with its benefits for the four
different stakeholder levels (cf. Fig. 3.1 or Ifenthaler, 2015). Figure 3.10 as an adap-
tation of Fig. 3.1 summarizes this LA framework graphically. It shows that the
results of the analyses of the SSC’s accompanying research are forwarded to the
various stakeholders via a feedback system (feedback emails and reports). The LA
engine on the right-hand side of Fig. 3.10 consists of two parts: the SSC and the
mathematics online course. Both support the stakeholders. It should be noted that
the SSC also uses its own analysis to reflect the impact of its support measures and
to adapt and refine them according to the conclusions drawn. One of these adjust-
ments was the development and installation of an online course for basic mathemat-
ics, which now also provides LA support (directly in form of a hint system and
indirectly in form of user data). First analyses of the user data captured via the
online course are presented in Egetenmeier, Krieg, Hommel, Maier, and Löffler
(2018). The following subsections describe this feedback system for the stakeholder
levels that are addressed at Aalen UAS.
Fig. 3.10 Learning Analytics at Aalen UAS associated with stakeholder levels
3.1 Micro-Level
The intention of the micro-level analytics is to help the learner to achieve the learn-
ing outcomes more successfully. At Aalen UAS there are two types of LA micro-
level support designed to improve students’ basic mathematical skills. The first is
the three-step hint system in the mathematics online course (Krieg et al., 2017),
which gives hints to solve the tasks in various degrees of detail. This support works
in real time so that students can use it directly while working with the online course.
In addition, the online course can help students to better assess their own perfor-
mance levels (Egetenmeier et al., 2018).
The second type of support at the micro-level are feedback emails sent to stu-
dents taking part in the SSC’s tests. These emails contain the individual test results
of the students as well as the average results of all students who participated in the
respective tests. By comparing their own results with the average of all participants,
students get a first indication of how their performance compares to the group. In
addition, the email contains a rating that communicates the expectations of the
university. Moreover, the students receive specific information about which topics
are not or only insufficiently mastered in order to repeat these topics in a targeted
manner. Additional support includes links to prep course materials and to the math-
ematics online course that can help with the revision. The emails are sent to the
students by the SSC once the correction of the prep course tests is complete. Fast
feedback is important for achieving high effectiveness, allowing students to react
quickly (Clow, 2012; Hattie & Timperley, 2007; Narciss, 2013) and increase study
success. Due to the feedback, students shall be motivated to improve their mathe-
matical skills independently and to internalize the mathematical basics in order to
be able to better follow the lectures and to achieve better exam results. Analyzing
the user data of the online course (cf. Egetenmeier et al., 2018) may give hints on
the effectiveness of the feedback. In particular, it is of interest whether the fre-
quency of participation in tutorials or of the use of the online course is altered by
the feedback emails.
3.2 Meso-Level
The meso-level analytics supports two groups of stakeholders: faculty and course
leaders and teachers. Both receive study course reports similar to the study course
fact sheets of Pohlenz, Ratzlaff, and Seyfried (2012). These reports contain infor-
mation on the composition of the first-semester group of the respective study course
concerning educational biography (as in Fig. 3.7), the participation behavior in the
prep course (as in Fig. 3.9), and their results and self-assessments in the SSC tests
(as in Maier, Hommel, & Egetenmeier, 2018). The study course of interest is com-
pared to the results for the entire university. A further description of the reports can
be found in Maier et al. (2018).
54 M. Hommel et al.
The information contained in the reports is designed to raise the teachers’ aware-
ness of the heterogeneity of the student population in terms of their educational
biographies, their mathematical prior knowledge, and their willingness to use sup-
port services. That way, teachers should be encouraged to adapt their lectures to the
needs of the group, for example, through additional exercises or tutorials or through
a basic introduction of specific topics that are not addressed in each type of school.
In this way, the overall quality of the courses can be increased, so that the students
can better follow the lecture.
Concerning faculty and course facilitators, the reports can support the curricu-
lum and learning design. In particular, phenomena that span several semesters can
be observed and taken into account in curricular adjustments. Examples are the
introduction of additional tutorials, admission tests, or the attempt to motivate stu-
dents to better use support services.
The SSC sends the reports to the person in charge at the beginning of the lecture
period when the test correction of the prep course has been completed. This allows
teachers to react to it already during the semester, which is essential for the LA
project to be highly effective. At the beginning of the following semester, teachers
receive a second report containing correlations of exam results with the participa-
tion in SSC support measures as outlined in Sect. 2.2.4 (Figs. 3.4, 3.5, and 3.6). This
information can help in the long-term development of the study course. For privacy
reasons, this information may only be shared with the teacher and not with the
course facilitators.
Since two groups of stakeholders belong to the meso-level, the different reaction
times of the interventions become clear (cf. Clow, 2012). While teachers can respond
very quickly within the semester, but only reach the students in their course, faculty
and course facilitators are more interested in long-term analyses that can affect
course development and have an impact on a larger group of students.
3.3 Macro-Level
The macro-level LA support is an institution report that contains institution-wide

analyses. On the one hand, it contains aggregated analyses for the entire university
from the study course reports of the meso-level. On the other hand, further analyses
such as the correlation of educational biographies and initial mathematical knowl-
edge or examination results are included (cf. Nagengast et al., 2017). In addition,
investigations of specific groups such as students who failed the exam are presented
(see Sect. 2.2.4). Long-term analyses over several semesters should uncover trends
that play an important role in future planning. This also includes the number of
registrations for the prep courses or changes in the distribution of educational biog-
raphies of the freshmen. All this information can support decision-makers of the
institution in strategic planning and further development of higher education. In this
way, the SSC can contribute to quality management during the study entry phase
(cf. Maier et al., 2018).
3.4 Mega-Level
LA support on the mega-level is expected to facilitate cross-institutional analytics.

It should identify and validate patterns within and across institutions and help to
inform educational policy-making (Ifenthaler, 2015). Concerning the mega-level,
Aalen UAS presents the results of the evidence-oriented accompanying research in
different committees, for example, the cooperation of schools and universities in
Baden-Wuerttemberg (cf. Dürrschnabel & Wurth, 2015) or the University Federation
SouthWest Germany (cf. Daberkow et al., 2015). In this way, the knowledge gained
from the investigations can be communicated to other institutions and compared
with their experiences. This could lead to cross-university measures that might also
have an impact on educational policy. Thereby, the community of educational
researchers also profits from the analyses.
4 Summary and Conclusion
High teaching quality, study success, and rising heterogeneity of students play an
increasingly important role at UAS. Therefore, the SSC of Aalen UAS offers differ-
ent measures for supporting students in the study entry phase. The positive impact
of these support measures, e.g., on the results of the first mathematics exam, can be
demonstrated by means of an evidence-oriented research based on an extensive
database (Sect. 2). The results of these analyses can be helpful for stakeholders on
different levels from students over teachers and faculty/university management to
governance (Sect. 3). If each stakeholder uses the respective information, this can
contribute to raising teaching quality and, thus, study success.
In order to achieve a high effectiveness, the SSC uses LA to reach all levels of
decision-makers and stakeholders. Since many other LA-based projects or imple-
mentations only focus on one (or two) level, this provides the opportunity to reach
the stakeholders with more impact as the results are accessible to each person
included. In terms of Clow (2012), the loop of the LA cycle is closed. As stakehold-
ers of the mega- and macro-level have a different view than the micro- and meso-
level stakeholders, the focus of our research shifts from impact studies (micro-,
meso-level) to long-term research (macro-, mega-level).
Although the LA framework of Aalen UAS is already very comprehensive, there
are some limitations which can be improved in future. One obvious limitation of the
feedback system in general is the lack of pedagogical recommendations (cf. Sect.
1.2). Hence, the implementation of a suitable advice system for each stakeholder
could further improve the feedback system of the SSC.
The manual correction of the tests which serve as the basis for many analyses is
another limitation because there is a certain time delay until the stakeholders receive
the feedback and can react accordingly. As online assessment is getting more and
more important, the transition from paper/pencil tests to online tests could help here
to improve the speed of response and, therefore, the effectiveness. Furthermore, the
56 M. Hommel et al.
availability of this user data may provide new insights into the learning process of
students and lead to a further benefit for all stakeholders.
The LA framework of Aalen UAS is based on data and analyses of the accompa-
nying research originally initiated for evaluating the effectiveness of the SSC’s sup-
port measures. Therefore, the test concept and the type of data to be collected were
chosen specifically in order to reach this aim. Since some aspects are not easily
transferable (e.g., test concept, form of response) to other institutions, a “one-to-
one” implementation of the model is not recommended. However, the research
questions in general, the idea of evaluating support measures, analyzing learning
behavior and study progress, as well as serving all levels of stakeholders with appro-
priate data are, of course, transferable. The basis for this is that concerns of data
protection have been clarified. Depending on institutional conditions, some adjust-
ments may be necessary. For large institutions, for example, the manual data entry
reaches its limits. This could be overcome, for instance, by using online tests.
Furthermore, the form of response and the presentation of the information may need
adjustments.
Despite the limitations mentioned, the LA framework of Aalen UAS provides
good approaches to a comprehensive model in the study entry phase. By implement-
ing further improvements, this can contribute even more to increasing study success
in the future.
Acknowledgments The project AkaMikon is funded by the German Federal Ministry of

Education and Research (BMBF) under grant number 01PL16015 as part of the Teaching Quality
Pact (http://www.qualitaetspakt-lehre.de). The responsibility for the content of this publication lies
with the author.
References
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to
increase student success. In S. Dawson, C. Haythornthwaite, S. Buckingham Shum, D. Gašević,
& R. Ferguson (Eds.), ICPS, Proceedings of the 2nd International Conference on Learning
Analytics and Knowledge (LAK 2012): University of British Columbia, Vancouver, Canada,
April 29–May 02, 2012 (pp. 267–270). New York, NY: ACM.
Avella, J. T., Kebritchi, M., Nunn, S. G., & Kanai, T. (2016). Learning analytics methods, benefits,
and challenges in higher education: A systematic literature review. Online Learning, 20(2), 13.
Baas, M., Buitendijk, S., van Elk, L., Filius, R., van Hees, J., Hobbes, H. J., … van der Woert, N.
(2015). Perspectives on Developments in Dutch Higher Education: 2015 Open and Online
Education Trend Report.
Bebermeier, S., & Nussbeck, F. W. (2014). Heterogenität der Studienanfänger/innen und Nutzung
von Unterstützungsmaßnahmen. Zeitschrift für Hochfrequenztechnik, 9(5), 83–100.
Biehler, R., Hochmuth, R., Fischer, P. R., & Wassong, T. (2011). Transition von Schule zu
Hochschule in der Mathematik: Probleme und Lösungsansätze. In R. Haug & L. Holzäpfel
(Eds.), Beiträge zum Mathematikunterricht 2011: Vorträge auf der 45. Tagung für Didaktik
der Mathematik (pp. 111–114). Münster, Germany: WTM Verlag für wissenschaftliche Texte
und Medien.
Brown, M. (2011). Learning analytics: The coming third wave. EDUCAUSE Learning Initiative
Brief, 1, 1–4.
Charleer, S., Vande Moere, A., Klerkx, J., Verbert, K., & Laet, T. d. (2017). Learning analytics
dashboards to support adviser-student dialogue. IEEE Transactions on Learning Technologies,
11, 389–399. https://doi.org/10.1109/TLT.2017.2720670
Chatti, M. A., Lukarov, V., Thüs, H., Muslim, A., Yousef, A. M. F., Wahid, U., … Schroeder, U.
(2014). Learning analytics: Challenges and future research directions. Retrieved from https://
eleed.campussource.de/archive/10/4035
Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In S. Dawson,
C. Haythornthwaite, S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), ICPS,
Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK
2012): University of British Columbia, Vancouver, Canada, April 29–May 02, 2012 (pp. 134–
138). New York, NY: ACM.
Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered through
learning analytics dashboards. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and
Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014
(pp. 629–633).
Daberkow, A., Löffler, A., Egetenmeier, A., König-Birk, J., Knaak, W., Pitsch, S., … Saftenberger,
S. (2015). Physikgrundlagen medial - ein simultanes Testexperiment von 7 Hochschulen. In
Zentrum für Hochschuldidaktik (DiZ) (Ed.), Tagungsband zum 2. HDMINT Symposium 2015:
24./25. September (pp. 192–197). Nuremberg, Germany: Technische Hochschule Nürnberg.
Dell’mour, R., & Landler, F. (2002). Akademische Grade zwischen Traum und Wirklichkeit:
Einflussfaktoren auf den Studienerfolg. Schriften des Instituts für Demographie der
Österreichischen Akademie der Wissenschaften: Vol. 17. Wien, Austria: Verlag der
Österreichischen Akademie der Wissenschaften.
Derr, K., Hübl, R., & Podgayetskaya, T. (2015). Formative Evaluation und Datenanalysen als Basis
zur schrittweisen Optimierung eines Online-Vorkurses Mathematik. In N. Nistor & S. Schirlitz
(Eds.), Medien in der Wissenschaft: Vol. 68. Digitale Medien und Interdisziplinarität:
Herausforderungen, Erfahrungen, Perspektiven (pp. 186–196). Münster, Germany: Waxmann.
Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR). (2014). Reader zur Fachtagung
„Studieneingangsphase“: Qualitätspakt Lehre. Bonn, Germany: Köln. Retrieved from
https://www.mintgruen.tu-berlin.de/fileadmin/user_upload/documents/MINTgruen/
Veroeffentlichungen/QPLehre2014Fachtagung-Reader.pdf
Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR). (2015). Reader zur Fachtagung „Lehr-
und Lernformen“: Qualitätspakt Lehre. Bonn, Germany. Retrieved from https://slidex.tips/
downloadFile/qualittspakt-lehre-reader-zur-fachtagung-lehr-und-lernformen
Drachsler, H., & Greller, W. (2016). Privacy and analytics – It’s a DELICATE issue: A checklist
for trusted learning analytics. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, & C. Penstein
Rosé (Eds.), ICPS, Proceedings of the 6th International Conference on Learning Analytics
and Knowledge (LAK 2016): The University of Edinburgh, Edinburgh, United Kingdom, April
25–29, 2016 (pp. 89–98). New York, NY: The Association for Computing Machinery. https://
doi.org/10.1145/2883851.2883893
Dürrschnabel, K., & Wurth, R. (2015). Cosh – Cooperation Schule–Hochschule. Mitteilungen der
Deutschen Mathematiker-Vereinigung, 23(3), 181. https://doi.org/10.1515/dmvm-2015-0067
Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schröder, U. (2012). Design and
implementation of a learning analytics toolkit for teachers. Educational Technology & Society,
15(3), 58–76.
Eckhardt, T. (2017). The education system in the Federal Republic of Germany 2014/2015: A
descript ion of the responsibilities, structures and developments in education policy for the
exchange of information in Europe. Retrieved from https://www.kmk.org/fileadmin/Dateien/
pdf/Eurydice/Bildungswesen-engl-pdfs/dossier_en_ebook.pdf
Egetenmeier, A., Krieg, S., Hommel, M., Maier, U., & Löffler, A. (2018). (R)Evolution in der
Datenauswertung: Durch Digitalisierung neue Impulse für die Evaluation schaffen. In
S. Harris-Huemmert, P. Pohlenz, & L. Mittauer (Eds.), Digitalisierung der Hochschullehre:
Neue Anforderungen an die Evaluation (pp. 15–28). Münster, Germany: Waxmann Verlag
GmbH.
58 M. Hommel et al.
Egetenmeier, A., Oder, B., Maier, U., Hommel, M., Nagengast, V., & Löffler, A. (2016).
(Keine) Angst vor dem Datenschutz?! Begleitforschung im Hochschulkontext. In U. Lucke,
A. Schwill, & R. Zender (Eds.), Vol. volume P-262. GI-Edition - lecture notes in informat-
ics (LNI) Proceedings, 14. E-Learning Fachtagung Informatik der Gesellschaft für Informatik
e.V (DeLFI 2016): Potsdam, Deutschland, 11.-14. September, 2016 (pp. 245–256). Bonn,
Germany: Gesellschaft für Informatik.
Erdel, B. (2010). Welche Determinanten beeinflussen den Studienerfolg? Eine empirische Analyse
zum Studienerfolg der ersten Kohorte der Bachelorstudenten in der Assessmentphase am
Fachbereich Wirtschaftswissenschaften der Friedrich-Alexander-Universität Erlangen-
Nürnberg (Berichte des Lehrstuhls für Soziologie und Empirische Sozialforschung No.
2010–2). Retrieved from Universität Erlangen-Nürnberg website: http://nbn-resolving.de/
urn:nbn:de:0168-ssoar-220222
Erdmann, M., & Mauermeister, S. (2016). Studienerfolgsforschung: Herausforderungen in einem
multidisziplinären Forschungsbereich. Handbuch Qualität in Studium und Lehre (58, pp. 1–28).
Fortenbacher, A., Beuster, L., Elkina, M., Kappe, L., Merceron, A., Pursian, A., …Wenzlaff, B.
(2013). LeMo: A learning analytics application focussing on user path analysis and interac-
tive visualization. In Proceedings of the 7th International Conference on Intelligent Data
Acquisition and Advanced Computing Systems (IDAACS 2013): Berlin, Germany, 12–14
September, 2013 (pp. 748–753). Piscataway, NJ: IEEE.
Govaerts, S., Duval, E., Verbert, K., & Pardo, A. (2012). The student activity meter for aware-
ness and self-reflection. In Proceedings of the ACM SIGCHI Conference on Human Factors
in Computing Systems (CHI EA’12) (pp. 869–884). New York, NY: ACM Press. https://doi.
org/10.1145/2212776.2212860
learning analytics. Educational Technology & Society, 15(3), 42–57.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1),
81–112. https://doi.org/10.3102/003465430298487
Heiss, C., & Embacher, F. (2016). Effizienz von Mathematik-Vorkursen an der Fachhochschule
Technikum Wien: ein datengestützter Reflexionsprozess. In A. Hoppenbrock, R. Biehler,
R. Hochmuth, & H.-G. Rück (Eds.), Konzepte und Studien zur Hochschuldidaktik und
Lehrerbildung Mathematik. Lehren und Lernen von Mathematik in der Studieneingangsphase:
Herausforderungen und Lösungsansätze (1st ed., pp. 277–294). Wiesbaden, Germany:
Springer Spektrum.
Heublein, U., Ebert, J., Hutzsch, C., Isleib, S., König, R., Richter, J., & Woisch, A. (2017a).
Motive und Ursachen des Studienabbruchs an baden-württembergischen Hochschulen und
beruflicher Verbleib der Studienabbrecherinnen und Studienabbrecher: Wichtigste Ergebnisse
und Handlungsfelder (No. 6|2017). Hannover, Germany: DZHW Deutsches Zentrum für
Hochschul- und Wissenschaftsforschung. Retrieved from https://mwk.baden-wuerttemberg.
de/fileadmin/redaktion/m-mwk/intern/dateien/pdf/Studium_und_Lehre/Studie_Motive_und_
Ursachen_des_Studienabbruchs_an_baden-württembergischen_Hochschulen_Kurzversion.
pdf
Heublein, U., Ebert, J., Hutzsch, C., Isleib, S., König, R., Richter, J., & Woisch, A. (2017b).
Zwischen Studienerwartungen und Studienwirklichkeit: Ursachen des Studienabbruchs,
beruflicher Verbleib der Studienabbrecherinnen und Studienabbrecher und Entwicklung der
Studienabbruchquote an deutschen Hochschulen. Forum Hochschule: Vol. 2017, 1. Hannover,
Germany: DZHW Deutsches Zentrum für Hochschul- und Wissenschaftsforschung. Retrieved
from https://www.dzhw.eu/pdf/pub_fh/fh-201701.pdf
Heublein, U., Hutzsch, C., Schreiber, J., Sommer, D., & Besuch, G. (2010). Ursachen des
Studienabbruchs in Bachelor- und in herkömmlichen Studiengängen: Ergebnisse einer
bundesweiten Befragung von Exmatrikulierten des Studienjahres 2007/08. Forum Hochschule
No. 2010, 2. Hannover, Germany: DZHW Deutsches Zentrum für Hochschul- und
Wissenschaftsforschung. Retrieved from http://www.dzhw.eu/pdf/21/studienabbruch_ursa-
chen.pdf
Heublein, U., Richter, J., Schmelzer, R., & Sommer, D. (2012). Die Entwicklung der Schwund-
und Studienabbruchquoten an den deutschen Hochschulen: Statistische Berechnungen auf der
Basis des Absolventenjahrgangs 2010. Forum Hochschule: Vol. 2012, 3. Hannover, Germany:
DZHW Deutsches Zentrum für Hochschul- und Wissenschaftsforschung. Retrieved from
https://www.dzhw.eu/pdf/pub_fh/fh-201203.pdf
Heublein, U., Richter, J., Schmelzer, R., & Sommer, D. (2014). Die Entwicklung der
Studienabbruchquoten an den deutschen Hochschulen: Statistische Berechnungen auf der
Basis des Absolventenjahrgangs 2012. Forum Hochschule: Vol. 2014, 4. Hannover, Germany:
DZHW Deutsches Zentrum für Hochschul- und Wissenschaftsforschung. Retrieved from
http://www.dzhw.eu/pdf/pub_fh/fh-201404.pdf
Heublein, U., & Wolter, A. (2011). Studienabbruch in Deutschland. Definition, Häufigkeit,
Ursachen, Maßnahmen. ZfPaed (Zeitschrift für Pädagogik), 57(2), 214–236.
Hinkelmann, M., Maucher, J., & Seidl, T. (2016). Softwaregestützte Studienverlaufsanalyse zur
frühzeitigen gezielten Studienberatung. die hochschullehre, 2, 1–11.
Hoppenbrock, A., Biehler, R., Hochmuth, R., & Rück, H.-G. (Eds.). (2016). Lehren und Lernen
von Mathematik in der Studieneingangsphase: Herausforderungen und Lösungsansätze (1.
Aufl. 2016). Konzepte und Studien zur Hochschuldidaktik und Lehrerbildung Mathematik.
Wiesbaden, Germany: Springer Spektrum.
Ifenthaler, D. (2015). Learning analytics. In J. M. Spector (Ed.), The SAGE encyclopedia of educa-
tional technology (Vol. 2, pp. 447–451). Thousand Oaks, CA: SAGE Publications. https://doi.
org/10.4135/9781483346397.n187
Ifenthaler, D., Mah, D.-K., & Yau, J. Y.-K. (2017). Studienerfolg mittels Learning Analytics.
Retrieved from https://www.e-teaching.org/etresources/pdf/erfahrungsbericht_2017_ifen-
thaler-et-al_studienerfolg-mittels-learning-analytics.pdf
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A his-
torical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological
Bulletin, 119(2), 254–284.
Krieg, S., Egetenmeier, A., Maier, U., & Löffler, A. (2017). Der Weg zum digitalen Bildungs(t)
raum – Durch digitale Aufgaben neue Lernumgebungen schaffen. In C. Igel (Ed.), Vol. 72.
Medien in der Wissenschaft, Bildungsräume: Proceedings der 25. Jahrestagung der Gesellschaft
für Medien in der Wissenschaft: 5. bis 8. September 2017 in Chemnitz (pp. 96–102). Münster,
Germany: Waxmann.
Kuh, G. D., Kinzie, J., Buckley, J. A., Bridges, B. K., & Hayek, J. C. (2006). What matters to stu-
dent success: A review of the literature. Commissioned Report for the National Symposium on
Postsecondary Student Success: Spearheading a Dialog on Student Success.
Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education—A literature
review. In A. Peña-Ayala (Ed.), Studies in systems, decision and control: V.94. Learning ana-
lytics: Fundament, applications, and trends: A view of the current state of the art to enhance
e-learning (pp. 1–23). Cham, Switzerland: Springer International Publishing. https://doi.
org/10.1007/978-3-319-52977-6_1
Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education.
Educause Review, 46(5), 31–40. Retrieved from http://er.educause.edu/~/media/Files/Article-
Downloads/ERM1151.pdf
Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed
to inform an institutional strategic plan. Educational Technology & Society, 15(3), 149–163.
Maier, U., Hommel, M., & Egetenmeier, A. (2018). Evidenzorientierte Begleitforschung in der
Studieneingangsphase und ihr Beitrag zur Qualitätsverbesserung in der Lehre. Handbuch
Qualität in Studium und Lehre (64, pp. 65–90).
Nagengast, V., Hommel, M., & Löffler, A. (2013). Studieneingangsphase an der Hochschule
Aalen – fachlich förden und Defizite analysieren. In Zentrum für Hochschuldidaktik (DiZ)
60 M. Hommel et al.
(Ed.), HDMINT.MINTTENDRIN Lehre erleben. Tagungsband zum 1. HDMINT Symposium

2013: Nürnberg, 7./8.11.2013 (pp. 200–208). Nuremberg, Germany: Technische Hochschule
Nürnberg.
Nagengast, V., Hommel, M., Maier, U., Egetenmeier, A., & Löffler, A. (2017).
Studieneingangsphase an der Hochschule Aalen – Entwicklungen und Erfahrungen. In Zentrum
für Hochschuldidaktik (DiZ) (Ed.), Tagungsband zum 3. Symposium zur Hochschullehre in
den MINT-Fächern: 25./26.September 2017 (pp. 129–135). Nuremberg, Germany: Technische
Hochschule Nürnberg.
Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital learning envi-
ronments on the basis of the interactive tutoring feedback model. Digital Education Review
(23), 7–26. Retrieved from https://www.phil.uni-passau.de/fileadmin/dokumente/lehrstuehle/
narciss/Narciss__S.__2013_._Designing_and_Evaluating_Tutoring_Feedback_Strategies.pdf
Papamitsiou, Z., & Economides, A. A. (2014). Learning analytics and educational data mining
in practice: A systematic literature review of empirical evidence. Educational Technology &
Society, 17(4), 49–64.
Pohlenz, P., Ratzlaff, O., & Seyfried, M. (2012). Studiengang Fact Sheets für eine evidenzbasierte
Steuerung von Lehre und Studium. HM (Hochschulmanagement) (3, pp. 83–88).
Reinmann, G. (2015). Heterogenität und forschendes Lernen: Hochschuldidaktische Möglichkeiten
und Grenzen. In B. Klages, M. Bonillo, S. Reinders, & A. Bohmeyer (Eds.), Gestaltungsraum
Hochschullehre: Potenziale nicht-traditionell Studierender nutzen (pp. 121–137). Berlin,
Germany: Budrich UniPress Ltd.
Rindermann, H., & Oubaid, V. (1999). Auswahl von Studienanfängern durch Universitäten -
Kriterien, Verfahren und Prognostizierbarkeit des Studienerfolgs. Zeitschrift für Differentielle
und Diagnostische Psychologie, 20(3), 172–191. https://doi.org/10.1024//0170-1789.20.3.172
Romero, C., & Ventura, S. (2013). Data mining in education. Wiley Interdisciplinary Reviews:
Data Mining and Knowledge Discovery, 3(1), 12–27. https://doi.org/10.1002/widm.1075
Sclater, N., Peasgood, A., & Mullan, J. (2016). Learning analytics in higher education: A review
of UK and international practice. Full report. Bristol.
Siemens, G. (2012). Learning analytics: Envisioning a research discipline and a domain of prac-
tice. In S. Dawson, C. Haythornthwaite, S. Buckingham Shum, D. Gašević, & R. Ferguson
(Eds.), ICPS, Proceedings of the 2nd International Conference on Learning Analytics and
Knowledge (LAK 2012): University of British Columbia, Vancouver, Canada, April 29–May
02, 2012 (pp. 4–8). New York, NY: ACM.
Tempelaar, D., Rienties, B., & Nguyen, Q. (2017). Adding dispositions to create pedagogy-based
Learning Analytics. Zeitschrift für Hochschulentwicklung, 12(1), 15–35.
Vigentini, L., Kondo, E., Samnick, K., Liu, D. Y. T., King, D., & Bridgeman, A. J. (2017). Recipes
for institutional adoption of a teacher-driven learning analytics tool: Case studies from three
Australian universities. In H. Partridge, K. Davis, & J. Thomas (Eds.), Me, Us, IT! Proceedings
ASCILITE2017: 34th International Conference on Innovation, Practice and Research in the
Use of Educational Technologies in Tertiary Education (pp. 422–432).
Willis, J. E., Slade, S., & Prinsloo, P. (2016). Ethical oversight of student data in learning
analytics: A typology derived from a cross-continental, cross-institutional perspective.
Educational Technology Research and Development, 64(5), 881–901. https://doi.org/10.1007/
s11423-016-9463-4
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning ana-
lytics. In M. Pistilli, J. Willis, D. Koch, K. Arnold, S. Teasley, & A. Pardo (Eds.), ICPS,
Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK
2014): Indianapolis, IN, USA, March 24–28, 2014 (pp. 203–211). New York, NY: Association
for Computing Machinery. https://doi.org/10.1145/2567574.2567588
Wong, B. T. M. (2017). Learning analytics in higher education: An analysis of case stud-
ies. Asian Association of Open Universities Journal, 12(1), 21–40. https://doi.org/10.1108/
AAOUJ-01-2017-0009
Chapter 4
Implementing Learning Analytics into
Existing Higher Education Legacy Systems
Daniel Klasen and Dirk Ifenthaler
1 Introduction
The field of learning analytics (LA) is generating growing interest in data and com-
puter science as well as educational science, hence, becoming an important aspect
of modern digital learning environments (Ifenthaler & Widanapathirana, 2014). LA
are often discussed and linked with regard to self-regulated learning where one
general assumption is that each learning process demands a certain degree of self-
regulation (Zimmerman, 2002). How effective a learner can regulate his or her
learning depends on cognitive, motivational, volitional and metacognitive disposi-
tions (Bannert, 2009). Accordingly, self-regulated learning can be seen as a cyclical
process including three major phases: (1) starting with a forethought phase includ-
ing task analysis, goal setting, planning and motivational aspects. (2) The actual
learning occurs in the performance phase, i.e., focusing, applying task strategies,
self-instruction and self-monitoring. (3) The last phase contains self-reflection, as
learners evaluate their outcomes versus their prior set goals. To close the loop,
results from the third phase will influence future learning activities (Zimmerman,
2002). Current findings show that self-regulated learning capabilities, especially
revision, coherence, concentration and goal setting, are related to students’ expected
support of LA systems (Gašević, Dawson, & Siemens, 2015). For example, LA
facilitate students through adaptive and personalised recommendations to better
plan their learning towards specific goals (McLoughlin & Lee, 2010). Other find-
ings show that many LA systems focus on visualisations and outline descriptive
D. Klasen
D. Ifenthaler (*)

https://doi.org/10.1007/978-3-319-64792-0_4
62 D. Klasen and D. Ifenthaler
information, such as time spent online, the progress towards the completion of a
course and comparisons with other students (Verbert, Manouselis, Drachsler, &
Duval, 2012). Such LA features help in terms of monitoring. However, to plan
upcoming learning activities or to adapt current strategies, further recommendations
based on dispositions of students, previous learning behaviour, self-assessment
results and learning outcomes are important (Schumacher & Ifenthaler, 2018). In
sum, students may benefit from LA through personalised support and adaptive con-
tent recommendations throughout their learning journey.
One of the features with a high impact potential on this personalised support are
prompts (Schumacher & Ifenthaler, 2018). Prompts are ad hoc messages which
provide or request individualised information from the students. They can be used
to offer hints to the current learning material, to trigger students’ self-reflection on
their learning process or to request student-specific information. At best, prompts
are directly injected into the students’ learning environment. Prompts are effective
means for supporting self-regulated learning (Bannert, 2009). They are an essential
instructional method for aiding certain aspects which are needed for self-regulated
learning. Prompts support learners in activating their metacognitive strategies.
These strategies make self-regulation, self-monitoring and evaluation possible
(Ifenthaler, 2012; Veenman, 1993).
Davis (2003) investigated when (before, during or after the actual learning pro-
cess) a prompt should be presented to the learner in order to achieve the best learn-
ing outcome. Accordingly, prompting depends on what the prompt is aiming at. If
the aim is to promote the planning of the learning procedures, a presentation before
the learning task is advisable. By contrast, prompting during the learning process is
appropriate, when the learner is to be induced to monitor and evaluate learning pro-
cedures (Davis, 2003; Ifenthaler & Lehmann, 2012).
However, implementing prompts into existing legacy systems in learning envi-
ronments with high data privacy concerns is quite a challenge. This research shows
how a prompting application has been implemented into such an existing university
environment by adding a plug-in to the local digital learning platform which injects
user-centric prompts to specific objects within students’ digital learning environ-
ment. In this paper, we describe the concept and implementation of the LeAP
(Learning Analytics Profile) application including flexible prompting and present
preliminary findings of the data we are able to generate.
2 Concept and Implementation
2.1 General Concept
The main idea of the LeAP application was to provide a system which can easily be
embedded into the existing legacy environment of the university and is easy to
maintain and to upgrade in future. Therefore, it had to fit into the world of legacy
systems while simultaneously generate few dependencies to the other established
applications.
4 Implementing Learning Analytics into Existing Higher Education Legacy Systems 63
Fig. 4.1 Concept of the LeAP application
We therefore decided to split the solution into as many different modules as nec-
essary. These modules communicate with each other via a RESTful API and can
easily be improved or replaced without affecting the rest of the solution. LeAP can
be divided into three types of components (see Fig. 4.1). The main part is the core
module which holds the largest part of the business logic and deals with the connec-
tion to the database. It consists of several sub-modules which are quite independent
of each other. Each of these modules provides a separate API to the other compo-
nents. The second type of component are the plug-ins for the existing legacy appli-
cations. Currently, this is mainly the university’s digital learning platform ILIAS
(Integriertes Lern-, Informations- und Arbeitskooperations-System; www.ilias.de).
As a further development, the integration into the campus management system and
further applications like email or the university library are planned. The first plug-in
was embedded into the web appearance of the digital learning platform. It gathers
the system-specific user data and sends them to the LeAP core application. In addi-
tion, the plug-in checks the availability of prompts for the current user, injects the
prompt into the web page and deals with the prompt’s response. The third type of
component are stand-alone web applications. At the current stage of the project, this
only includes the administration web interface. It is a stand-alone web application,
written with the Angular.js library which communicates with the core application
via a separate administration API as shown in Fig. 4.1.
2.2 Data Privacy Issues
One of our main concerns was the handling of data privacy issues. As almost every
LA feature collects and processes user data by default, it was inevitable to consider
this topic, particularly in regard of the country’s data privacy act and the require-
ments originated by the General Data Protection Regulation (GDPR) by the
European Union. We decided to work within the running, productive environment
of our university as soon as possible. Therefore, we were able to collect real data
and were not biased by an experimental setting. But convincing the university’s IT
department to set up our solution within their running environment required addi-
tional security and privacy arrangements. Such issues have been documented in
recent publications regarding ethical issues and privacy dilemmas in LA (Ifenthaler
& Schumacher, 2016; Ifenthaler & Tracey, 2016; Slade & Prinsloo, 2013).
As shown in Fig. 4.2, we decided to use a pseudonymisation in two steps.
Wherever we are in direct touch with the students’ activities, we use a 32-bit hash
value as an identifier. All tracking events and prompting requests use this hash value
to communicate with the LeAP core application. The LeAP core API then takes this
hash, enriches it with a secret phrase (a so-called pepper) and hashes it again. The
doubled hash is then stored within the core’s database. As a result, we can match
new student-generated data to already existing data but are not able to directly trace
back a specific student by a given date within the database.
Another benefit of the cooperation with the university’s administration is that we
do not need to collect demographic student data, as we can catch hold of them from
the university’s administration afterwards. We are able to receive this data pseud-
onymised in the same way, so it can be matched with the rest of our collected data.
Upon completion of the current project phase, we will be able to combine the track-
ing data, the prompting feedbacks, the students’ grades and demographic data for a
full-featured analysis without need to have access to this personal data during the
data collection phase.
2.3 LeAP Core
The LeAP core component is developed in Java and deployed as a Spring-Boot

application. Spring-Boot applications are Java systems which use the spring web
framework and are deployed with an integrated web application server. Therefore,
they can be started as a separate process without the need of an extra web applica-
tion server like Tomcat or Glassfish. In fact, a Tomcat, Undertow or Jetty web server
is embedded directly into the executable java file when building the application
(Spring Boot Project, 2018).
Fig. 4.2 Encryption of student’s identity

The structure of the core component is built upon several disjoint modules as
shown in Fig. 4.3. These modules offer a separate API to one of the other compo-
nent types outside of the core. This independence of the modules ensures an easy
maintenance and improvement of individual modules without interfering with each
other. The application’s core part offers a few functionalities which can be used by
all modules. The core mainly consists of universally available data objects and
database functionality. Beneath the data objects are students, courses, resources
and events. In contrast, prompts are not part of the core and are organised within
a separate module. Data stored in the application is categorised into two types.
Fig. 4.3 Technical structure of the LeAP core architecture

The first type are resource data like courses and objects. These are stored with an
obvious external relation to the object within the source system. For example, read-
ing materials are stored with an external id, which is similar to the id given to the
file within the e-learning platform. The second type of data are individual-related.
Beneath these are the students themselves and events which can be assigned to
them. These dates have no obvious relation to an external object. They are identi-
fied by an individual hash value which is built upon the student’s university account
and additional secret as described before. This data is not completely anonymous,
but it ensures a certain amount of privacy through this pseudonymity. Thereby, new
user-generated data can be connected to a specific hash; however, the user cannot
directly be identified by this hash. Data like name, gender, or age are not stored
within the LeAP core as they can be gathered from the university’s administration
later on.
For the projects pilot phase, we only use one instance of the core component
which is responsible for the connection to the database and handles all data streams
which occur in the current learning analytics environment. But the concept is ori-
ented to duplicate this core component to spread data load and to approach a variety
of security requirements. We operate one API at 24/7 which accepts the incoming
tracking events and simultaneously operates an API for the lecturer’s administration
interface which can easily be taken down for steady improvements.
2.4 Plug-In for Digital Learning Platform
The student’s first point of contact with the LeAP application is the learning man-
agement system. We developed a plug-in for our local learning management sys-
tem ILIAS which coordinates the tracking and prompting within this system and
allows students to choose their current tracking status. The plug-in is written as a
UserInterfaceHook which adds a new function to the visible layout of ILIAS. The
functionality can be enabled for a specific course, which allows the students to see
a new tab ‘LA-Profile’ for setting their personal tracking status. These status are
‘active’, ‘inactive’ and ‘anonymous’. While in status ‘inactive’, no data is tracked.
In status ‘active’, the data is allocated to the described, individual, pseudonymous
hash. Whereas in status ‘anonymous’, the data is tracked, but not allocated to a
personalised id. As depicted in Fig. 4.4, additional JavaScript libraries for track-
ing and prompting are dynamically embedded during the rendering phase of the
page. This new code is augmented with tracking and user information and handles
the communication with the LeAP core application. Thus, the tracking and
prompting features almost completely run within the user’s web browser and do
not interfere with the ILIAS system. As ILIAS is written in PHP, the plug-in is
also written in PHP. The tracking and prompting libraries are asynchronous
JavaScript.
Fig. 4.4 LeAP plug-in figure for injection
2.5 Prompts
Besides the pseudonymous and anonymous tracking of the students’ activities,

prompting is currently the second main feature of the LeAP project. Tracking allows
us to identify individuals, which should receive personalised prompts. For example,
students who over- or underuse some of the learning features or materials. But
prompts can also be given to a complete course. Prompts are always person and
location related. We can put a prompt for every student at the start location of the
course or position a prompt for an individual student to a specific learning material.
The prompt is then fired when the student hits that location. But whereas the loca-
tion is identified by an obvious identifier, persons are only visible in their hash value
representation. No personal data like the name of the persons are available within
the prompting functionality. Students can only be chosen by their course member-
ship and activities.
When a prompt is fired, it is displayed as a small message window in an overlay
above the active page as shown in Fig. 4.5. The underlying page is greyed out and
cannot be used as long as the prompt is visible. The prompt can consist of a static
information message, a question with a text input possibility, a question with a five-
point Likert answer possibility, a checkbox or a combination of these. In addition,
we can present a link to a research questionnaire which dynamically adds the stu-
dent’s personal hash value. Thereby, we are able to collect data for accompanying
research without collecting the student’s personal data or forcing them to reuse a
given token. The various questionnaires are all brought together by the hash, which
remains constant. Beside student and location, prompts can also be executed at a
Fig. 4.5 Prompt example of a five-point Likert question
given time and for a given duration. Prompts can therefore be active for a few hours
or several weeks. Multiple, different prompts can be active at the same time for
several students.
3 Pilot Study
3.1 Research Focus
The pilot study focussed on the usability and practicability of the LeAP application.
The research focus was to (1) validate the storage of tracking data, (2) performance
of the prompting feature and (3) use of the privacy settings.
3.2 Tracking Data (RQ1)
Initial data was collected after the system was running reliably for 2 months (since
the start of the fall semester 2017). It was activated in two courses with approxi-
mately N = 400 students. One course was in the field of economic and business
education, the other in the field of computer sciences. We collected more than
120,000 events and tracked the usage of over 200 learning resources. The underly-
ing technology stack works flawless. The collected data is reliable and satisfies the
requirements for later analysis.
3.3 Prompts (RQ2)
During the fall semester 2017, we performed nine prompts in the productive learn-
ing environment. Each prompt lasted for 1 week. We prompted between 150 and
250 students and received response rates between 11% and 64%. The handling of
the prompting tool is flawless. The pilot lecturers had no difficulties to create, man-
age and submit their prompts. The prompts have been widely accepted and we
received no information about noticeable difficulties. Additional survey data is cur-
rently analysed which investigates the students’ perception towards learning sup-
port of the prompts.
3.4 Data Privacy (RQ3)
The default tracking for students’ data at the beginning of the semester is set to
‘anonymous’. The students are free to change this to ‘active’ or ‘inactive’ at every
point in time. We informed them several times about the functionality and options.
Indeed, we informed them that it is an active research project and would be happy
to have as much participants as possible. But we also guaranteed that we are not able
to identify the individuals until the end of the semester and therefore it could not
have an influence on their grading or future studies. After 3 weeks, we had na = 65
active students, ni = 4 inactive students and nn = 348 anonymous students.
4 Discussion
4.1 Data Privacy
As we are seeking to provide a full learner centric system in the future, our approach
starts with the learners’ decision to provide their learning progress data. The solu-
tion with using a MD5 hash value of the students’ university accounts at the front,
and a doubled hashed value in the core application ensures a satisfying amount of
privacy for the projects pilot phase (Ifenthaler & Schumacher, 2016). We are able to
compute an anonymous, complete, coherent dataset at the end of the semester, with-
out the need to store critical, personal data during the semester.
But as a MD5 hash is not unique, it exists a minuscule possibility to dilute our
dataset. In theory, two different university accounts could be hashed to the same
value. The current system would not be able to separate them. Nonetheless, this
probability is quite low. The hashing and merging of the different data sources is
therefore a topic of current research in our project.
The students appreciate the option to include or exclude themselves from the
data tracking but mostly ignore this possibility and stay in status ‘anonym’. To what
Fig. 4.6 Screenshot of the privacy feature of the LA dashboard
extent this is based on an active decision or passive laziness is a topic of further

investigation and depended on their individual privacy calculus for disclosing per-
sonal data (Ifenthaler & Schumacher, 2016).
After a qualitative follow-up study with participants in the first semester and an
in-depth conversation with experts in data protection and privacy regulation, we
decided to change the privacy settings including two options (see Fig. 4.6): (1) LA
profile active including anonymous tracking and personalised support and and (2)
LA profile not active including no tracking but basic LA features, e.g., personal goal
setting or reminders. We also updated the data privacy setting to be compliant with
the GDPR which requires that students may export all stored data as well as may
request to delete all stored data (see Fig. 4.6).
4.2 Impact of Prompts on the Learning Progress
As this part of the project started just at the beginning of the fall semester 2017, we
are not yet able to provide convincing insights regarding the impact on the students’
learning progresses. We are currently performing a larger research study focussing
on the learning support, acceptance and learning outcomes of the students. Beside
the prompts within the productive digital learning environment, we set up a dedi-
cated copy of the university’s learning platform and used this laboratory system to
investigate the impact of different prompting types on the students learning progress
under laboratory conditions with various experimental groups.
5 Conclusion
We implemented a tracking and prompting solution into the existing digital learning
infrastructure of our university by injecting the respective functionality through
separate JavaScript libraries into the legacy systems. By tracking the students via a
pseudonymous hash, we are able to collect students’ data throughout various sys-
tems without the necessity to collect further personal data (Ifenthaler & Schumacher,
2016). We are further able to merge this data with other university known data like
demographic data and grades at the end of the semester into a complete, anonymous
dataset for further investigation.
The solution is used to perform various educational research studies, focussing
on effects of prompting for self-regulated learning (Bannert, 2009). We are further
planning to extend the various LA features. The next step is the extension of the
students’ direct feedback. The students will get a more transparent feedback on the
amount and type of data which was collected and how this data can be allocated to
their current learning processes (see Fig. 4.7). Furthermore, we will steadily improve
the application and plan to extend the area of research to more courses in the
following semester. Another development includes a teacher application for insights
into individual learning processes, opportunities to interact with students whenever
needed (Kuhnel, Seiler, Honal, & Ifenthaler, 2018) and further developing learning
materials and curricular planning (Ifenthaler, Gibson, & Dobozy, 2018).
Fig. 4.7 Overview of the LA student dashboard as plug-in of the digital learning management
system
References
Bannert, M. (2009). Promoting self-regulated learning through prompts. Zeitschrift für Pädagogische
Psychologie, 23(2), 139–145.
Davis, E. (2003). Prompting middle school science students for productive reflection: Generic and
directed prompts. Journal of the Learning Sciences, 12(1), 91–142.
problem-solving scenarios. Journal of Educational Technology & Society, 15(1), 38–52.
Ifenthaler, D., Gibson, D. C., & Dobozy, E. (2018). Informing learning design through analyt-
ics: Applying network graph analysis. Australasian Journal of Educational Technology, 34(2),
117–132. https://doi.org/10.14742/ajet.3767
Ifenthaler, D., & Lehmann, T. (2012). Preactional self-regulation as a tool for successful problem
solving and learning. Technology, Instruction, Cognition and Learning, 9(1-2), 97–110.
analytics. Educational Technology Research and Development, 64(5), 923–938. https://doi.org/
10.1007/s11423-016-9477-y
Ifenthaler, D., & Tracey, M. W. (2016). Exploring the relationship of ethics and privacy in learn-
ing analytics and design: Implications for the field of educational technology. Educational
Technology Research and Development, 64(5), 877–880. https://doi.org/10.1007/s11423-016-
9480-3
Kuhnel, M., Seiler, L., Honal, A., & Ifenthaler, D. (2018). Mobile learning analytics in higher edu-
cation: Usability testing and evaluation of an app prototype. Interactive Technology and Smart
Education. https://doi.org/10.1108/itse-04-2018-0024
McLoughlin, C., & Lee, M. J. W. (2010). Personalized and self regulated learning in the Web
2.0 era: International exemplars of innovative pedagogy using social software. Australasian
Journal of Educational Technology, 26(1), 28–43.
Schumacher, C., & Ifenthaler, D. (2018). Features students really expect from learning analytics.
Spring Boot Project. (2018). Retrieved June 29, 2018, from http://spring.io/projects/spring-boot
Veenman, M. V. J. (1993). Intellectual ability and metacognitive skill: Determinants of discovery
learning in computerized learning environments. Amsterdam, The Netherlands: University of
Amsterdam.
Verbert, K., Manouselis, N., Drachsler, H., & Duval, E. (2012). Dataset-driven research to support
learning and knowledge analytics. Educational Technology & Society, 15(3), 133–148.
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice,
41(2), 64–70.
Chapter 5
When Students Get Stuck: Adaptive
Remote Labs as a Way to Support
Students in Practical Engineering
Education
Anja Hawlitschek, Till Krenz, and Sebastian Zug
1 Introduction
The field of computer science has to deal with a relatively high number (over 40%)
of dropouts at German universities (Heublein, 2014). However, dropout in com-
puter science is not only a problem at German universities but also in other
European countries (Kori et al., 2015) or in the USA (Talton et al., 2006). The
dropout rate of female students is often even higher than that of their male fellow
students (Talton et al., 2006), which might be a result of being underrepresented in
the discipline (Cox & Fisher, 2008). The reasons for dropout are complex. Most
often the students have false expectations about the contents of study, which lead
to motivational problems, or they are frustrated due to high performance require-
ments. At the same time, the increasing heterogeneity of students leads to dropouts,
in particular due to problems with different prior knowledge but also because of
sociodemographic factors, e.g., an increasing number of students who have to bal-
ance study, work, and/or parenting (Isleib & Heublein, 2017). Especially, prior
knowledge and academic preparedness of students are correlated with retention in
computer science programs (Horton & Craig, 2015; Kori et al., 2015; Talton et al.,
2006). Also motivation and interest of the students play an important role. The
higher the motivation and interest in the content, the lower the probability of drop-
out (Kori et al., 2015, 2016).
The situation at course level is similar. Within a meta-analysis in 161 introduc-
tory programming courses in 15 countries worldwide, Watson and Li (2014)
revealed a dropout rate of approximately 32%. The percentage of students who
A. Hawlitschek (*)
Magdeburg-Stendal University of Applied Sciences, Magdeburg, Germany
T. Krenz · S. Zug
Otto-von-Guericke-University Magdeburg, Magdeburg, Germany

https://doi.org/10.1007/978-3-319-64792-0_5
74 A. Hawlitschek et al.
did not pass the introductory programming course remained nearly constant
between 1980 and 2013. There were no significant differences in dropout rates
with regard to programming language taught. Furthermore, while the authors
found significant differences between the dropout rate in the different countries
(Portugal and Germany had the highest dropout rates with over 50%, whereas
Canada and Taiwan, e.g., had noticeable lower rates of about 20%), because of
small sample sizes, these results should not be overestimated or generalized. If
reasons for dropout are already reflected on the course level, this could be a start-
ing point for providing individual support to students who have a higher probabil-
ity of dropping out. With the help of learning analytics, it becomes possible to
detect students at risk automatically (Papamitsiou & Economides, 2014). Learning
analytics is the collection, storage, analysis, and evaluation of learner data to
optimize learning and learning environments (Ferguson, 2012). A growing num-
ber of universities all over the world already use the data generated by their stu-
dents for the evaluation of teaching, the provision of adapted content, and as an
early warning system. The latter, for example, filters out students at risk of drop-
ping out on the basis of their activities in the learning management system, e.g.,
time spent in exercises or quizzes (Arnold & Pistilli, 2012). There are different
options to support these students: lecturers probably offer additional material or
repeat the basics for the course or individual students. The additional effort
addresses the specific needs of the learners, for example, concerning the sequence,
difficulty, or scope of content (Leutner, 2002; Melis et al., 2001; van Seters et al.,
2012). The goal of implementing adaptivity is to facilitate individualized learning
environments to support efficient and effective learning and avert high dropout
rates. If it is possible to identify the needs of users on the basis of patterns of user
behavior, it is also possible to implement a more fine-grained form of adaptivity
without the usage of assessment tests and questionnaires. The challenge here is
that knowledge about user behavior, which reveals students at risk might not be
sufficient for helping these students. To give an example, on the basis of user
behavior, it is not directly evident whether a user spends little time on an exercise
in the learning management system and has a result below average in an accom-
panying quiz because (1) he is demotivated because the task is to difficult or (2)
he had too little time because he had to work to finance his study or (3) he is
frustrated due to low usability of the learning management system or (4) for any
other reasons. Different reasons for an undesirable user behavior require a differ-
ent reaction of the learning system or the lecturer. This is only possible if the
underlying causes are known. While user behavior alone can provide evidence
that there are problems in the learning process and that intervention might be
necessary, the choice of what type of intervention is needed will usually not be
based solely on user behavior. Therefore, in this study, we will start at an earlier
point of the analytics and begin by examining which learner characteristics are
relevant for dropout in a blended-learning course in computer science. In a sec-
ond step, we examine whether user behavior is related to such factors and/or to
dropout rates.
5 When Students Get Stuck: Adaptive Remote Labs as a Way to Support Students… 75
2 Dropout in Blended Learning
In computer science, as in other STEM subjects, studying in laboratories is espe-

cially important. In these laboratories theory and practice are combined, and stu-
dents acquire practical skills for their professional career. Blended learning is a
promising approach for a laboratory learning setting. Blended learning is the attempt
to combine the time in the course on-campus, which is highly relevant for the learn-
ing performance (Schneider & Preckel, 2017; Schulmeister, 2017), with the advan-
tages of online learning, such as greater local and temporal flexibility of the students.
In comparison with courses that take place only online, the dropout rates in blended
learning are lower, presumably due to the regular face-to-face time with the lecturer
and other students (Park & Choi, 2009). Results from studies suggest that blended
learning might also be superior to courses without any online learning, i.e., which
only take place on-campus (Al-Qahtani & Higgins, 2013; Bernard, Borokhovski,
Schmid, Tamim, & Abrami, 2014; López-Pérez, Pérez-López, & Rodríguez-Ariza,
2011). The remote control of a laboratory (via web interface) provides students with
experiences and competencies they will need in a digitized workplace. In addition,
there are the advantages already mentioned: Students can access the learning envi-
ronment regardless of location and time and are not bound to limited laboratory
hours. They can work in the laboratory as often and as long as necessary for their
individual learning processes. However, despite the advantage of blended learning,
to combine the best of e-learning and face-to-face-learning, the online phase is still
a challenge because there is no direct contact between the lecturer and the students.
Thus, the probability of problems (e.g., if code is not doing what it is supposed to
do) leading to frustration and in the long run to dropout is much more likely to occur
than in face-to-face time on campus with the possibility of direct feedback and help.
In the scientific literature, different factors in the use of digital learning environ-
ments are examined with regard to the dropout rate. Park and Choi (2009) distin-
guish factors that affect the decision to drop out in those that occur prior to the
course and those that are relevant during the course. Factors prior to the course are
sociodemographic variables. Often, studies hereby focus on age and gender (Marks,
Sibley, & Arbaugh, 2005). Factors which affect the possibility to drop out during the
course can be distinguished in external factors resulting from influences from out-
side the course, e.g., family time constraints and job working hours. Internal factors
arise from the student’s engagement with the learning setting and the digital learn-
ing environment. Learners are not a homogenous mass. There are differences in
cognitive and affective variables (Narciss, Proske, & Koerndle, 2007), affecting the
perception and the effects of a learning environment, for example, whether the
instructional design fits the needs of the learner or whether usability issues might
result in a lack of motivation. In this study we focus on the internal factors because
these are especially important for gaining insight into the learning processes and
related factors which are relevant for the decision to dropout (see also the results
from Park & Choi, 2009).
With regard to the internal factors, we can distinguish approaches that have a
focus on motivational components of learning and approaches with a focus on cog-
nitive processing.
2.1 Motivation and Dropout
Motivation is a basis for learning. Motivation determines whether and how learners
(1) deal with the content and (2) use a digital learning environment. Some studies
target learners’ satisfaction, which in fact appears to have a relevant impact on the
dropout rate (Fredericksen, Pickett, Shea, Pelz, & Swan, 2000; Park & Choi, 2009).
The more satisfied learners are with the learning environment, the lower the likeli-
hood of dropouts (Levy, 2007). However, satisfaction is a very broad concept that
can be influenced by different underlying factors. This is also reflected in question-
naires used in some of the studies, which integrate items for ease of use, usefulness,
intrinsic motivation, and social interaction (Levy, 2007). In this study we want to
analyze different facets of motivation in order to adapt interventions more precisely
to the learners needs. Therefore, we focus on the technology acceptance model
which highlights the relevance of user evaluations of learning environments against
the background of a cost-benefit model of motivation. Relevant questions for the
user therefore are: Is the digital learning environment useful for me? Is the effort I
have to invest justified in the light of the benefits? The Technology Acceptance
Model (TAM) and the further developments, like TAM2 and UTAUT, have gained
particular influence concerning studies on the behavioral intentions to use and the
actual usage of software (Davis, Bagozzi, & Warshaw, 1989; Venkatesh & Davis,
2000; Venkatesh, Morris, Davis, & Davis, 2003). Furthermore TAM is also used to
analyze and explain the effectiveness of digital learning environments (Legris,
Ingham, & Collerette, 2003; Liaw, 2008). Perceived usefulness and perceived ease
of use are the most influential factors in the model. The more satisfied a learner is
with the usefulness and ease of use of a digital learning environment, the higher the
persistence of the learner and the lower the dropout rate (Joo, Lim, & Kim, 2011;
Park & Choi, 2009). The self-efficacy of learners in dealing with the learning envi-
ronment or requirements of the content seems to be a crucial intervening variable
(Liaw, 2008; Schneider & Preckel, 2017; Wu, Tennyson, & Hsia, 2010). Additionally
learners can also be highly motivated when dealing with a digital learning environ-
ment because they are interested in the content and/or they enjoy working on the
tasks, i.e., they have intrinsic motivation. The benefits that intrinsically motivated
learners derive from engaging with the remote lab are thusly less focused on out-
comes, but more on intrinsic incentives of the activity as such. The assumption that
learners with more intrinsic motivation drop out less frequently and have a higher
learning performance is obvious (Giesbers, Rienties, Tempelaar, & Gijselaers,
2013; Law, Lee, & Yu, 2010).
Accordingly our first research question is as follows: Do persistent learner and
dropouts show differences concerning motivational variables as perceived useful-
ness, ease of use, intrinsic motivation, and self-efficacy?
2.2 Cognitive Load and Dropout
Based on the assumption of a limited cognitive capacity in working memory,

research on cognitive load theory (CLT) tries to identify instructional designs which
make the usage of cognitive resources for dealing with information as efficient as
possible (Plass, Moreno, & Brünken, 2010; Schnotz & Kürschner, 2007; Sweller,
Ayres, & Kalyuga, 2011). CLT differentiates between different kinds of cognitive
load (Kalyuga, 2011). Extraneous cognitive load (ECL) is caused through subopti-
mal design of an instruction. An inefficient design requires cognitive capacity that
is not due to learning but due to other cognitive activities. During learning extrane-
ous cognitive load should be as low as possible, ensuring that more cognitive capac-
ity is available for the learning processes. Intrinsic cognitive load (ICL) on the other
hand is caused by complexity of task and information, especially by the number of
interrelated elements that have to be processed simultaneously for understanding
the content (element interactivity). However, ICL depends also on the prior knowl-
edge of the learner. More experienced learners have knowledge structures stored in
long-term memory, which help them to process and organize novel information in
working memory. Therefore, they are able to treat single elements of a task as a
whole element (or schema) which in fact leads to decreased element interactivity
(Chen, Kalyuga, & Sweller, 2017). Research consistently reveals that to take the
domain-specific prior knowledge into account is of high relevance for efficient
instructional design (e.g., Chen, Kalyuga, & Sweller, 2017; Kalyuga, 2007;
Schneider & Preckel, 2017). Depending on prior knowledge, the learner needs more
or less support to process the learning content and to avoid cognitive overload or
boredom. Additionally, prior knowledge seems to have a compensation effect:
learners with low prior knowledge highly depend on appropriate instructional
design to reach an optimal learning performance, while learners with higher prior
knowledge could also deal with poor instructional design, e.g., an instructional
design which causes a high amount of extraneous cognitive load (Kalyuga, 2007).
In CLT some researchers assume a third type of cognitive load, namely, germane
cognitive load, which is caused through schema acquisition; however, there is an
ongoing discussion about the necessity to distinguish between intrinsic and ger-
mane cognitive load. A reconceptualization of germane cognitive load as germane
processing, e.g., the amount of mental effort invested dealing with intrinsic cogni-
tive load goes hand in hand (Kalyuga, 2011; Leppink, Paas, van Gog, van der
Vleuten, & van Merriënboer, 2014; Sweller, Ayres, & Kalyuga, 2011). The mental
effort learners invest in the cognitive processing of learning content is on the other
hand a question of motivation (Bures, Abrami, & Amundsen, 2000). Although the
influence of motivation on the amount of invested mental effort was considered in
research on cognitive load early on (Paas, Tuovinen, van Merriënboer, & Darabi,
2005; Moreno, 2006), there is still a research gap (Leutner, 2014; Mayer, 2014;
Park, Plass, & Brünken, 2014). Leppink et al. (2014) examine an interesting
approach by operationalizing germane cognitive load (or rather germane process-
ing) with items that apparently measure the perceived usefulness of the content for
the learning process. This way they implicitly implement a factor which is highly
relevant for motivation as is already mentioned in the context of TAM. However, in
their study they found no significant correlation between germane cognitive load (or
usefulness respectively) and the learning performance.
Whereas it seems plausible that the amount of extraneous cognitive load and
germane processing is crucial for students dropping out or persisting, there are no
empirical results yet. The potential effects of prior knowledge seem to be espe-
cially important. In computer science, there are students in the first semester that
have been programming for years, attending hackathons, and using GitHub, while
others are just beginning with their first “Hello World.” Since the remote lab is a
complex learning environment in which students actively solve problems and
thereby explore and construct knowledge, it is cognitively very demanding in par-
ticular for novice learners. Results of a study on a remote lab indicate that the
learning performance of the students at least partially depend on their prior knowl-
edge (Zug, Hawlitschek, & Krenz, 2017). Students with lower prior knowledge
have lower grades in the exam. However, it is not clear if this effect also is transfer-
rable on dropout rates.
So our second research question is: Do persistent learner and dropouts show dif-
ferences in cognitive variables like extraneous, intrinsic, and germane cognitive
load and their prior knowledge?
2.3 User Behavior and Dropout
Programming is an iterative process, in which the functionalities are implemented

as features, step by step. It is common to write a part of a program, for example, a
function or a class, with its basic components first and check if the execution of the
program with the inclusion of the new code works. If the execution or compilation
fails, the code needs to be revisited and amended. As soon as the program compiles
with the new code, the complexity of the function or class can be extended, or new
features can be implemented. Rinse and repeat.
An experienced programmer will add several lines of code before checking its
correctness by trying to compile the code, while a novice might only add a few lines
or commands before compilation, since it is easier to isolate the cause of an error
with the latter strategy. It could be expected that an experienced programmer’s code
revisions would grow faster and have fewer failing builds, the time spent between
builds would tend to be longer, and the amount of added lines per revision would be
higher, than it would be expected for an inexperienced programmer. Especially situ-
ations where the code compilation fails several times in succession, we consider to
be of high relevance. This could be an indication for an inexperienced programmer,
who fails to interpret the error messages in a way that would allow them to get the
code working. The complex process of writing program code could thusly be
reduced to the occurrence of such error streaks, in order to classify persons as expe-
rienced and inexperienced programmers on a macro level. On a micro level, a sys-
tem that is aware of the error streak concept could provide assistance to students that
are currently stuck.
Therefore, the third research question is: Do dropouts and persistent learners
show differences concerning the probability of an error streak? Is the probability of
an error streak related with prior knowledge?
In the following sections, we will examine how students who have successfully
completed the entire course (i.e., got a participation certificate) differ from students
who left the course at any point in time. On the basis of the findings, an adaptation
to the needs of specific target groups can take place.
3 Study
3.1 Description of the Course
The subject of the study is a course at the Faculty of Computer Science of a German
university. The investigated course started with 70 students in the first lecture, 22 of
them dropped out prematurely. So, the dropout rate in this course was about 31%.
This is slightly better than the general dropout in computer science, but there is still
much room for improvement.
The course conveys the fundamentals of embedded systems in theory and prac-
tice. In addition to a lecture and weekly appointments with tutors, the students had
to program real robots located in the laboratory via remote access in five exercises.
These practical exercises are built on each other. Whereas in the first exercise the
students only had to establish a connection to the robots, in the last exercise they had
to program the robots to escape from a maze. The program code has to be developed
in C++ for Atmel microcontrollers.
For the exercises we provided a digital learning environment with task descrip-
tion and literature on the one hand and a programming interface with livestream
from the robots on the other hand. The students prepared their code, compiled it,
and sent the executable to one of the robots. Based on outputs and by the video
stream, the students evaluated the correctness. At the end of each exercise season
(2–3 weeks), the program code and the results are checked by a tutor.
3.2 Methods and Instrumentation
The study was conducted in the winter term 2017/2018 (see Fig. 5.1). During the first
lecture, the students filled out a quantitative questionnaire concerning their prior
knowledge and sociodemographic variables. The prior knowledge test consisted of
two parts. The first part was a multiple-choice test based on the content of the course.
The test was supplemented by two code snippets in the programming language Java,
whose functionality the students had to evaluate. In the second part, the students had
to self-esteem their prior knowledge concerning different thematic fields of com-
puter science as well as their general programming skills in comparison with their
fellow students (Siegmund, Kästner, Liebig, Apel, & Hanenberg, 2014).
Fig. 5.1 Procedure of course and study
The second questionnaire was submitted after the second exercise. In this ques-
tionnaire the students had to rate their intrinsic motivation while working on the
exercises in the remote laboratory (based on Isen & Reeve, 2005) and the ease of
use of the learning environment (Legris, Ingham, & Collerette, 2003). The extra-
neous and intrinsic cognitive load as well as the germane cognitive load was
examined with an instrument by Leppink et al. (2014). For the measurement of
ECL and ICL, we used the original questionnaire. For the measurement of GCL,
we used one item to measure perceived mental effort in understanding the content
(“I invested a very high mental effort in enhancing my knowledge and under-
standing.”; see Leppink et al., 2014, study 2). We applied the remaining items to
operationalize the perceived usefulness of the learning environment. While use-
fulness in TAM studies is usually operationalized in terms of software efficiency
measures (Legris, Ingham, & Collerette, 2003), concerning genuine learning
environments and in the context of our thematic focus on dropout, this operation-
alization seems more appropriate to us. We used a Likert-type rating scale ranging
from 1 (very low) to 5 (very high).
The remote system used in our project stores the whole programming code,
whenever the user starts the compilation process, alongside the messages the com-
piler returned: error messages, warnings, and compiling reports. For the analysis
presented in this article, we transformed these detailed information into a vector
of consecutive build statuses, classifying each compilation attempt as failing or
successful. As a next step, we calculated the probabilities of one status turning into
the other or staying the same. These probabilities can be visualized as a simple
network plot.
3.3 Sample
In the first questionnaire 58 students (f, 8; m, 49; missing, 1) with a relatively homo-
geneous age (M = 23.6; SD = 4.2) took part. The second questionnaire was accom-
plished by 37 students (f, 4; m, 28; missing, 5). The participants were students of the
3rd to 5th semester. The majority were undergraduate students from computer science
(80.7%); additionally, there were 10.5% students from computer systems in engineer-
ing (B.A.) and some from other computer science-related bachelor programs.
4 Results
With analyses of variance (ANOVA), we examined the differences between stu-

dents who dropped out and students who persisted. The results of the prior knowl-
edge test revealed higher means for the persistent students (M = 8.45; SD = 3.89) in
comparison with the dropouts (M = 6.76; SD = 3.65), but no significant differences
between both groups (F(1.55) = 2.60, p = 0.11, η2 = 0.05). The self-estimation of
their prior knowledge on different thematic fields in the context of the course also
showed no significant differences (see Table 5.1).
Concerning the self-estimation of the programming skills in comparison to the
fellow students, the means were nearly the same in both groups. There was no sig-
nificant difference (F(1.54) = 0.00, p = 0.95, η2 = 0.00) between dropouts (M = 2.95;
SD = 0.89) and persistent learners (M = 2.94; SD = 0.95).
We applied a principal component analysis (with oblimin rotation) to analyze the
items we used for measuring ease of use. Two components were extracted, which
could be interpreted as actual ease of use (e.g., “The remote lab is easy to use.”) and
technical reliability (e.g., “The remote laboratory has worked reliable.”). The results
of the group comparisons on the motivational variables showed higher means for
the persistent students in intrinsic motivation and ease of use. However the ANOVA
yielded no significant difference between the groups concerning motivational
variables (Table 5.2).
Table 5.1 Prior knowledge group comparison

Dropout Persistent
Variables learners (N = 22) learners (N = 35)
“Please rate your prior knowledge
concerning …” M SD M SD F p η2
Roboter applications 2.14 1.24 1.83 1.24 0.82 0.36 0.02
Embedded controller/boards 2.05 1.04 1.91 1.17 0.18 0.67 0.00
Embedded operating systems 1.45 0.96 1.37 0.64 0.15 0.69 0.00
Smartphone apps 2.00 1.19 2.31 1.07 1.05 0.30 0.02
Web front end 2.86 0.99 2.43 1.19 2.03 0.16 0.04
Table 5.2 Motivation group comparison

Dropout Persistent
Variables learners learners
M SD M SD F p η2
N = 21 N = 34
Self-efficacy (Cronbach’s alpha, 0.86) 3.27 0.84 3.39 0.79 0.29 0.59 0.00
N = 9 N = 27
Intrinsic motivation (Cronbach’s alpha, 0.90) 3.39 0.70 3.74 0.78 1.29 0.26 0.04
Ease of use (Cronbach’s alpha, 0.89) 4.00 1.02 4.55 0.78 2.61 0.11 0.07
Technical reliability (Cronbach’s alpha, 0.86) 2.62 1.18 2.67 1.03 0.01 0.91 0.00
Perceived usefulness (Cronbach’s alpha, 0.84) 3.34 0.32 3.67 0.88 1.07 0.30 0.03
Table 5.3 Cognitive load group comparisons

Dropout Persistent
Variables learners (N = 9) learners (N = 26)
M SD M SD F p η2
Intrinsic cognitive load 3.60 0.82 3.05 0.82 2.73 0.10 0.08
Extraneous cognitive load** 3.87 0.81 2.54 1.05 10.75 0.00 0.25
**p < 0.01
We applied a principal component analysis (with oblimin rotation) to analyze the

items for measuring extraneous, intrinsic, and germane cognitive load. Against our
expectations, the analysis only yielded two components—intrinsic cognitive load
(Cronbach’s alpha, 0.86) and extraneous cognitive load (Cronbach’s alpha, 0.87).
The item for measuring germane cognitive load actually loaded on the intrinsic
cognitive load component.
The group comparison yielded higher means for the dropout learners for both
load types (Table 5.3). However, the results of the ANOVA revealed a significant
difference between the groups only concerning extraneous cognitive load.
We analyzed the differences between both groups concerning the probability of
error streaks with ANOVA. Indeed the means for the dropout learners (N = 14,
M = 0.41, SD = 0.14) were significantly higher F(1.53) = 8.14, p = 0.00, η2 = 0.14)
than for the persistent learners (N = 40, M = 0.24, SD = 0.17). With a regression
analysis, we checked whether prior knowledge had a significant effect on the prob-
ability of error streaks. Indeed our finding indicate that students with lower prior
knowledge had a higher probability of error streaks (b = −0.29, t = −1.96, p = 0.05,
R2 = 0.06).
5 Discussion
In our study we tried to identify learner characteristics which are relevant for drop-
out rates in computer science courses. We therefore focused on a course with a
combination of face-to-face instruction and online study. Such a blended learning
approach gives students the possibility to learn at their own pace and in their indi-
vidual learning spaces, at their chosen time, while at the same time give them the
opportunity of direct interaction with the teacher and fellow students in the lecture
on-campus. This configuration offers manifold methods of additional support for
dropout candidates.
To ensure a specific assistance, we analyzed whether we could identify differ-
ences between motivational as well as cognitive variables between students who
drop out and students who persist in the course. We assume that finding such differ-
ences is the first step for making our remote lab adaptive. An adaptive learning
environment should automatically detect whether a student is at risk of dropping out
and give adequate support. To know why a student is about to dropout is a
precondition to provide a suitable intervention. It is a difference if a student has
motivational or cognitive problems because of a lack of usability or because her

programming skills are way too low to deal with the challenges of an exercise or
because of other problems.
Unlike in previous studies (e.g., Kori et al., 2015, 2016), we could not find signifi-
cant differences on the motivational variables between students who dropped out and
persisting students (our first research question). Neither the intrinsic motivation or
the self-efficacy nor the usefulness of the content or the ease of use of the learning
environment were different between both groups. Motivation is a complex theoreti-
cal construct, with a lot of influencing variables. Hence, we can only guess why we
have results which not support previous research. The ratings of the ease of use of the
remote lab were relatively high on average so we might conclude that given a suffi-
cient usability, the effects of that variable are not as relevant as in a poorly working
system. This should be a target of further research. Given the fact that attendance and
learning in the course is not entirely self-determined but also driven by external goals
(e.g., a need of a participation certificate), it might be useful to include items in the
questionnaire, measuring not only intrinsic but also extrinsic motivation (see also
Kori et al., 2016). However, since a limitation of our study is the small size of partici-
pants and especially of dropouts in our sample, the results have to be interpreted with
care. This also holds true for the results on prior knowledge which were in contrast
to earlier research as well (e.g., Horton & Craig, 2015; Talton et al., 2006). Again we
could not find statistically significant differences between the groups, though the
mean of the prior knowledge test was rather lower for the dropouts.
The cognitive variables on the other hand revealed an interesting pattern (our
second research question). While we could not find a significant difference between
both groups concerning intrinsic cognitive load, this was different for extraneous
cognitive load. The dropout group rated the cognitive load which was irrelevant for
learning significantly higher than the persistent group. That result goes hand in hand
with earlier results from cognitive load theory concerning the high relevance of
eliminating extraneous cognitive load (Sweller, Ayres, & Kalyuga, 2011). Our
results indicate that extraneous cognitive load not only affect learning outcomes but
also persistence in a course. Students who drop out had problems that mainly arise
from the design of the instruction and not necessarily from the difficulty of the exer-
cises itself. For them, it was not always clear, what they should do in an exercise and
what the next steps should be. Apparently they got stuck in the instruction rather
than in the programming of the code.
However, there were also students who had the latter problem: from our results,
we consider the detection of error streaks as a promising approach for learning
analytics in computer science (our third research question). There was a significant
difference between students who dropout and students who persist in the probabil-
ity of error streaks. The former had a significantly higher probability of error
streaks in the process of programming. The less prior knowledge the students had
(according to prior knowledge test) the higher was the probability of error streaks.
Although it seems likely that the probability of error streaks and extraneous
cognitive load might correlate, there is no statistic correlation (r = 0.08, p = 0.64).
So in our study, students at risk had two different problems which we have to deal
with differently.
5.1 E
xtraneous Cognitive Load: Practical Implications
and Future Work
Concerning extraneous cognitive load, there are two approaches how to proceed.
The first one is a learning analytic approach. Because we know that there are stu-
dents that got stuck in the instruction, in the following semester we can explicitly
search for a pattern of user behavior this learner might show. Since we know that
these students have difficulties to understand the task and the further steps to go on,
we could explicitly look for user behavior which might correlate with disorienta-
tion, uncertainty, and help-seeking behavior, i.e., extensive clicks or time in the task
section or a high proportion of switching between task section and editor. The sec-
ond approach is to improve the design of the instruction to avoid extraneous cogni-
tive load. Empirical research on instructional design of remote labs, for example,
suggests different forms of guidance, e.g., prompts, process constraints or scaffolds
to help students to keep extraneous cognitive load as low as possible, and manage
intrinsic cognitive load as well (de Jong & Lazonder, 2014). Learners with lower
prior knowledge highly benefit from guidance, while for a learner with higher prior
knowledge guidance often is redundant or even annoying, this should be a case for
adaptivity as well (Kalyuga, 2007).
5.2 Error Streaks: Practical Implications and Future Work
Apart from the ad hoc and postmortem detection of error streaks, the aim of this
endeavor is to administer assistance to students in situations where they are stuck
and unable to help themselves, in order to reduce the time students spent on a cer-
tain problem and ultimately prevent students from dropping out of the course. The
detection of error streaks would allow the lecturer and trainers to intervene in person
or to make the system pull up appropriate instructions to guide the students out of
their error valley. In person interventions could be triggered by the system, which
would flag the user and notify the lecturer about the occurrence of an error streak.
The trainers could then sit down with the student, analyze the problem, and help to
solve misconceptions or understandings the student might have. Of course, the
trainers could point the students to resources, which cover the problematic topic. An
alternative in-person intervention could be to invite other students for a common
debugging session. They would then proceed to solve a similar task using the
method of pair programming. In such a process the experienced students would be
enabled to make the knowledge behind their capabilities explicit, thusly helping the
less experienced student to confront their knowledge deficits with appropriate strat-
egies. In system interventions could administer, whenever an error streak occurs in
a manner that has been observed and solved several times before and certain
resources proved to be key in their solution.
One method of implementing adaptive support is directly related to the error
messages. Compilers or interpreters of programming languages encode the error in
“cryptic” expressions. The correct interpretation of these messages in some cases
requires years of experience. Students without the necessary background knowl-

edge might apply trial and error programming strategies instead of evaluating the
compiler outputs systematically. In a future implementation of our framework, we
intend to support the students at this step on different levels. There exist some data-
bases providing examples and additional information for specific error messages.
Hence, the students are able to earn experiences in a realistic but augmented envi-
ronment, where the error class is explained by isolated examples, possible solutions
are sketched out, and links to further resources are provided (Czaplicki, 2015).
In order to improve the detection of error streaks in further research, we will
define and detect more differentiated statuses that allow employing more sophisti-
cated and individually tailored assistance (see also Berges et al., 2016). Those sta-
tuses could include the duration of an error streak, the amount of repeated errors,
and the meaning of specific errors. Another important part is the counter part of an
error streak: success streaks. Whenever the code compiles without errors, several
times in succession, we can assume that the user is not satisfied with the way the
program is acting. Syntax errors might be absent, but logical and semantic errors are
still present. Especially when programming embedded systems, which interact with
their surroundings, the process of finding the configuration and values for sensors
and actors that are needed to accomplish the given task can be a time and energy
consuming part of the whole process. Automatically detecting such situations would
expand the scope of application for these methods. While the current state, pre-
sented in this paper, allows to help students with little experience that struggle with
the basics of programming, the extension of detecting logic and semantic errors
would enable the lecturers to offer helpful assistance to more experienced students
and advanced students projects, which focus the specific set of skills useful in the
context of programming embedded systems (Table 5.4).
Table 5.4 Practical statuses, detection strategies, and error classes

Practical steps Corresponding
and statuses Representations in logs Measurements error class
Code compiles Built success message Count of successful builds –
without errors
Code compiles with Error messages; Count of failed builds Syntax errors
errors describing the error
Code that compiled Error message Probability of a code revision Syntax errors
without errors following a built that worked before turning
before, now fails success message into non-compiling code
Code that failed Built success message Probability of a code revision –
before, now following an error that didn’t work before
compiles message turning into compiling code
Code compiles; Indication: built without – –
features are errors
functional
Code compiles; Indication: successive Size and time differences Logic and/or
features are not builds without errors between code versions semantic errors
functional
This study is a first step to an adaptive remote lab tailored to the needs of the
learner. We could show that the perception of extraneous cognitive load as well as
the probability of error streaks is relevant for dropout rate. On the basis of our find-
ing, we can now automatically detect learner that got stuck (in either way) and apply
interventions suited for the different needs of these learners. We assume a combina-
tion of both, explorative analysis of variables which affect the decision to drop out
as well as detection of related patterns of user behavior as a promising way for
defining and implementing rules for adaptivity in a digital learning environment.
Acknowledgments This work was partially supported by the German Federal Ministry of
Education and Research (BMBF, Funding number: 16DHL1033).
References
Al-Qahtani, A. A., & Higgins, S. (2013). Effects of traditional, blended and e-learning on students’
achievement in higher education. Journal of Computer Assisted Learning, 29(3), 220–234.
https://doi.org/10.1111/j.1365-2729.2012.00490.x
Arnold, K. E. & Pistilli, M. (2012). Course Signals at Purdue: Using learning analytics to increase
student success. In Proceedings of the LAK12: 2nd International Conference on Learning
Analytics and Knowledge (pp. 267–270). https://doi.org/10.1145/2330601.2330666
Berges, M., Striewe, M., Shah, P., Goedicke, M., & Hubwieser, P. (2016). Towards deriving pro-
gramming competencies from student errors. In Proceedings of the 4th International Conference
on Learning and Teaching in Computing and Engineering (LaTiCE’16) (pp. 19–23). Los
Alamitos, CA: IEEE Xplore Digital Library.
Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014). A meta-
analysis of blended learning and technology use in higher education: From the general to the
applied. Journal of Computing in Higher Education, 26(1), 87–122. https://doi.org/10.1007/
s12528-013-9077-3
Bures, E. M., Abrami, P. C., & Amundsen, C. (2000). Student motivation to learn via com-
puter conferencing. Research in Higher Education, 41(5), 593–621. https://doi.org/10.102
3/A:1007071415363
Chen, O., Kalyuga, S., & Sweller, J. (2017). The expertise reversal effect is a variant of the more
general element interactivity effect. Educational Psychology Review, 29(2), 393–405. https://
doi.org/10.1007/s10648-016-9359-1
Cox, A., & Fisher, M. (2008). A qualitative investigation of an all-female group in a software engi-
neering course project. Journal of Information Technology Education, 7, 1–20.
Czaplicki, E. (2015). Compiler errors for humans. Rethinking the terminal UX in Elm 0.15.1,
http://elm-lang.org/blog/compiler-errors-for-humans
Davis, F., Bagozzi, P., & Warshaw, P. (1989). User acceptance of computer technology - a compari-
son of two theoretical models. Management Science, 35(8), 982–1003. https://doi.org/10.1287/
mnsc.35.8.982
de Jong, T., & Lazonder, A. W. (2014). The guided discovery principle in multimedia learn-
ing. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.,
pp. 371–390). Cambridge, MA: Cambridge University Press. https://doi.org/10.1017/
CBO9781139547369.019
Journal of Technology Enhanced Learning, 4(5/6), 304–317. https://doi.org/10.1504/
IJTEL.2012.051816
Fredericksen, E., Pickett, A., Shea, P., Pelz, W., & Swan, K. (2000). Student satisfaction and per-
ceived learning with on-line courses: Principles and examples from the SUNY learning net-
work. Journal of Asynchronous Learning Networks, 4(2), 7–41.
Giesbers, B., Rienties, B., Tempelaar, D., & Gijselaers, W. (2013). Investigating the relations
between motivation, tool use, participation, and performance in an e-learning course using web-
videoconferencing. Computers in Human Behavior, 29(1), 285–292. https://doi.org/10.1016/j.
chb.2012.09.005
Heublein, U. (2014). Student drop-out from German higher education institutions. European
Journal of Education, 49(4), 497–513. https://doi.org/10.1111/ejed.12097
Horton, D., & Craig, M. (2015). Drop, fail, pass, continue: Persistence in CS1 and beyond in
traditional and inverted delivery. SIGCSE’15. In Proceedings of the 46th ACM Technical
Symposium on Computer Science Education (pp. 235–240) Kansas City, MO. https://doi.
org/10.1145/2676723.2677273
Isen, A. M., & Reeve, J. (2005). The influence of positive affect on intrinsic and extrinsic motiva-
tion: Facilitating enjoyment of play, responsible work behavior, and self-control. Motivation
and Emotion, 29, 295–323. https://doi.org/10.1007/s11031-006-9019-8
Isleib, S., & Heublein, U. (2017). Ursachen des Studienabbruchs und Anforderungen an die
Prävention. Empirische Pädagogik, 30. Jahrgang (Heft 3/4), 513–530. Landau in der Pfalz,
Germany: Verlag Empirische Pädagogik.
Joo, Y. J., Lim, K. Y., & Kim, E. K. (2011). Online university students’ satisfaction and per-
sistence: Examining perceived level of presence, usefulness and ease of use as predictors in
a structural model. Computers & Education, 57(2), 1654–1664. https://doi.org/10.1016/j.
compedu.2011.02.008
Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction.
Educational Psychology Review, 19, 509–539. https://doi.org/10.1007/s10648-007-9054-3
Kalyuga, S. (2011). Cognitive load theory: How many types of load does it really need? Educational
Psychology Review, 23(1), 1–19. https://doi.org/10.1007/s10648-010-9150-7
Kori, K., Pedaste, M., Leijen, Ä., & Tõnisson, E. (2016). The role of programming experience
in ICT students’ learning motivation and academic achievement. International Journal of
Information and Education Technology, 6(5), 331–337. https://doi.org/10.7763/IJIET.2016.
V6.709
Kori, K., Pedaste, M., Tõnisson, E, Palts, T., Altin, H., Rantsus, R., … Rüütmann, T. (2015). First-
year dropout in ICT studies. In Proceedings of IEEE Global Engineering Education Conference
(pp. 444–452). https://doi.org/10.1109/EDUCON.2015.7096008
Law, K. M. Y., Lee, V. C. S., & Yu, Y. T. (2010). Learning motivation in e-learning facilitated
computer programming courses. Computers & Education, 55(1), 218–228. https://doi.
org/10.1016/j.compedu.2010.01.007
Legris, P., Ingham, J., & Collerette, P. (2003). Why do people use information technology? A
critical review of the technology acceptance model. Information & Management, 40, 191–204.
https://doi.org/10.1016/S0378-7206(01)00143-4
Leppink, J., Paas, F., van Gog, T., van der Vleuten, C. P. M., & van Merriënboer, J. J. G. (2014).
Effects of pairs of problems and examples on task performance and different types of cognitive
load. Learning and Instruction, 30, 32–42. https://doi.org/10.1016/j.learninstruc.2013.12.001
Leutner, D. (2002). Adaptivität und Adaptierbarkeit multimedialer Lehr- und Informationssysteme.
In L. Issing & P. Klimsa (Eds.), Information und Lernen mit Multimedia und Internet (pp. 115–
125). Weinheim, Germany: Beltz.
Leutner, D. (2014). Motivation and emotion as mediators in multimedia learning. Learning and
Instruction, 29, 174–175. https://doi.org/10.1016/j.learninstruc.2013.05.004
Levy, Y. (2007). Comparing dropouts and persistence in e-learning courses. Computers &
Education, 48, 185–204. https://doi.org/10.1016/j.compedu.2004.12.004
Liaw, S.-S. (2008). Investigating students’ perceived satisfaction, behavioral intention, and effec-
tiveness of e-learning: A case study of the Blackboard system. Computers & Education, 51,
864–873. https://doi.org/10.1016/j.compedu.2007.09.005
López-Pérez, M. V., Pérez-López, M. C., & Rodríguez-Ariza, L. (2011). Blended learning in higher
education: Students’ perceptions and their relation to outcomes. Computers & Education, 56,
818–826.
Marks, R. B., Sibley, S. D., & Arbaugh, J. B. (2005). A structural equation model of predictors
for effective online learning. Journal of Management Education, 29(4), 531–563. https://doi.
org/10.1177/1052562904271199
Mayer, R. E. (2014). Incorporating motivation into multimedia learning. Learning and Instruction,
29, 171–173. https://doi.org/10.1016/j.learninstruc.2013.04.003
Melis, E., Andrès, E., Büdenbender, J., Frischauf, A., Goguadze, G., Libbrecht, P., … Ullrich, C.
(2001). ActiveMath: A generic and adaptive web-based learning environment. International
Journal of Artificial Intelligence in Education, 12, 385–407.
Moreno, R. (2006). When worked examples don’t work: Is cognitive load theory at an impasse?
Learning and Instruction, 16(2), 170–181.
Narciss, S., Proske, A., & Koerndle, H. (2007). Promoting self-regulated learning in web-
based learning environments. Computers in Human Behavior, 23(3), 1126–1144. https://doi.
org/10.1016/j.chb.2006.10.006
Paas, F., Tuovinen, J. E., van Merriënboer, J. J., & Darabi, A. A. (2005). A motivational perspec-
tive on the relation between mental effort and performance. Optimizing learner involvement in
instruction. Educational Technology Research and Development, 53(3), 25–34.
Papamitsiou, Z., & Economides, A. (2014). Learning analytics and educational data mining in
practice: A systematic literature review of empirical evidence. Educational Technology &
Society, 17(4), 49–64.
Park, B., Plass, J. L., & Brünken, R. (2014). Cognitive and affective processes in multimedia learn-
ing. Learning and Instruction, 29, 125–127. https://doi.org/10.1016/j.learninstruc.2013.05.005
Park, J.-H., & Choi, H. J. (2009). Factors influencing adult learners’ decision to drop out or persist
in online learning. Educational Technology & Society, 12(4), 207–217.
Plass, J. L., Moreno, R., & Brünken, R. (2010). Cognitive load theory. Cambridge, MA: Cambridge
University Press. https://doi.org/10.1017/CBO9780511844744
Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education:
A systematic review of meta-analyses. Psychological Bulletin, 143(6), 565–600. https://doi.
org/10.1037/bul0000098
Schnotz, W., & Kürschner, C. (2007). A reconsideration of cognitive load theory. Educational
Psychology Review, 19(4), 469–508. https://doi.org/10.1007/s10648-007-9053-4
Schulmeister, R. (2017). Presence and self-study in blended learning. eLeed, 12,
urn:nbn:de:0009-5-45027. https://eleed.campussource.de/archive/12/4502
Siegmund, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2014). Measuring and model-
ing programming experience. Empirical Software Engineering, 19(5), 1299–1334. https://doi.
org/10.1007/s10664-013-9286-4
Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York, NY: Springer.
https://doi.org/10.1007/978-1-4419-8126-4
Talton, J. O., Peterson, D. L., Kamin, S., Israel, D., & Al-Muhtadi, J. (2006). Scavenger hunt:
Computer science retention through orientation. In Proceedings of the 37th SIGCSE Technical
Symposium on Computer Science Education (pp. 443–447).
van Seters, J. R., Ossevoort, M. A., Tramper, J., & Goedhart, M. J. (2012). The influence of stu-
dent characteristics on the use of adaptive e-learning material. Computers & Education, 58,
942–952. https://doi.org/10.1016/j.compedu.2011.11.002
Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model:
Four longitudinal field studies. Management Science, 46, 186–204. https://doi.org/10.1287/
mnsc.46.2.186.11926
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of infor-
mation technology: Toward a unified view. MIS Quarterly, 27, 425–478. https://doi.
org/10.2307/30036540
Watson, C., & Li, F. W. B. (2014). Failure rates in introductory programming revisited. In
Proceedings of the 2014 Conference on Innovation Technology in computer science education
(ITiCSE’14) (pp. 39–44). New York, NY: Association for Computing Machinery (ACM).
Wu, J.-H., Tennyson, R. D., & Hsia, T.-L. (2010). A study of student satisfaction in a blended
e-learning system environment. Computers & Education, 55(1), 155–164. https://doi.
org/10.1016/j.compedu.2009.12.012
Zug, S., Hawlitschek, A., & Krenz, T. (2017). What are the key features of future Remote Labs?
A critical evaluation of an existing one. In FDIBA Conference Proceedings, http://www.elab.
ovgu.de/elab_media/_users/hawlitsc/Paper_FDIBA_eLab/FDIBA_eLab-p-90.pdf
Part II
Issues and Challenges for Implementing
Learning Analytics
Chapter 6
Learning Analytics Challenges
to Overcome in Higher Education
Institutions
Philipp Leitner, Markus Ebner, and Martin Ebner
1 Introduction
Over the past decade, learning analytics (LA) have received more and more atten-
tion as a rapidly growing and promising research field in the area of technology-
enhanced learning (TEL) (Ferguson, 2012; Khalil & Ebner, 2015). Since it was first
mentioned in the Horizon Report of 2012 (Johnson et al., 2012), different tools have
been used and initiatives carried out concerning different aspects of LA. Thereby,
LA is now finally reaching the point at which it will affect research and practice, as
well as policy- and decision-making (Gašević, Dawson, & Siemens, 2015).
Currently, many different definitions for the term learning analytics are accepted.
Long and Siemens (2011) defined it as “the measurement, collection, analysis, and
reporting of data about learners and their contexts, for purposes of understanding
and optimizing learning and the environment in which it occurs.” Duval (2012)
summarized LA by saying “learning analytics is about collecting traces that learners
leave behind and using those traces to improve learning.” Despite the different
approaches, all definitions of LA indicate that it should provide actionable insights
(Siemens et al., 2011).
Therefore, the purpose should remain in focus when implementing LA initia-
tives. Obviously, the potential actions strongly depend on the utilization of data and
the information contained. However, what kind of data representation is necessary
to implement LA in an institution, and what ethical and moral aspects need to be
considered? Currently, the members of the European Union are particularly strongly
affected by the enforcement of the EU General Data Protection Regulation (GDPR)
(Leitner, Broos, & Ebner, 2018). The issues of data ownership and privacy are
becoming increasingly significant (Drachsler & Greller, 2016). Therefore, the
P. Leitner (*) · M. Ebner · M. Ebner

Graz University of Technology, Graz, Austria

https://doi.org/10.1007/978-3-319-64792-0_6
92 P. Leitner et al.
location and accessibility of the data need to be kept in mind (Leitner et al., 2018).
For example, where is the data stored? On an internal or external server hosted by a
service provider? Additionally, many LA projects do not move past the prototype
phase because of issues related to transferability and scalability (Leitner, Khalil, &
Ebner, 2017). These aspects should already be considered at the beginning of the
development.
The goal of this study was to provide a practical tool that can be used to identify
risks and challenges that arise when implementing LA initiatives and how to
approach these. This gives implementers the opportunity to deal with these prob-
lems at an early stage and, thereby, not lose time or invest effort needlessly later on
when the realization of the initiative becomes critical. In this study, we identified
and categorized seven criteria for implementing successful LA initiatives. Although
we are aware that these areas are co-dependent, we addressed them individually
throughout this study.
In the remainder of this chapter, we showcase relevant and related work, placing
an emphasis on similar research questions, and extract relevant problems that gener-
ally emerge during the implementation of LA and identify possible solutions. In
Sect. 3, an overview is provided of the seven areas that are most significant when
implementing LA projects. The reason behind choosing these areas is described in
greater detail. In Sects. 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, and 3.7, we describe what we
consider to be part of these areas and what we explicitly exclude and which chal-
lenges exist and which approaches to solve appear promising in more detail. Finally,
we conclude with a discussion and remarks about future work.
2 Related Work
In this chapter, the results of a survey of previous work regarding possible chal-
lenges and solutions when implementing LA initiatives are presented. We read the
literature to find work on similar topics and determine how the authors met these
challenges and what kind of solutions and/or framework they used/proposed. The
literature review of Leitner et al. (2017) showed that, in the last few years, various
publications have been published in which parts of the challenges summarized in
our seven main categories are described. In her paper, Ferguson (2012) documented
the concerns about ethics and privacy which began to surface once tools used to
analyze student data became more powerful and readily available. She additionally
addressed four challenges, one of which was the development of a set of ethical
guidelines. Prior to this, Campbell (2007) had already defined a framework for
locating potential areas of misunderstanding in LA, which he based on definitions,
values, principles, and loyalties. Later, to clearly differentiate between ethics and
privacy, Drachsler and Greller (2016) defined ethics as “the philosophy of moral
that involves systematizing, defending, and recommending concepts of right and
wrong conduct. In that sense, ethics is rather different to privacy. In fact, privacy is
a living concept made out of personal boundary negotiations with the surrounding
6 Learning Analytics Challenges to Overcome in Higher Education Institutions 93
ethical environment.” Ferguson, Hoel, Scheffel, and Drachsler (2016) summarized

the challenges presented by the special issue of ethics and privacy in LA in 21
points, as shown in Table 6.1.
The first six challenges are related to helping learners achieve success during
their studies. Therefore, the data should or—even better—must be complete, accu-
rate, and up-to-date. It is the learner’s responsibility to ensure this. On the other
hand, the institutions also have a responsibility to ensure a state-of-the-art, valid,
and reliable evaluation process, which is carried out in an understandable way.
Challenge 7, originally derived from the field of the medical sciences (Murray,
1990), however, relates to the issue that informed consent is also needed today with
regard to LA. Students should be involved as collaborators and, therefore, give their
informed consent to data access. The obtained analysis from the data is then used to
support learning and improve the learner’s chances of success. Challenges 8–10 are
concerned with the rights and interests of students and teachers, as well as the
responsibility held by educational institutions to safeguard and protected these.
Providing access to data and allowing the possibility to make corrections and/or file
a complaint also play important roles (Rodríguez-Triana, Martínez-Monés, &
Villagrá-Sobrino, 2016). The next two challenges are concerned with providing
equal access to education for everyone (Challenge 11) and a fair and equally applied
legal system for all citizens (Challenge 12). Challenges 13–19 are related to
data protection and place a focus on the legal responsibility for data security.
Table 6.1 Learning analytics challenges and dimensions (Ferguson et al., 2016)
1. Use data to benefit learners
2. Provide accurate and timely data
3. Ensure accuracy and validity of analyzed results
4. Offer opportunities to correct data and analysis
5. Ensure results are comprehensible to end users
6. Present data/results in a way that supports learning
7. Gain informed consent
8. Safeguard individuals’ interests and rights
9. Provide additional safeguards for vulnerable individuals
10. Publicize mechanisms for complaint and correction of errors
11. Share insights and findings across digital divides
12. Comply with the law
13. Ensure that data collection, usage, and involvement of third parties are transparent
14. Integrate data from different sources with care
15. Manage and care for data responsibly
16. Consider how, and to whom, data will be accessible
17. Ensure data are held securely
18. Limit time for which data are held before destruction and for which consent is valid
19. Clarify ownership of data
20. Anonymize and de-identify individuals
21. Provide additional safeguards for sensitive data
The harvested data are the property of another person, and the institution must
assure data protection and security. The last two challenges are concerned with the
privacy of data and how data should be used and treated (cf. Ferguson et al., 2016).
To meet these challenges, the scientific community already takes a variety of
approaches with regard to data protection and ethics in connection with LA
(Ferguson et al., 2016): for example, a code of conduct was developed that can be
used as a taxonomy of ethical, legal, and logistical issues for LA (Sclater, 2016).
Rodríguez-Triana et al. (2016) expanded the recommendations of Sclater’s (2016)
code and added consent, transparency, access, accountability, data protection, valid-
ity, and avoidance of adverse effects. A framework for privacy and data protection
has been proposed by Steiner, Kickmeier-Rust, and Albert (2016). Cormack (2016)
has published a paper which deals with European data protection practices and in
particular with the transparent communication of data usage. The codes of conduct
and frameworks developed so far have been supplemented by Berg, Mol, Kismihók,
and Sclater (2016) with tools and approaches that enable us to put them into prac-
tice. Khalil and Ebner (2016) focused on the de-identification and anonymization of
data for analysis within LA. An examination of the study conducted by Hoel and
Chen (2016) shows that the discussion on data exchange and Big Data in education
is still at an early stage. Prinsloo and Slade (2016) addressed the rights and prob-
lems of students as well as the supervising institutions, arguing that the primary
responsibility for LA system providers is to promote individual autonomy and pro-
vide each individual learner with enough information to make informed decisions
(cf. Ferguson et al., 2016). To help institutions enter the area of LA, Drachsler and
Greller (2016) developed a checklist (the DELICATE checklist), which helps users
identify and examine possible problems and obstacles that could hinder the intro-
duction of LA in the education sector in advance. The term DELICATE stands for
the eight points that need to be considered if one wants to use LA (see Drachsler &
Greller, 2016).
In the context of the SHEILA (Supporting Higher Education to Integrate
Learning Analytics) project, a team of research and institutional leaders in LA is
currently developing a policy framework for formative assessment and personalized
learning. They have used the Rapid Outcome Mapping Approach (ROMA) and vali-
dated their outputs through case studies. Their focus has been placed on the devel-
opment of a policy agenda for higher educational institutions by taking advantage
of direct engagement with the different stakeholders (Macfadyen, Dawson, Pardo,
& Gasevic, 2014). Tsai and Gasevic (2017) identified several challenges related to
strategic planning and policy:
• Challenge 1—Shortage of leadership: The leadership lacks the capabilities to
guarantee the implementation of LA in the environment of the institution.
Therefore, different stakeholders and their interests must be taken into account to
ensure their commitment to the topic. Otherwise, these stakeholders may become
stoppers.
• Challenge 2—Shortage of equal engagement: There are gaps between the vari-
ous stakeholders within institutions with regard to understanding LA. Teams
who work in technical areas showed the highest level of understanding, while
other teams did not know much about LA. This can be seen as a barrier for the
institutional acceptance of LA.
• Challenge 3—Shortage of pedagogy-based approaches: When designing LA
tools, it is also important to include pedagogical approaches in the LA process.
Institutions tend to focus more on technical aspects rather than pedagogical
aspects.
• Challenge 4—Shortage of sufficient training: As highlighted in challenge 2,
there is a lack of understanding of how LA can be beneficial to all stakeholders.
A good staff training program, which helps them improve their skill sets on this
topic, is key to success.
• Challenge 5—Shortage of studies empirically validating the impact: A budget
must be allocated to support LA. Therefore, senior staff members need a basis
for the decision-making to do so. However, the evaluation of the success of LA
seems to be a challenging task.
• Challenge 6—Shortage of learning analytics-specific policies: Institutions have
regulations regarding data and ethics. However, few institutions have codes of
practice for LA. This lack of clear guidance regarding LA practice needs to be
addressed.
Furthermore, Tsai and Gasevic (2017) reviewed eight policies (Jisc, LACE,
LEA’s Box, NUS, NTU, OU, CSU, USyd) concerning their suitability based on the
six abovementioned challenges. Although the policies partially lack pedagogical
approaches, guidance for the development of data literacy, and evaluations of the
effectiveness, they serve as valuable references for institutions interested in estab-
lishing LA in their field of work. Particularly institutions that are interested in devel-
oping their own practice guidelines for LA (Tsai & Gasevic, 2017) can benefit from
the findings.
In our research, we found that several publications have focused on different
aspects of this topic. Overall, it can be said that the creation of clear guidelines
based on a code of practice is needed when planning to introduce LA in an institu-
tion. Our knowledge and thoughts are summarized in seven main categories and
presented in the next section.
3 Seven Main Categories for LA Implementations
Bearing in mind the related work, the issues identified during previous research, as
well as our own experiences with implementing LA projects and initiatives in higher
education (De Laet et al., 2018; Leitner et al., 2018; Leitner & Ebner, 2017), we
developed a framework for LA implementations. Based on the results of the litera-
ture review and a workshop with LA specialists, stakeholders, and researchers, dif-
ferent issues were identified. These core issues were discussed, were verified, and
could be categorized into seven main areas (Fig. 6.1).
In the following subsections, we explain the seven categories in detail, pointing
out the challenges they present and providing possible solutions.
Fig. 6.1 Framework with seven main categories for LA initiatives
3.1 Purpose and Gain
The expectations related to improving learning and teaching when talking about LA
in higher education are extremely high. However, at an institutional level, the line
between LA and academic analytics is blurred. Therefore, it is advisable to distin-
guish between the different stakeholders with regard to the various goals and perspec-
tives of stakeholders such as learners, educators, researchers, and administrators.
The goal of the learners is to improve their performance. LA supports this by
providing adaptive feedback, recommendations, and individual responses on their
learning performance (Romero & Ventura, 2013).
The educators are interested in understanding the students’ learning processes;
understanding social, cognitive, and behavioral aspects; reflecting on their teaching
methods and performance; as well as optimizing their instructions to achieve a bet-
ter learning outcome (Leitner et al., 2017). They want to be able to assess the
students’ activities more effectively and draw conclusions to find out where they
need to take more action to improve the students’ learning performance.
Researchers use the data to develop theoretical models for new and improved
teaching and learning methods. This includes pursuing the goal to predict future
learning paths and support the needs of learners more appropriately. Educational
technologists and researchers in the field of pedagogy review existing didactical
models and develop new didactical ones by carrying out field studies in classrooms.
For this reason, they conduct research continuously and adapt LA techniques based
on the data collected to meet the new expectations of the younger generation.
Administrators are interested in implementing their agendas in a more efficient
environment. Their aim is to offer students a more pleasant and efficient learning
environment. Additional goals are to reduce the failure rates and numbers of drop-
outs, increase performance, and, thus, optimize and improve the curricula. The gov-
ernment is responsible for the enforcement of data privacy and data protection issues.
Challenges may occur when dealing with the different stakeholders. If stake-
holders are confronted with hard facts without being asked for their thoughts and
opinion first, they may rebel. Additionally, despite the generally positive intentions
of those introducing LA into institutions, stakeholders often have their own thoughts
about LA. Students and teachers might be afraid that the results of the analytics, if
made public, would put them in bad positions. Or even worse, the representatives of
the different stakeholders have their own in-game and expect to use the results to
expose their counterparts. Therefore, it is necessary to make the goals of the LA
initiative transparent, clarifying exactly what is going to happen with the informa-
tion and explicitly what is not.
When the purpose of the LA initiative is very clear from the beginning, this does
not seem to be a problem. However, if it is not, the situation might become compli-
cated when attempting to explain the LA initiative to the stakeholders. Fuzziness
not only puts you in a weak negotiating position but can also become a major prob-
lem when the stakeholders try to bring you over onto their side. Therefore, imple-
menters need to specify and adhere to the ultimate objective of the LA initiative.
3.2 Representation and Actions
The purpose of LA is to use the data collected to optimize the students’ learning
processes and improve teaching. The aim is to make learning itself more predictable
and visible. Actions derived from this can serve as a basis for developing a cata-
logue of measures to support risk groups and provide them with better assistance
during their study. Based on this, recommendations are made to support learners
and encourage them to reflect on their behaviors. The information is provided within
a suitable environment and clearly visualized as being included in the student’s
personalized learning process. The personalization of the working environment and
the associated advantages are placed in the foreground. This should have the effect
of motivating the learner in terms of improving their attitude. The feedback received
is intended to stimulate reflection and lead to a shift in goals and the associated
improvement in learning success.
Choosing the right environment for the learner’s feedback and the correct visual-
ization technique can present a large challenge for all parties involved. Due to the
quantity of data harvested and the focus placed on quantitative metrics, teachers
sometimes consider LA to be antithetical to an educational sense of teaching.
Dashboards with performance metrics are becoming increasingly popular in these
contexts (Clow, 2013). The interpretation of this data can sometimes seem incredi-
bly difficult if it has not been properly prepared before it is presented to the student.
Therefore, it can be better not to provide the student with all information related to
the learning outcome. A mentor can discuss the results with the student. However,
university staff who are acting as mentors need specialized training so they can
interpret the data and pedagogical and psychological skills to discuss his/her results
with the student and provide deeper insights about the data.
3.3 Data
Universities and schools are constantly analyzing data from their students for a vari-
ety of reasons. LA can, therefore, be seen as an innovative continuation of this prin-
ciple, applied to make use of the advantages of modern technology and the various
data sources available today. The data can be examined and analyzed for their impact
in the learning context to improve the quality of learning and teaching, as well as
enhance the chances of the students’ success. Of course, universities require the indi-
vidual’s permission to collect and evaluate sensitive data for the purpose of
LA. Students must be made aware of the purpose of collecting and the process of
analyzing the data. Consent is mandatory for the use of these data, which then can be
used as a basis for strategic decisions by the various stakeholders. Teachers are able to
monitor and analyze a student’s behavior and actions while they are interacting with
the learning management system. Thus, teachers are provided with insights into the
student’s learning culture, for example, whether the student has submitted all of their
assignments or how actively they engage in their studies. Derived models can be used
to provide better student support so that they can reach their goals more efficiently.
Students leave various data traces while using the university infrastructure. The
data collected will be used together with statistical models and methods for the pur-
pose of LA when a benefit for student learning is expected. Students may want to
know why they have been categorized as potential risk candidates in specific courses.
Therefore, the data and models used must be communicated and explained to them
by trained staff in a comprehensible way to provide them with guidance. Access to
that data must be secured, and only a few staff members are allowed to have access
permissions to students’ data. The institutions must enact policies that address data
protection and access. Students must be informed of who has access to the data.
The data used will not only have impact on the individual student but also influ-
ence the practice of teaching at the university. Therefore, the data have to be re-
evaluated over time and adjusted to meet the new demands. Furthermore, to ensure
the best support and quality of the data, students need to keep their data up-to-date.
Giving them the (proactive) opportunity to check and update their data supports
them and the university during this process. Additionally, all of these points must
comply with the GDPR and local data protection acts.
A policy needs to be created for LA that aligns with the organization’s core prin-
ciples. Transparent communication about where the data are stored, what is being
done to ensure data security and privacy, and how the data are evaluated and used (and
by whom) is essential. Responsible handling of the students’ data by all stakeholders,
especially university staff members, is important. Further training and skill-building

for responsible tutors/mentors in interpretation of the students’ data and action-taking
in this context are required. Interventions should be recommended to the student,
which are based on the collected data, and must be delivered in a transparent and
comprehensible way (e.g., which methods and models have been used) to ensure
broad students acceptance and engagement. Students need to clearly understand how
the data are interpreted and manipulated and which techniques are used to ensure
optimal handling and verifiable recommendations.
3.4 IT Infrastructure
IT infrastructure refers to a set of information technology (IT) components such as

hardware, software, network resources, and services that are the foundation for the
operation and management of an enterprise IT environment (Laan, 2011). This
infrastructure allows organizations in higher education to deliver IT services to its
students, teachers, and administrative staff. This IT infrastructure is usually internal
and deployed with in-house facilities, but it is possible to commission an external
provider. However, IT infrastructure is the basis for any LA measurements and,
therefore, has to be considered carefully.
Why is it important to think about the IT infrastructure? To understand its rele-
vance, it is necessary to know where the data is located. Therefore, we can distin-
guish between two different scenarios. First, the data are stored and processed in a
university-owned service center. Thereby, the responsibilities and liabilities are
located at the university itself, and national and organizational rules must be obeyed.
This scenario has the advantage that access to the data and the data ownership are
located at the university, which makes it easier to work with the data. However, it
also presents some disadvantages, such as the fact that initiatives with special tech-
nology requirements need to comply with the standardized rules held by the internal
service provider. Also, the cost-benefit ratio should be kept in mind because hosting
and support services privately might be more expensive than outsourcing.
The second scenario concerns working with external service providers. In this
scenario, individual solutions can be applied, as many providers are available that
might meet the specific needs. In contrast to the internal service center of a univer-
sity, external service providers can concentrate their efforts on their smaller and
highly specialized digital product. Furthermore, the costs that arise can easily be
estimated and should be much lower than providing a private, individual solution.
The negative aspects of working with an external service provider are related to
issues of access and data ownership as well as meeting the necessary security stan-
dards when working with sensitive data, such as student performance data.
Regardless of whether one works with an internal or external service provider, it
takes time to establish the appropriate basis. Therefore, efforts should be made from
the beginning to search for possible solutions to set up the necessary IT infrastruc-
ture and contact and establish connections with relevant people (Leitner et al.,
2018). This will save time and resources when the implementation of an LA initia-
tive becomes critical.
3.5 Development and Operation
This category combines the process of developing and operating LA initiatives. It

includes a wide range of different developments, from designing a simple question-
naire to developing an enterprise software solution. Additionally, the activities cover
research on and the development, prototyping, modification, reuse, reengineering,
monitoring, and maintenance of LA initiatives or projects.
Once the first prototype has been produced, implemented in a real-life context,
and evaluated, the discussion can proceed to the next step. How can the prototype be
realized? How can it move from the prototype phase to the production phase? These
are quite critical questions because new tasks and challenges will appear. For exam-
ple, the scalability of the implementation has to be taken into account. The number
of learners may differ arbitrarily, and this can lead to a complete new concept for the
existing IT infrastructure. Furthermore, processes which were first created manually
must be redefined so that they can be performed at least semiautomatically or com-
pletely automatically.
Even if student data is stored, this is typically done via different information
systems. Normally, several information systems are responsible for performing dif-
ferent tasks and, therefore, storing the data—in different formats, on different serv-
ers, and with different data owners. The efforts that are required to receive and
manage all data can be stressful and tedious. Additionally, converting raw data into
a useful format can be another big challenge. This is a highly complicated process,
which needs thorough planning and a consistent final implementation. Additionally,
the implementation should include working in different layers and should probably
be implemented in a modular manner. In doing so, any changes in the circumstances
of the different, associated information systems can easily be adapted.
From the first stages of any learning measurement, we suggest that the scope
should be specified in detail. Will the LA be established merely for testing and to
obtain initial impressions, or will it be implemented at a university-wide level?
Scalability is maybe one of the most frequently underestimated problems in today’s
IT industry. Furthermore, we strongly emphasize planning the LA implementation
beforehand, so that the costs can be estimated as exactly as possible. A distinction
must be made as to whether processes have to be carried out manually, semiauto-
matically, or fully automatically.
3.6 Privacy
The term privacy is defined as an intrinsic part of a person’s identity and integrity
and constitutes one of the basic human rights in developed countries, as it should be
an established element of the legal systems (Drachsler & Greller, 2016). All LA
implementations have to ensure the privacy of the involved parties. Learners must
trust the final systems, and, therefore, keeping information private is of the utmost
importance. Additionally, depending on the country where the higher education

institution is situated, different regulations in addition to the General Data Protection
Regulation (GDPR), which is applicable in Europe, are enforced. Organizations
have to deal with different tasks while finding a suitable legal framework that covers
the GDPR. They could take another tack and start to minimize the data harvested
and/or take actions to anonymize or pseudo-anonymize their data.
Nevertheless, even when keeping privacy in mind when handling LA initiatives
from the beginning, the situation can become highly complex. For example, by
merging different data sources, new and surprising results can be visualized, and,
therefore, new insights which were never intended can be provided. Due to the fact
that universities are huge institutions, there is a high risk that unauthorized people
receive access to these data interpretations.
Another large problem that is closely related to privacy is the fact that a person/
learner is reduced to their stored data. Society is made up of individuals, so every
situation has to be considered in a differentiated way. For example, an activity pro-
file can be created, but we will never know exactly how those activities actually took
place and to which extent. The reduction of people to categories and profiles can be
particularly dangerous, because a learning individual could be reduced to a merely
few parameters. Since society seems to like to fall back on so-called facts, the deri-
vation of causal connections on the basis of learning algorithms always needs to be
critically questioned. This also means that gaps in data need to be analyzed and
handled.
Finally, the general lifetime of personal data is a topic that requires further dis-
cussion. The data may be interesting at a time when the activities and learning out-
comes are relevant, but the data may no longer be relevant in the future. Arguments
for keeping data could be presented for the purposes of training algorithms and
machine learning. Improvements could also be made by providing a larger data
resource. However, these steps should only be carried out with absolute anonymiza-
tion of the data. Khalil and Ebner have shown how this should be done (Khalil &
Ebner, 2016).
First, privacy is a fundamental right of every person and must be respected. This
means that any LA implementation must take this into account from the very begin-
ning. However, this is often difficult or perhaps not clarified at all, because complex
situations can arise as a result of data mergers. Therefore, we suggest working with
the highest possible level of transparency, because this encourages confidence: the
learners know what happens to their data and what statements can be made. At the
same time, unauthorized people cannot be allowed to access the data, and the per-
sonnel need to be well-trained in terms of data interpretation but also know how to
deal with questions about privacy. If doubts with regard to privacy arise, the LA
measure must always be omitted.
Finally, we would like to point out once again that the mere use of data—i.e.,
“facts”—will not be sufficient to adequately represent such a complex situation as a
learning process. LA is only an auxiliary tool that can be used to gain a better under-
standing of this process.
3.7 Ethics
Ethics is defined as a moral code of norms and conventions that involves systematiz-
ing, defending, and recommending concepts of right and wrong conduct. It exists
external to a person in society (Drachsler & Greller, 2016). In the context of LA,
various ethical and practical concerns arise as the potential exists to harvest person-
alized data and intervene at an individual level (Prinsloo & Slade, 2015). Therefore,
privacy poses as a major challenge for the implementation of LA initiatives.
Additionally, working with sensitive data presents a particular challenge.
Sensitive data includes information on medical conditions, financial information,
religious beliefs, or sexual orientation, but also about student performance. If made
public, such information could result in harm to that particular person. Therefore, it
is necessary to ensure restrictions on who has access to the information and for
which purpose(s) it is used.
Some questions arise when looking at data from an ethical point of view. First,
which data of a person are permitted to be harvested, used, and processed, regard-
less of whether they are a student or an educator? Second, which information can be
communicated to someone, and what may be the resulting consequences? These are
increasing concerns in the context of ethics, because LA enables the improvement
of accuracy of the predictions for different learning profiles by combining different
data sources. The LA implementers must find a suitable way to meet high ethical
standards and ensure a beneficial outcome for all stakeholders.
Another important point is the option to opt in and opt out for participants from
harvesting, storing, and processing the individual data of a single person. However,
how should institutions relying on LA deal with students who take the right to opt
out? When implementing LA in an institution, it is advisable to involve all stakehold-
ers at an early state in the process of creating rules and the legal framework for the
use of data. Transparency is key, as well as understanding the different needs of the
interesting groups involved in the process. All intentions, goals, and benefits for har-
vesting and using the data have to be explained in a clear and comprehensible way to
all stakeholders. The consent for using the data begins with the log-in into the sys-
tem, which tracks data from their users. During this process, the consent of all parties
involved must be communicated. In this context, the areas in which the data will be
used must be clearly communicated. During discussions, the possibilities of interpre-
tation of the provided information need to be described to prevent misunderstandings
and incorrect decisions. As a precautionary measure, the institutions can introduce
codes of conduct and procedures that provide initial support on this subject.
At the Learning Analytics and Knowledge Conference 2018 (LAK18) in Sydney,
a draft code of ethics v1.0 was presented (Lang, Macfadyen, Slade, Prinsloo, &
Sclater, 2018). This document may be considered as a foundation for ethical matters
when implementing LA initiatives. Additionally, a legal counsel could offer their
advice when the interpretation of a topic or situation seems unclear. The European
Learning Analytics Exchange (LACE) project offers workshops on ethics and pri-
vacy in LA (EP4LA). The LACE project also plays a key role in advancing the
issues on the ethical dilemmas of using LA.
4 Conclusion
Within higher education institutes, researchers are still full of enthusiasm and excite-
ment about LA and its potential. Furthermore, LA is now at the point at which affects
research, practice, and policy- and decision-making equally (Gašević et al., 2015).
However, to facilitate successful LA initiatives, a few things have to be kept in
mind. In this chapter, we presented seven main criteria, which can be used for initial
orientation when implementing LA. The order of appearance was intentionally cho-
sen, although the order of application depends on the implementer.
We hope that the classification of the seven main criteria, the presented chal-
lenges, and the approaches that can be taken to overcome them will be helpful to
implementers of LA initiatives. We are aware that the presented examples cover
only a small range of the challenges an implementer might encounter, but we hope
the results of this study can help researchers and educators understand the bigger
picture and become aware of other potential issues.
In future research, we plan to investigate the seven categories in more detail
to identify different examples and validate our framework to foster future LA
measurements.
Acknowledgments This research project is co-funded by the European Commission Erasmus+

program, in the context of the project 562167EPP-1-2015-1-BE-EPPKA3- PI-FORWARD.
Please visit our website http://stelaproject.eu.
References
Berg, A. M., Mol, S. T., Kismihók, G., & Sclater, N. (2016). The role of a reference synthetic data
generator within the field of learning analytics. Journal of Learning Analytics, 3(1), 107–128.
Campbell, J. P. (2007). Utilizing student data within the course management system to determine
undergraduate student academic success: An exploratory study. PhD, Purdue University, West
Lafayette, IN.
Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683–695.
Cormack, A. N. (2016). A data protection framework for learning analytics. Journal of Learning
Analytics, 3(1), 91–106.
De Laet, T., Broos, T., Verbert, K., van Staalduinen, J.-P., Ebner, M. & Leitner, P. (2018). Involving
stakeholders in learning analytics: Opportunity or threat for learning analytics at scale? In
Proceedings 8th international conference on learning analytics & knowledge (pp. 602–606).
Sydney.
Drachsler, H., & Greller, W. (2016). Privacy and analytics - it’s a DELICATE issue: A checklist
to establish trusted learning analytics. In Proceedings of the 6th international conference on
learning analytics and knowledge (pp. 89–96). https://doi.org/10.1145/2883851.2883893
Duval, E. (2012). Learning analytics and educational data mining, In Erik Duval’s weblog, 30
January 2012. Retrieved from https://erikduval.wordpress.com/2012/01/30/learning-analytics-
and-educational-data-mining/. Accessed 16 March 2018.
Journal of Technology Enhanced Learning (IJTEL), 4(5/6), 304–317. https://doi.org/10.1504/
IJTEL.2012.051816
Ferguson, R., Hoel, T., Scheffel, M., & Drachsler, H. (2016). Guest editorial: Ethics and privacy in
learning analytics. Journal of learning analytics, 3(1), 5–15. https://doi.org/10.18608/jla.2016.31.2
Hoel, T., & Chen, W. (2016). Privacy-driven design of learning analytics applications - explor-
ing the design space of solutions for data sharing and interoperability. Journal of Learning
Analytics, 3(1), 139–158.
Johnson, L., Adams, S., Cummins, M., Estrada, V., Freeman, A., & Ludgate, H. (2012). The NMC
horizon report: 2012 higher education edition. The New Media Consortium.
Khalil, M., & Ebner, M. (2015). Learning analytics: principles and constraints. In Proceedings
of world conference on educational multimedia, hypermedia and telecommunications
(pp. 1326–1336).
Khalil, M., & Ebner, M. (2016). De-identification in learning analytics. Journal of Learning
Analytics, 3(1), 129–138.
Laan, S. (2011). IT infrastructure architecture: Infrastructure building blocks and concepts.
Morrisville, NC: Lulu Press.
Lang, C., Macfadyen, L. P., Slade, S., Prinsloo, P., & Sclater, N. (2018, March). The complexities
of developing a personal code of ethics for learning analytics practitioners: Implications for
institutions and the field. In Proceedings of the 8th international conference on learning analyt-
ics and knowledge (pp. 436–440). ACM.
Leitner, P., Broos, T., & Ebner, M. (2018). Lessons learned when transferring learning analytics
interventions across institutions. In Companion proceedings 8th international conference on
learning analytics & knowledge (pp. 621–629). Sydney.
Leitner, P., & Ebner, M. (2017). In P. Zaphiris & A. Ioannou (Eds.), Learning and collabora-
tion technologies. Technology in Education: 4th international conference, LCT 2017, held as
part of HCI international 2017, Vancouver, BC, Canada, July 9–14, 2017, proceedings, part II
(pp. 293–301). Cham, Switzerland: Springer.
Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education - a litera-
ture review. In Learning analytics: Fundaments, applications, and trends (pp. 1–23). Cham,
Switzerland: Springer.
Long, P. & Siemens, G., (2011). Penetrating the fog: Analytics in learning and education. Educause
Review, 46(5), 30. Retrieved from https://er.educause.edu/articles/2011/9/penetrating-the-fog-
analytics-in-learning-and-education. Accessed 16 March 2018.
Macfadyen, L. P., Dawson, S., Pardo, A., & Gasevic, D. (2014). Embracing big data in complex
educational systems: The learning analytics imperative and the policy challenge. Research &
Practice in Assessment, 9, 17–28.
Murray, P. M. (1990). The history of informed consent. The Iowa Orthopaedic Journal, 10,
104–109.
Prinsloo, P., & Slade, S. (2015). Student privacy self-management: Implications for learning ana-
lytics. In Proceedings of the fifth international conference on learning analytics and knowl-
edge (pp. 83–92). New York: ACM.
Prinsloo, P., & Slade, S. (2016). Student vulnerability, agency and learning analytics: An explora-
tion. Journal of Learning Analytics, 3(1), 159–182.
Rodríguez-Triana, M. J., Martínez-Monés, A., & Villagrá-Sobrino, S. (2016). Learning analytics
in small-scale teacher-led innovations: Ethical and data privacy issues. Journal of Learning
Analytics, 3(1), 43–65.
Romero, C., & Ventura, S. (2013). Data mining in education. Wiley Interdisciplinary Reviews:
Data Mining and Knowledge Discovery, 3(1), 12–27.
Sclater, N. (2016). Developing a code of practice for learning analytics. Journal of Learning
Analytics, 3(1), 16–42.
Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S., Ferguson,
R., … Baker, R. S. J. D. (2011). Open learning analytics: An integrated & modularized plat-
form. Proposal to design, implement and evaluate an open platform to integrate heterogeneous
learning analytics techniques. Society for Learning Analytics Research.
Steiner, C. M., Kickmeier-Rust, M. D., & Albert, D. (2016). LEA in private: A privacy and data pro-
tection framework for a learning analytics toolbox. Journal of Learning Analytics, 3(1), 66–90.
Tsai, Y. S., & Gasevic, D. (2017). Learning analytics in higher education---challenges and poli-
cies: A review of eight learning analytics policies. In Proceedings of the seventh international
learning analytics & knowledge conference (pp. 233–242). New York: ACM.
Chapter 7
The LAPS Project: Using Machine
Learning Techniques for Early Student
Support
Mathias Hinkelmann and Tobias Jordine
1 Introduction
Early support of students can be a successful instrument to improve the academic

achievements. Universities are challenged to identify students who may take advan-
tage of early support from the university’s student and learning support center. This
chapter will introduce the LAPS project (“Learning Analytics für Prüfungsleistungen
und Studienerfolg”/“Learning analytics for exams and study success”), developed
and used at the Hochschule der Medien (HdM), Stuttgart, Germany, which is set up
to cover these challenges. The particular approach of LAPS project is that com-
pleted study progressions are analyzed via machine learning techniques. These
results are compared to the grades reached by the students in their study program so
far. Since the progression of an enrolled student will statistically not differ from
students, who either completed or failed in their study program, the comparison can
be used to make an individual statement about students’ risk of failure or possibility
of success. Based on the findings of the risk calculation, students can be advised
more focused. In addition, the findings support both, under and top performing
students. Also, the results of the statistical analysis by the LAPS software can be
used as a factual basis for discussions aiming at improvements of the study pro-
grams. This chapter is structured as follows: The Sect. 2 introduces current
approaches and projects in the learning analytics research area. Section 3 provides
insights into several aspects of the presented project: the used data basis is shown,
technical implementation details are provided, and findings of the feasibility study
are presented. Further, the privacy and ethical considerations that were identified
are explained. The section concludes how LAPS can be used for academic quality
M. Hinkelmann (*) · T. Jordine

Department of Computer Science and Media, Hochschule der Medien, Stuttgart, Germany

https://doi.org/10.1007/978-3-319-64792-0_7
106 M. Hinkelmann and T. Jordine
assurance. The software and the LAPS process are reviewed in Sect. 4. Finally,
Sect. 5 concludes this chapter by providing information about possible improve-
ments concerning the LAPS project.
2 Existing Work
Based on Ferguson (2012), the research area of learning analytics has its roots in
business intelligence, web analytics, educational data mining, and recommender
systems and is defined as follows:
Learning analytics is the measurement, collection, analysis and reporting of data about
learners and their contexts, for purposes of understanding and optimising learning and the
environments in which it occurs. (“1st International Conference on Learning Analytics and
Knowledge 2011,” 2010)
Besides technological aspects like data collection and big data analysis, learning
analytics must be seen as holistic as technology, socialization, and pedagogy are
involved (Siemens, 2010). Figure 7.1 shows the process of a traditional learning
analytics approach.
As the LAPS project is aiming at the German academic system, related German
approaches are presented below.
• Study progression analysis approaches
In the case of study progression analysis, the focus is on comparing the progression
of individual students with the study plan and the progress of the entire group of
students in a study program. This approach, which is pursued at many universities,
is represented by the “tempo 30” project of the Ravensburg-Weingarten University
or “StuVa” at the University of Freiburg (Hermann & Ottmann, 2006). A special
approach to study progression analysis is module-based monitoring (Jaeger &
Sanders, 2009). In this particular approach, budget-oriented views on university
management and approaches to ensuring the quality of teaching are considered.
• Predictor models
With this approach, a presumed predictor for academic success is analyzed in detail.
At the Kiel University of Applied Sciences, for example, two study programs were
Conceptual understanding
Curriculum Social network
analysis
Semantic data Intelligent
data Analysis
Signals (failure & success)
Linked data
Data trails
Mobile
Technological
Social media
Learners Personalization &
Prediction Social
Personal learning off-put data Adaptation
environments
Inferred from Pedagogical
Learning management existing data
systems
Fig. 7.1 The learning analytics process (Siemens, 2010)

7 The LAPS Project: Using Machine Learning Techniques for Early Student Support 107
used to examine the success of studies in the first semester as a predictor for overall
success and to show that early indicators are indeed present. These indicators can
be used to control advisory services (Christensen & Meier, 2014). Other studies
(e.g., Trapmann, Hell, Weigand, & Schuler, 2007) have investigated the extent to
which school grades can be used as predictors. But recent developments in the field
of increasingly heterogeneous access to higher education show that simple predic-
tors are no longer sufficient. Instead, multidimensional predictors must be used.
3 The Laps Project
3.1 Data Basis
Efficient administration of universities requires the use of Campus-Management-

Systems (CMS) which allows to track and support the entire Student-Life-Cycle
starting from his/her application till his/her de-registration. One of the key elements
of such a CMS is the management of exams. Because of the required legal certainty,
all CMS record all exam-related student data. This means that a high data quality in
terms of students’ master data as well as collected exam data is available.
Additionally, retention periods given by legal provisions lead to a large data basis.
The development of this data basis adds value for the organization of study and
exam regulations. For doing so, personal data like type and grade of the university
entrance qualification, date of enrollment, date of de-registration, as well as detailed
information about students’ exams can be used and analyzed. Due to the similarity
of the tasks and requirements for a CMS, it can be assumed that the considerations
for the indexing of data within the CMS are not limited to specific systems (e.g., the
products of HIS eG1) but are directly transferable to other CMS. Table 7.1 shows
typical data available in Campus-Management-Systems.
Table 7.1 Typical data available in campus-management-systems

Student-related data Exam-related data
• Gender Data on academic performance
• Birthday • Name of the academic achievement
• Grade of the university entrance • Number of ECTSs awarded
qualification • Assignment to a study stage
• Type of the university entrance • Mandatory/optional/additional academic
qualification achievement
• Date of the university entrance Data on the trail
qualification • Date of exam
• Course of studies • Assignment to a semester
• Date of enrollment • Exam result
• Date of de-registration • Attempt counter
• Study success • Identification of withdrawals from exams
https://www.his.de/.
1
3.2 The LAPS Approach
An approach that uses the data described in the above is developed at the Stuttgart
Media University since 2014. Analyses that were made in advance of the develop-
ment of LAPS have shown that:
1. A combination of the type of higher education entrance qualification, the grade
of the higher education entrance qualification, and the time interval for admis-
sion to a course of study.
2. The gender of the students has a measurable influence on the probability of drop-
out. Analysis performed at HdM in advance of the LAPS project has shown that
male students have a higher risk to fail. This finding is independent from the
percentage of male and female students in a study program.
Overall, preliminary studies have shown that simple, experience-based predictors
are not sufficient to identify critical study situations (Trapmann et al., 2007) and that
a systematic and multidimensional analysis of the data (students’ master data and
data on the examination events) is required. This requires an automated, algorithmic
evaluation. The LAPS software therefore uses machine learning methods. During
machine learning, patterns are not set manually but are “learned” automatically
from existing training data.
The transfer of this approach to the analysis of study situations is possible due to
the existence of completed study progressions. Data of de-registered students are
used to determine specific study situations. This approach is explained in the fol-
lowing, starting from the general principle of an automated learning process illus-
trated in Fig. 7.2.
In the training phase, a model is trained which is used to calculate forecasts and
classifications. As training data, the LAPS software uses the enrollment data, study
progress data, and study success data of all students who have already completed
Fig. 7.2 The principle of machine learning

their bachelor studies. Each of these students is described by a data record, which in
turn consists of a list of characteristic feature/value pairs: For example, “gen-
der = female and type of university entrance qualification = Abitur” are two feature/
value pairs of the enrollment data and “not successful exams after the first semes-
ter = 1 and ECTS after the first semester = 20” are two feature/value pairs of the
study progression data. The machine learning method used in LAPS is the Apriori
algorithm (Agrawal, Imieliński, & Swami, 1993). The model calculated by this
algorithm is a set of association rules. Each association rule describes a frequently
occurring combination of characteristic feature/value pairs in the form of an impli-
cation: A → B. In general, both premise A and conclusion B can represent any
subjunctive link between characteristic feature/value pairs. A possible rule would
be, e.g., “(number of exams graded with fail after 2nd semester = 3 and number of
ECTS after 2nd semester < 20) → studies successful = no.”
A rule is only recognized as relevant by the learning algorithm and included in
the model if its support and its confidence are greater than a minimum value that can
be set by the user. The support describes the composite probability P (A, B), i.e., the
relative frequency with which the premise and conclusion occur together in a train-
ing data set. On the other hand, confidence describes the conditional probability P
(B|A), i.e., the relative frequency for which the conclusion is also true in training
data in which the premise is true. Support is therefore a measure of whether the pat-
tern consisting of A and B occurs frequently enough in the training data to be con-
sidered statistically relevant. The confidence specifies the certainty with which rule
A → B applies.
All association rules whose support and confidence are greater than the mini-
mum values form the trained model. After training, the trained model is used as
follows to predict a critical course of study: For a student to be analyzed, all cur-
rently available characteristic value pairs are entered in the system as a query. For
this specific query, the subset MS of the association rule set contained in the model
is determined, for which the premise A is fulfilled with the entered student data.
Since the conclusion of all the rules contained in the model is constantly “study suc-
cessful = no,” the confidence of each rule in MS indicates the probability with which
the respective rule predicts an unsuccessful completion. The median is calculated
from the confidence values of all rules applicable to the student (quantity of MS).
This median represents a preliminary risk probability for the student. After the pre-
liminary risk probabilities have been calculated for all students, they are adjusted
for the purpose of better differentiation by assigning the final risk score of 100% to
the student with the highest risk and 0% to the student with the lowest preliminary
risk score. The final risk values of all other students are derived from their prelimi-
nary risk values by linear scaling.
By using the Apriori algorithm, a large number of possible risk dimensions with
different characteristics can be defined for the analyses in LAPS. In this definition,
it is not required to consider the relevance of the analysis dimension or characteris-
tics. This task is performed during the training phase, in which the relevance of
combinations of these characteristics is determined. The following list shows the
currently used risk dimensions used by the LAPS software:
Dimensions of personal data:

• Age at beginning of study
• Gender
Dimensions of educational biography:
• Type of the university entrance qualification
• Date of the university entrance qualification
• Time interval between acquisition of the university entrance qualification and
start of studies
Study course analysis dimensions by semester:
• Total of achieved ECTS points
• Achieved average grade
• Number of failed exams
• Number of successful exams
• Number of deferred exams
• Frequency of nonappearances in enrolled exams
The course of study dimensions grouped by semesters is additionally assigned
different characteristics. The risk dimension sum of the achieved ECTS credits is
analyzed after the first semester of studies with the following characteristics:
• <10 ECTS credits
• More than 30 ECTS credits
This results in more than 200 possible individual risk characteristics, which are
linked in the training phase on the basis of completed courses of study and lead to
the analysis model, which comprises several thousand combinations of risk dimen-
sions. In the current version of LAPS, the risk dimensions and characteristics can be
configured. It is the responsibility of user of the system to define a threshold for the
predicted risk value above which the affected students are automatically classified
as critical. In LAPS, student and examination data is updated every semester via a
file upload interface incrementally. After the import, a training phase takes place
automatically, which is followed by the analysis of the currently enrolled students.
The described recognition of critical study progressions by predicting the risk of
termination does not represent the only application of the model trained in
LAPS. The learned rules are also able to identify typical patterns of under- and
overstraining or frequent postponements of examinations.
Besides the functionality to identify risks, LAPS is also capable to identify study
progressions with a high potential. This information can be used to support top per-
forming students, e.g., with a fellowship or additional classes. To identify these
students, achieved ECTS credits per semester are calculated and compared with set
point of ECTS credits. If the achieved ECTS and the current average grade are sig-
nificantly better than the mean of the cohort, respectively, the set point of ECTS
credits, the student is identified as a top performer.
3.3 Feasibility Study and Use in Consultations with Students
At an earlier stage of the project, students’ risk data were analyzed and used in con-
sultation situations by staff members of the student support center and course lead-
ers. In contrast to the current version of LAPS, students were directly contacted by
the users of the system when a risky study progression was identified. This version
of the tool only supported the identification of risks and was not able to detect posi-
tive study progressions.
Having this setup, students were invited for a consultation discussion. The results
of the risk analysis served as an evidence-based foundation of this talk and helped
students to understand their situation. This was especially useful when students had
a different impression on their study progression. It was found out that by using the
LAPS software, students can be advised at an earlier stage of their studies and can
be one addition to reduce students’ dropouts as additional support like trainings or
adjustments of the study progression can be offered.
Users’ feedback of this early version of LAPS was positive. It was liked that in
contrast to traditional grade overviews, the LAPS profiles are much more detailed
and potential risks are immediately visible. This allows to develop individual coun-
teractions. But the feasibility study also showed that the handling with students’
personal data was not ideal, since lecturers (i.e., persons who do the grading as well)
can access and view students’ risk details without their permission. This is why it
was required to define premises for the privacy and ethics for the project, which will
be explained in detail in the following section.
3.4 Privacy and Ethics
The LAPS software serves to create an evidence-based discussion basis with stu-
dents at an early stage of their studies. This evidence-based approach contrasts with
legitimate data privacy aspects. For the LAPS project, privacy and ethical premises
are a foundation of the whole project. These premises are voluntariness, self-
determination and self-responsibility, respecting individuality, confidentiality, as
well as anonymity and are taken into account in several ways, which are explained
in the following. Figure 7.3 provides an overview of the LAPS data access process.
When students de-register for any reason, their personal data is no longer visible
for any user of the system. In the case of enrolled students, students must opt-in to
be considered by the system. Only with their explicit agreement it is possible to
view their personal data and risk analysis. Students always have the chance to
change their decision whether they take part or not. New students can take part at
the LAPS project during enrollment, whereas current students are informed via
e-mail. Transparency is very important as students get informed how their data is
used. For example, in advance of the opt-in, a privacy information sheet that explains
the use of data is presented to each student. Additionally, the project is presented at
the general student meeting each semester as well as an information booth where
students can ask project members about LAPS once a semester.
Fig. 7.3 The LAPS process
Access to the data is strictly limited by the limitation of the user group. In the
case of the HdM, course leaders have access to the personal student data of their
respective course after they have taken part at a LAPS consultation introduction
workshop. This workshop is aiming to help course leaders to understand the data
and analysis results calculated by the system and how they can use this information
for a successful consultation. Besides course leaders, staff members of the student
support center have access to the results of the students who agreed to take part at
the LAPS project.
When a risky or an exceptional good study progression is identified, students are
informed via an automatically generated e-mail. After receiving the e-mail, the
decision is up to the students to ignore it or to choose an individual consultation
discussion with either members of the student support center or their according
course leaders. As part of the ethical and privacy decisions of LAPS, students will
not directly get the results of the analysis. This is intended to prevent self-fulfilling
prophecy: Without having the knowledge how to interpret the analysis and identify-
ing specific needs and students’ personal life situation, the results could be misun-
derstood as the algorithm is only able to do calculations based on data stored in the
Campus-Management-System.
The project is already compatible with the EU-DSGVO (General Data Protection
Regulation, 2018). This ensures that the project complies with the currently valid
data privacy laws.
3.5 Functionalities of the Tool to Support Students
The relevance of the data on the enrolled students increases with the import of the
examination results from the previous semester. At the HdM, the system is updated
in the seventh week of the lecture period of the following semester onward due to
administrative constraints. After the update of student data, a list of critical study
progressions of participating students can be reviewed. The displayed data is ini-
tially anonymized in the list view (all students) as well as in the individual view
(individual student). The de-anonymization of individual cases must be done con-
sciously by clicking a button and is only possible when the student takes part at the
project. This should limit bias effects with regard to the identity of the individual
student. This detailed view provides the advisor compact information about the stu-
dent (see Table 7.2).
Table 7.2 Detailed student information in LAPS

General • Course of studies
• Enrollment date and type of the university entrance qualification
• Start semester
• State of studies
Student • Birthday
• Age at enrollment
• Gender
• Name (after explicit de-anonymization)
• Banner ID
• E-mail address
• Type of the university entrance qualification
• Grade of the university entrance qualification
• Date of the university entrance qualification
Examination data • Number of examinations
• Successful examinations
• Failed examinations
• Excused cancellations
• Total ECTS
• Current average grade
• Average grade base studies
• Average grade main studies
• Risk score
76.0% (13.3%) Semester with Type of university

Successful study?
examinations semester
861 / 1133 / 6485
No
>=1 College >1
: Failure probability
6485: Number of students considered
Fig. 7.4 Identified risk and its representation in LAPS
This information is supplemented by a report and presentation of the actual sta-

tus of the examination results and the course of studies at various levels:
• The semester table lists the acquired ECTS, the ECTS total, the average grade
(weighted according to ECTS), and the number and status (passed, failed,
approved cancelled) of the examinations taken per semester.
• An overview of the registered examination performances of the previous semes-
ters is provided.
• The performance chart lists detailed information on all examination (e.g., ident
number, description, status, ECTS, grade). By clicking on the ident number of an
examination, a detailed view is presented, and the grades for all available semes-
ters can be seen. In this way, the student’s performance can be compared to the
overall cohort.
The individual view is completed by the risk details: a graphical representation
of the distribution of risks (with which frequency risk criteria of a certain probabil-
ity of failure apply to the student) as well as a representation of the risk criteria
applicable to the student. Figure 7.4 provides an example of an automatically identi-
fied risk.
3.6 Using LAPS for Quality Assurance
Besides the functionality to support students based on the LAPS profile, the soft-
ware supports quality assurance of study programs. The following functionalities
are designed to provide information about specific programs, lectures, and student
cohorts. The analysis results presented are based on the anonymous LAPS profiles,
which mean that personal data is not visible.
• Programs
In this view, study program information can be obtained. The following data is
available for each program: number of enrolled students, number of dropouts,
number of successful study progressions, average risk possibility, minimum/
average/maximum student age, gender distribution, average grade of the univer-
sity entrance qualification, and retreats from examinations.
• Cohorts
For the development of study programs, it is important to get information on the
consequences of the changes on module level, e.g., to the examination regula-
tions and the curriculum. The cohort’s view allows to compare the distribution of
students obtained ECTS credits per semester and to identify possible structural
problems when students do not achieve the required ECTS.
• Lectures
This view allows an in-detail analysis for each semester of lectures and provides
access to distribution of grades, number of successful examinations, average
grade, number of retreats, and number of registrations.
4 Discussion
Although the LAPS project is developed at HdM, it is open to be used at any other
university. This achieved by being released as an open-source software and the data
import is not bound to a specific CMS. The only requirement is that the CMS data
needs to be exported into a LAPS-readable CSV format. For doing so, it is required
to write export scripts that allow to export the data. Additionally, it could be possible
to adjust the definitions mentioned in the above as the study progression differs
from each university.
Nevertheless, some lacks and points of discussion were identified for the project,
which are described in the following:
• Using students’ gender as part of the risk calculation.
As a part of the risk analysis, the LAPS software uses students’ gender informa-
tion. It is not intended to make differences or judgements between the genders.
In fact, the risk analyzation results can identify potential problems of gender
groups.
• Validity of the used data model.
When the LAPS software is used for consulting students, advisors need to be
clear about the underlying data that are used to calculate the possibility of a posi-
tive/negative study progression. As the algorithm considers sociodemographic
and examination data, all derived risk probabilities are based on these facts.
During a consultation situation, it is required to know that the algorithm may
identify a student progression as risky which is due to a small number of ECTS
credits obtained during the first semesters. This could have multiple reasons, e.g.,
illness of the student. To cover this issue, the LAPS software is inextricably
bound to the LAPS consultation process which includes a mandatory consulta-
tion introduction workshop.
Illnesses that result in a long-lasting absence from university, such as illnesses
that are part of the general risk of life, such as influenza, etc., usually result in
vacation semesters and are considered accordingly in the LAPS risk assessment.
The underlying data for such events is recorded by the CMS. Other data, e.g.,
other health issues and possible labor of students, is not recorded due to the strict
privacy laws in Germany. These students’ information can be taken into consid-
eration during the consultation, which is part of the LAPS project.
• Low response rate for course leaders taking part in the LAPS consultation intro-
duction workshop.
As described in the above, course leaders need to take part in mandatory work-
shops that provide an introduction to consultation of students using LAPS data.
All course leaders of the bachelor programs were invited, but only a few of them
responded. The main problem was that the workshop was planned during semes-
ter holidays and many of the invited course leaders were not available for two
consecutive days. This could be improved by splitting up the workshop into
smaller lessons (e.g., 3 × 3 h) during semester.
• Student response rate could have been better for the first run.
For the first run of LAPS, all students were invited via e-mail. For the targeted
student group (all students in the fourth semester or below, N = 1500), 98 stu-
dents (6.53%) participated. It is planned to increase the number of participating
students by integrating the opt-in registration into the enrollment process.
Nevertheless, only a single student of the students who filled the opt-in form did
not want to participate.
• Data analysis is CPU intensive and requires time.
As the analysis is complex data mining process, the calculation should be inte-
grated into a batch job executed when the application is not used interactively,
e.g., during nighttime.
5 Future Work
For the further development of the LAPS project, it is planned to analyze data on
how students accept the system. It has to be tracked how many of the students who
choose the opt-in are receiving a system-generated e-mail. Last but not least, it has
to be tracked how many of them are taking advantage of the conversation offer with
either staff members of the student support center or their according course leaders.
A consultation guideline is currently developed to give support on how to use the
system in such situations.
Technically, an automated ETL process (extraction, transformation, load), which
is a standard process model for data warehouse and big data computing, could
improve the upload of new student and examination data. By adding a functionality
to track student progression within a lecture (e.g., by integrating the results of
interim tests), students would be able to get information about their progression in
a specific lecture. In addition to automatically identified risks, a manual student tag-
ging functionality could extend the LAPS software: student progressions are anony-
mously presented to course leaders, and based on their experience, they could
decide if the student needs additional support.
Acknowledgments The LAPS team likes to thank the developers of S-Beat,2 namely, Dominik
Herbst, Niclas Steigelmann, and Annkristin Stratmann, which served as the technical foundation
for the feasibility study. Additionally, we like to thank the authors of the “Softwaregestützte
Studienverlaufsanalyse zur frühzeitigen gezielten Studienberatung” article, namely, Prof. Dr.
Johannes Maucher and Prof. Dr. Tobias Seidl, which served as a foundation for this book chapter.
Finally, we like to thank the “Digital Innovations for Smart Teaching—Better Learning” promo-
tional program for allowing us to further improve the project.
References
1st International Conference on Learning Analytics and Knowledge 2011. (2010, July 22).
Retrieved February 26, 2018, from https://tekri.athabascau.ca/analytics/
Agrawal, R., Imieliński, T., & Swami, A. (1993). Mining association rules between sets of items in
large databases. In ACM sigmod record (Vol. 22, pp. 207–216). New York: ACM.
Christensen, B., & Meier, J.-H. (2014). Zur Frühidentifikation von Studienabbrüchen. Das
Hochschulwesen, 6, 182–185.
Journal of Technology Enhanced Learning, 4(5/6), 304. https://doi.org/10.1504/
IJTEL.2012.051816
General Data Protection Regulation (2018). (EU) 2016/679 §. Retrieved from http://eur-lex.
europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679&from=DE
Hermann, C., & Ottmann, T. (2006). StuVa–Werkzeugunterstützte Studienverlaufsanalyse zur
Unterstützung der Studienberatung. In HDI (pp. 127–136).
Jaeger, M., & Sanders, S. (2009). Kreditpunkte als Kennzahl für die Hochschulfinanzierung. In
Hannover: HIS–Forum Hochschule.
Siemens, G. (2010, August 25). What are learning analytics? Retrieved February 25, 2018, from
http://www.elearnspace.org/blog/2010/08/25/what-are-learning-analytics/
Trapmann, S., Hell, B., Weigand, S., & Schuler, H. (2007). Die Validität von Schulnoten zur
Vorhersage des Studienerfolgs - eine Metaanalyse. Zeitschrift für Pädagogische Psychologie,
21(1), 11–27. https://doi.org/10.1024/1010-0652.21.1.11
http://s-beat.de.
2
Chapter 8
Monitoring the Use of Learning Strategies
in a Web-Based Pre-course in Mathematics
A Comparison of Quantitative and Qualitative Data
Sources
Katja Derr, Reinhold Hübl, and Mohammed Zaki Ahmed
1 Introduction
One of the challenges tertiary institutions have to face is the growing diversity in
first year students’ educational backgrounds and knowledge levels. Not all under-
graduates seem to be adequately prepared for the demands of their course; in techni-
cal degree programs many students have knowledge gaps in secondary school
mathematics and some even struggle to apply basic (Armstrong & Croft, 1999) or
“extremely basic” (Ballard & Johnson, 2004) rules. Such deficits are a considerable
threat to academic achievement in STEM subjects (science, technology, engineer-
ing, and mathematics) (Croft, Harrison, & Robinson, 2009; Knospe, 2011).
Technical faculties have met this problem by providing preparatory and bridging
courses in mathematics, offered face-to-face (Abel & Weber, 2014), online (Krumke,
Roegner, Schüler, Seiler, & Stens, 2012) or in blended versions (Biehler, Fischer, &
Wassong, 2012). High participation rates suggest that these are welcomed by stu-
dents (Bargel, 2015). Web-based learning environments have been found particu-
larly useful when addressing heterogeneous groups of learners and students who not
(yet) live near the campus. They also allow to collect learner data at a very early
point in time, in the “liminal phase” between secondary and tertiary education
(Clark & Lovric, 2009).
Such data are considered relevant from different perspectives. First, mathematics
test results can be used to predict tertiary achievement in engineering and, based
on these observations, develop “early warning systems” for “at-risk” students
(Greller & Drachsler, 2012). Second, analyzing learning behavior during the pre-course
K. Derr (*) · R. Hübl

DHBW, Mannheim, Germany
M. Z. Ahmed
Plymouth University, Plymouth, UK

https://doi.org/10.1007/978-3-319-64792-0_8
120 K. Derr et al.
may help identifying effective and less effective uses of learning strategies. Such
observations could result in suggestions for individual learners and thus support
their transition to tertiary education. Third, analyses of pre-course outcomes inform
practitioners of “what works” and thus contribute to the growing body of literature
on “transition pedagogy” (Kift, Nelson, & Clarke, 2010).
The evaluation of preparatory courses, however, can be conceptually and meth-
odologically challenging. Being extracurricular activities, pre-courses are not man-
datory and students are free to participate or withdraw at any time. Such threats to
internal consistency may be increased in web-based environments which, compared
to traditional face-to-face courses, are characterized by poorer learner commitment
(Ashby, Sadera, & Mcnary, 2011; Smith & Ferguson, 2005; Street, 2010) and lower
answer rates (Cook, Heath, & Thompson, 2000; Fan & Yan, 2010; Tourangeau,
Conrad, & Couper, 2013). Finally, organizational and technical barriers may pro-
hibit relating pre-course learner data to subsequent student performance.
This study measured learner behavior in a web-based pre-course in mathematics
and related these outcomes to achievement in five engineering courses at Baden-
Wuerttemberg Cooperative State University Mannheim (subsequently abbreviated
DHBW for Duale Hochschule Baden-Württemberg). Funded by the joint research
project optes (www.optes.de), the team at DHBW Mannheim successively devel-
oped, revised, and re-evaluated the course program consisting of diagnostic self-
tests, interactive learning modules, and additional support structures.
Using the theory of self-regulated learning as a theoretical framework, the inter-
play between students’ preconditions when entering the course, their learning
behavior, and the learning environment was accounted for in quantitative and quali-
tative analyses. By exploring which variables positively influenced pre-course
learning gains or academic achievement, this study aimed at
• Identification of variables that distinguish between successful and less successful
pre-course participation of “at-risk” students.
• Clarifying if and how data collected from web-based pre-courses can contribute
to the emerging field of learning analytics (Greller & Drachsler, 2012; Scholes,
2016).
• Making suggestions for the support of “at-risk” students in the transition phase
between secondary and tertiary education.
2 Literature Review
It is generally agreed upon that secondary and tertiary achievement are strongly cor-
related with each other (Hattie, 2009) and that this relation is of particular relevance
in engineering (Ackerman, Kanfer, & Beier, 2013). Thus cognitive predictors like
secondary school GPA (Hell, Linsner, & Kurz, 2008; Söderlind & Geschwind,
2017), school grades in mathematics (Faulkner, Hannigan, & Gill, 2010; Liston &
O'Donoghue, 2009), as well as placement tests in mathematics (Carr, Bowe, & Ní
8 Monitoring the Use of Learning Strategies in a Web-Based Pre-course in Mathematics 121
Fhloinn, 2013; Ehrenberg, 2010; Zhang, Anderson, Ohland, & Thorndyke, 2004)
have been found significantly related to measures of academic achievement, like
tertiary grade point average (GPA) or retention in STEM subjects.
To isolate the impact of remedial courses from the effect of these cognitive pre-
dictors and to quantify their effects has been found difficult. For the UK, Lagerlöf
and Seltzer (2009) as well as Di Pietro (2012) found only weak or no effects of
participation in a remedial mathematics course on “at-risk” students’ achievement in
economics. Similar observations were made at US-American universities by Ballard
and Johnson (2004), Moss and Yeaton (2006), and Bettinger and Long (2009).
Greefrath, Koepf, and Neugebauer (2016) found that participation in A-level math-
ematics classes and results in a placement test were the strongest predictors of first
year mathematics performance in computer science and electrical engineering at
two German universities. Participation in a blended pre-course positively affected
placement test scores, but not necessarily first year exam grades. The authors
suggested that the influence of the pre-course was not strong enough to overpower
the dominant role of prior knowledge. Similar observations were made in studies on
face-to-face courses by Polaczek and Henn (2008) as well as Abel and Weber (2014).
These studies, however, did not evaluate students’ learning activities during the
course. Closing knowledge gaps in a relatively short period of time demands a lot of
effort and is also likely to be influenced by the course’s design. Vuik, Daalderop,
Daudt, and van Kints (2012), for example, performed a quantitative evaluation of a
web-based course for aerospace engineering and computer science students. In their
study pre-course participants outperformed nonparticipants in their first mathematics
exam, particularly when they had been classified “active participants.”
When interpreting such results it needs to be considered that the ability to benefit
from preparatory courses is dependent on prior domain knowledge, as well. High-
performing students are more likely to make effective use of learning strategies, to
plan and structure the learning process, and to self-evaluate the outcomes of this
process (Pintrich, Smith, Garcia, & McKeachie, 1991; Weinstein, Zimmermann,
& Palmer, 1988). The concept of self-regulated learning provides a theoretical
framework that accounts for the complexity of the learning process and interactions
between learner characteristics (e.g., prior knowledge, age, or gender), environmen-
tal factors (e.g., design of the course), and the mediating effects of learner behavior
(e.g., use of learning strategies) (Azevedo, 2005).
Evaluations of students’ use of metacognitive strategies have shown, for example,
that time management and organizational strategies are good predictors of academic
achievement (Barnard, Lan, To, Osland Paton, & Lai, 2009; Barnard-Brak, Lan, &
Paton, 2010; Broadbent & Poon, 2015; Carson, 2011; Credé & Phillips, 2011;
Entwistle & McCune, 2004). Inexperienced students and students with poor domain
knowledge seem less able to structure and plan the learning process and are more
likely to procrastinate (Michinov, Brunot, Le Bohec, Juhel, & Delaval, 2011; Plant,
Ericsson, Hill, & Asberg, 2005). The learning environment may positively affect
this group’s learning behavior by providing external guidance and structure (Artino
& Stephens, 2009; Azevedo & Cromley, 2004).
122 K. Derr et al.
Task strategies like rehearsal or self-monitoring have been found less consistent
predictors of achievement; while Morris, Finnegan, and Wu (2005), Samson (2015)
and Tempelaar, Rienties, and Giesbers (2015) found that taking self-tests positively
affected learning outcomes, two meta-studies reported contradicting results
(Broadbent & Poon, 2015) or no effects (Credé & Phillips, 2011).
The effort students put into their learning may also be dependent on motivational
aspects like task interest or task value: attitude towards the subject has repeatedly
been found to correlate with performance (Richardson, Abraham, & Bond, 2012;
Robbins et al., 2004). As mathematics is not the prior study interest of engineering
students, negative attitudes could be an obstacle for successful pre-course participation
(Meyer & Eley, 1999).
The motivation to learn may also be influenced by social interaction with peers
and lecturers. Help seeking refers to a learner’s ability to activate social resources
(Karabenick, 2004; Newman, 2002). As suggested by Zimmerman and Moylan
(2009), it indicates a high level of self-regulation if learners seek out help from others
to improve their learning. Not all students, however, are able to benefit from help-
seeking or from peer-learning activities, making it difficult to quantify the effects of
social environment on achievement (Barnard et al., 2009; Broadbent, 2017).
Finally, students’ ability to self-reflect and evaluate the learning process is an
essential characteristic of successful learning processes (Zimmerman, Moylan,
Hudesman, White, & Flugman, 2011). Learning environments may induce or
suppress self-reflection; an extremely high workload, for example, is likely to
evoke surface approaches to learning and a stronger focus on grades and scores
(Dweck, 1986).
3 Method
A multi-method case study was conducted, using quiz and survey results, log files,
interviews, and administrative data. Based on Yin’s case study framework, the
research design was a holistic single case, one university’s implementation of a
web-based pre-course in mathematics for engineering students (Yin, 2009). The
first part of the study used mainly quantitative methods to gain data from whole
student cohorts. In-depths insights were captured through a set of guided interviews
at the end of the study.
3.1 Pre-course Design
Prospective students were able to access the web-based pre-course in June; the first
semester started in October. Students could find the course on the university’s
homepage but were also informed via mailing lists, encouraging them to register and
take the diagnostic pre-test. This two hour self-test covered ten mathematical fields,
from Arithmetic to Vectors, each addressed by four to six items (see curriculum as
suggested by SEFI mathematics working group, 2013, as well as cosh, 2014). After
submitting the test, participants received a diagnostic feedback, suggesting learning
contents if test scores per mathematical field fell below a predefined threshold.
All learning modules were open for self-study, combining texts, graphs, animations
and videos, examples, and exercises. At the end of each module, students could
take a subject-related final test, consisting of 10–15 randomized items. Students
who wanted additional support in their learning could enroll in either a weeklong
face-to-face course or a one-month e-tutoring course.
The complete interactive learning material, animations, tests, and surveys were
developed by the team at DHBW Mannheim. The technical environment used for
this project was the open-source learning management system (LMS) Moodle 3.1.
Some considerable changes to the LMS’s design were made in order to improve
usability. The feedback based on students’ results in the diagnostic pre-test was
significantly improved by a plug-in developed by Dreier (2014) for his student
research project in computer science.
At the beginning of the semester, all first year engineering students participated
in another diagnostic test, or post-test, taken at the university’s computer labs.
The post-test covered the same ten mathematical fields and was of similar difficulty
(for a more detailed description of course and tool development process, see Derr,
Hübl, & Ahmed, 2015). The difference between post-test and pre-test result, the
gain score, then indicated the learning outcome per student. Fig. 8.1 shows an
overview of the different pre-course elements and data sets.
Fig. 8.1 Overview of the design of the pre-course and the data collected for course evaluation
124 K. Derr et al.
3.2 Data Collection
In this report, data and results obtained from the years 2014 to 2016 are summarized,
but some earlier evaluations will be referred to, as well. Relations between students’
prior knowledge, first year performance, and graduation, for example, were analyzed
using anonymized administrative data from three complete cohorts who graduated
between 2014 and 2016. Participants were students from five degree programs
(computer science, electrical engineering, industrial engineering, mechatronics,
mechanical engineering). Each year between 70 and 80 per cent of all first year
students registered on the web-based platform and participated in the diagnostic
pre-test (see Table 8.1). These students were ascribed to the group of pre-course
participants (regardless of their learning activities on the platform).
Nearly all first year students participated in the post-test. The first year examination
Mathematics I was taken 6 months later.
The student perspective was based on interviews with nine purposefully selected
first year students who had participated in the pre-course and had been considered
to be “at risk” based on their pre-test results.
3.3 Data Analysis
3.3.1 Quantitative Data
Test results and questionnaire data were inputted into SPSS V 23. Descriptive analyses
and single linear regressions were used to analyze and control for interactions between
predictive variables. A p-value of less than 0.05 was considered statistically significant;
p-values of less than 0.01 or 0.001 were reported if applicable.
Table 8.1 Summary of collected data

Interest Dataset 2014 2015 2016
Pre-course participants
Prior knowledge in mathematics Diagnostic pre-test 603 551 596
Demographic and attitude towards Survey 593 535 582
mathematics
Use of learning strategies Survey 200 117 122
Learning activities Survey and log files 603 551 596
Pre-course learning gains Post-test minus pre-test 603 551 596
Pre-course evaluation Survey 205 117 122
Nonparticipants
Prior knowledge in mathematics Post-test 105 156 171
First year students 708 707 767
First year mathematics achievement Exam grades (Mathematics I) 674 660 747
Student experience Interview 9
Final year studentsa
Study success Grade point average 589 650 554
Cohort of 2011 graduated in 2014; cohort of 2012 graduated in 2015; cohort of 2013 graduated in 2016
a
3.3.2 Qualitative Data
The interest of the final interview study was to learn about “at-risk” students’ expe-
riences in the pre-course and during their first months at university and relate these
observations to findings made in the quantitative evaluations. It was expected that
the qualitative results would clarify and enrich those outcomes. At the same time,
the interviews were to show if further themes that had not yet been addressed
would emerge.
The single interviews, ranging from 25 to 35 min, were conducted using a
semi-structured interview technique that allowed responding to the situation at
hand. A list of open-ended questions was used to guide the interview but varied in
order, wording or focus (Robson, 2011). All interviews were digitally recorded,
transcribed verbatim by one of the authors, checked for accuracy, and loaded into
MAXQDA V12. Each transcript was coded by examining the raw data and identify-
ing statements referring to the study interest. The analysis was performed at two
levels, within each case and across cases (Stake, 1994).
3.4 Ethical Considerations
The university’s data privacy official gave ethical approval. Pre-course participants
were informed of the purpose of the study and agreed that their data were collected,
anonymized, and evaluated. Students aged under 18 provided parental consent. Tests
and questionnaires were completed voluntarily and anonymously. The interviews
with first year students were prepared by giving short information about background
and goal of the study in selected first year mathematics lectures. An e-mail invitation
with an attached information sheet was then sent to all potential participants. Students
willing to participate were asked to respond to the researcher by email. Students
attended the face-to-face interviews voluntarily and were informed that all data were
treated confidentially. The pseudonyms used were Anne, Ben, Chris, Daniel, Eric,
Frederic, Julia, Marc, and Nora. All data were kept securely and anonymized.
3.5 Limitations
Pre-course participation in this project was free for all entering engineering stu-
dents, causing a bias that needs to be accounted for in all interpretations.
Regarding the evaluation of tracking data, it needs to be considered that only
learning activities in the university’s LMS could be monitored, but students may
also use external links, social networks, learning tools, or apps (Pardo & Kloos,
2011; Tempelaar et al., 2015).
Missing information may also have weakened the representativeness of the scales
used in this study. While response rates in the e-tutored courses were acceptable
to good (between 64% in 2014 and 38% in 2016), they were relatively poor in the
126 K. Derr et al.
self-study group (27% in 2014 and only 16% in 2016). Distributions in the group of
respondents (prior knowledge, first year performance) were not significantly different
from the general student body, but it can be assumed that students who participated
in the evaluation survey had different mindsets and feelings towards the pre-course
than those who did not (Nulty, 2008; Tourangeau et al., 2013) (Table 8.1).
4 Results
4.1 P
rior Knowledge in Mathematics and Study Success
in Engineering
Using data collected from three previous cohorts (entering 2011–2013), the first year
examination Mathematics I was identified as a significant predictor of study success
at the end of the engineering degree program. In a linear regression, this exam alone
explained up to 43% of the variance in cumulated grade point average (GPA) at the
end of the course and thus was considered a good early indicator of study success.
Based on these observations, students’ prior knowledge in mathematics was
found the strongest determinant of Mathematics I. In a multiple regression, results
in the diagnostic pre-test outperformed all other person-related variables (including
gender, age, gap between school and university, German federal state, type of
secondary school, mathematics grades at secondary school, and secondary GPA).
In 2014, for example, a student with a pre-test mean score of 40 was predicted
Mathematics I grades 0.6 higher than a similar student with a pre-test mean score of
20 (see Table 8.3).
Secondary school grade point average (GPA) was the second best predictor of
first year mathematics performance (plus 0.4 grades in Mathematics I for each
increase of 1.0 in GPA in 2014). By comparison, the influence of demographic and
other school-related variables was weaker and much more inconsistent (for a more
detailed quantitative report, see Derr, Hübl, & Ahmed, 2018).
4.2 E
ffects of Pre-course Participation on First Year
Performance
After having established the relevance of prior knowledge in mathematics for study
success in engineering, it was investigated if participation in the pre-course would
show a moderating effect on this relation. Each year between 70 and 80 per cent of
first year students participated in both tests. The average pre-test score (in %) in this
group varied between 49.1 (2015) and 51.1 (2016); the average post-test score
ranged from 53.9 (2015) to 56.6 (2015). By comparison, students who had not par-
ticipated in the pre-test achieved a post-test mean score between 42.7 (2015) and
47.3 (2014). Between-group difference was significant (p < 0.01), but in both groups
a large variance in test results could be observed (see Fig. 8.2; Table 8.2).
Fig. 8.2 Pre-post-test scores (2014–2016) pre-course participants (=participation both tests) ver-
sus nonparticipants (participation post-test only)
Table 8.2 Pre-post-test scores (2014–2016) pre-course participants (y) and nonparticipants (n)
Both tests (y) Post-test only (n)
2014 2015 2016
Pre-test Post-test Pre-test Post-test Pre-test Post-test 2014 2015 2016
n 603 603 551 551 596 596 105 156 171
Mean 49.7 55.2 49.1 53.9 51.1 56.6 47.3 42.7 43.7
Median 49.4 53.3 49.4 53.3 50.6 55.6 46.7 40 42.2
Variance 255.4 304.9 215.4 266.5 262 304.4 330.4 288.8 288
The average gain score (post-test minus pre-test) across cohorts was 5.3
(median = 5.7), with a maximum value of 61.8 and a minimum of −40.1. Students
with poor pre-test results (mean score <50), thus considered the “at-risk” group, had
an average gain score of 8.1 (median = 7.3; max. = 61.8; min. = −25.3).
Added to the multiple regression predicting first year mathematics performance,
the gain score significantly contributed to the model. Compared to the dominant
role of prior knowledge, this effect was not very strong; a noticeable change in
Mathematics I was only predicted for students with very high learning gains. For
example, in 2014 a student with a gain score of 20 was predicted an increase in
Mathematics I grades by 0.28 (B = 0.014), compared to a similar student with a gain
score of zero.
The quantitative analyses also indicated that pre-course participants on average
were able to improve their starting position at university. Students who had not par-
ticipated in the pre-test or the pre-course program showed significantly poorer first
128 K. Derr et al.
Table 8.3 Regression analysis Mathematics I (2014–2016)

2014 2015 2016
SE SE SE
B B β B B β B B β
1. Gender a
0.17 0.12 0.06 0.03 0.12 0.01 0.16 0.11 0.06
2. Age (years) 0.05 0.04 0.09 −0.04 0.05 −0.07 −0.04 0.04 −0.07
3. Gap school/university −0.07 0.06 −0.08 0.12 0.06 0.16 0.17 0.05 0.25
(years)
4. Federal Stateb
Rhineland-Palatinate 0.05 0.11 0.02 0.10 0.12 0.05 0.04 0.11 0.02
Hesse −0.05 0.11 −0.02 −0.05 0.12 −0.02 −0.13 0.12 −0.05
NRW −0.07 0.14 −0.02 0.10 0.15 0.03 −0.23 0.14 −0.08
Bavaria 0.37 0.15 0.11* 0.16 0.14 0.06 −0.14 0.16 −0.04
5. Type of school: 0.13 0.16 0.06 0.45 0.16 0.21** 0.40 0.16 0.18*
Gymnasiumc
6. Secondary school −0.06 0.14 −0.02 0.39 0.16 0.18* 0.09 0.16 0.04
mathematics grades
7. Secondary school GPA 0.38 0.10 0.20** 0.36 0.11 0.20** 0.62 0.10 0.36**
8. Pre-test score (%) 0.03 0.00 0.49** 0.02 0.00 0.24** 0.02 0.00 0.26**
9. Gain score (%) 0.01 0.00 0.18** 0.01 0.00 0.13* 0.01 0.00 0.03*
R2/R2 adj. 0.35 / 0.33 0.27 / 0.25 0.31 / 0.29
F for change in R2 17.47** 10.21** 13.64**
Mathematics I exam grades measured on a scale from 1 to 5
B unstandardized beta coefficient, SE B standard error, β standardized beta coefficient; *p < 0.01;
**p < 0.001
a
Baseline, male
b
Baseline, Baden-Wuerttemberg
c
Baseline, vocational school
Table 8.4 Mathematics I grades (2014–2016) pre-course participants (y) and nonparticipants (n)
2014 2015 2016
y n y n y n
n 578 96 519 141 583 164
Mean 2.8 3.3 2.7 3.3 2.7 3.2
Median 2.8 3.3 2.7 3.4 2.7 3.2
Variance 0.97 1.24 0.90 0.96 1.01 1.08
ANOVA: 2014: F(1, 672) = 28.3, p < 0.001; 2015: F(1, 658) = 39.7, p < 0.001; 2016: F(1,
745) = 29.1, p < 0.001
year mathematics performance and more failures. The difference between partici-
pants and nonparticipants accounted for distances between 0.5 and 0.6 grades in
Mathematics I (on a scale from 1 to 5), and ANOVA suggested significant between-
group differences for all three cohorts (see Table 8.4).
The student perspective, as well, suggested a favorable interpretation of pre-

course participation; all interviewees claimed that they had benefitted from the
course. While these outcomes are only a spotlight, the many positive comments
indicate that additional mathematics support was strongly needed and welcomed by
students entering tertiary education.
“And if I hadn’t taken part in this pre-course I would have thought: ‘Why was he able to just
leave out the brackets there?’ Because that part isn’t explained any more … And that’s why
[…] to follow the lecture it really does help.” [Frederic].
“Yes, it was pretty helpful, that you could find out, yeah, okay, that is where you’re lacking
a bit of knowledge, because otherwise I would have walked right into the first lecture and
would have been struck dead. And with this course it wasn’t so bad.” [Ben].
It also emerged that students were quite aware of the relevance of mathematics for
their course and that their first year experience had increased this awareness.
Julia: “Many pre-course contents were useful later and I thought ‘oh, thank goodness I repeated
that’.”Interviewer: “Can you name an example?”Julia: “There was this lecture in maths where
I noticed that … Wait, it was prime … some ization.”Interviewer: “Prime factorization?” Julia:
“Exactly. Because then I thought, goodness, where could you possibly need that? And then it
was needed in this proof and I was quite happy that I had done that.” [Julia].
“The basics in maths, those aren’t highly complicated calculations. You have to be able to
solve them quickly and not ponder for three hours.” [Nora].
4.3 Drivers of Successful Pre-course Participation
In order to identify variables that helped distinguish between successful and less
successful pre-course participation, analyses of variance on gain score were per-
formed for different sets of independent variables. All investigations were carried
out with a special focus on the group of “at-risk” students, thus controlled for prior
domain knowledge (=results in the diagnostic pre-test). Variables that showed a
significant influence on the gain score were also added to the multiple regressions
predicting first year achievement in mathematics.
4.3.1 Attitude Towards Mathematics
Two subscales from the Trends in International Mathematics and Science Study
(TIMSS) were used to investigate the relevance of students’ attitudes towards the
subject for pre-course learning outcomes (Kadijevich, 2006, p. 41f; Mullis, Martin,
Foy, & Arora, 2012, p. 333f). It had been hypothesized that high scores on items
like “I am interested in mathematics” (from the subscale “Liking mathematics”)
or “I learn things quickly in mathematics” (from the subscale “Self-confidence in
learning mathematics”) would positively affect pre-course learning gains.
The scales correlated with each other, thus replicating previously reported relations
between mathematics liking and self-confidence (Parsons, Croft, & Harrison, 2009).
130 K. Derr et al.
It should be noted, however, that the results were skewed and that only a minority of
students expressed outright negative feelings towards the subject. Very positive
attitudes were mainly observed for students with very good results in the diagnostic
pre-test. Both attitude scales were unrelated to pre-course learning gains.
In the interviews, all participants stated that they had liked mathematics at school
and that they had been good at it. This was remarkable as, based on their pre-test
results, all interviewees were in the “at-risk” group.
4.3.2 Time Management and Organizational Strategies
Seven items from the subsets “Cognitive and metacognitive Strategies” and “Resource
management strategies” of the LIST inventory were related to pre-course learning
gains (Schiefele & Wild, 1994). LIST is a German adaptation of the Motivated
Strategies for Learning Questionnaire MSLQ (Pintrich et al., 1991). The answer pat-
terns in this analysis were quite irregular; students who “strongly agreed” to items
like “I always followed a certain learning schedule” also had high or very high pre-
test scores, whereas the rest of the data showed non-linear distributions. The time
management and organizational scales thus only allowed to distinguish between stu-
dents with a very proficient use of learning strategies and the rest of the sample. All
seven items were unrelated to the gain score and thus failed to differentiate between
more and less successful pre-course participants.
In the interviews students found it difficult to describe how they had planned and
structured the learning process. It became apparent that those who had participated
in the e-tutored course [Anna, Ben, Marc, Nora] had acted upon the schedule pro-
vided by this course, while those who studied alone did not follow a certain plan.
One exception was Frederic, who had worked through the complete pre-course
alone and had managed to complete one learning module per week. In the interview
he admitted that he might have benefitted from a course on “learning to learn” and
on the issue of time management but was also sceptical if he would find the time for
“yet another course”:
“And something like that could be useful, maybe a small lecture for time management. But
I don’t know if anyone would stick to it, if anybody would really do it […]. It’s probably
difficult to carry out, you just think: ‘Yeah, sure’ but then … you forget about it.” [Frederic].
4.3.3 Time on Task
Different sources to measure quantitative aspects of learning were available, like

students’ answers to the evaluation questionnaire (number of learning modules,
weeks, hours per week) and the LMS’s log files. Self-reported study time per week,
for example, was moderately correlated with outcomes; students who spent more
hours learning on average had poorer pre-test results and also higher learning gains.
Similarly, the number of learning modules a student had accessed was positively
related to the gain score. However, ANOVA or single regression with these variables
did not account for significant differences. The interviewees’ accounts of their time
on task varied strongly, from “about a day, taken all together” [Chris] to about 10 h
per week over a period of 10 weeks [Frederic].
4.3.4 Task Strategies
Task strategies like reading or rehearsal were measured by tracking the number of
learning module pages students had accessed and by the number of (randomized)
self-tests submitted at the end of each learning module.
A high number of learning module page views could be ascribed to a higher gain
score, but this relation was very weak and not significant. By comparison, the num-
ber of test attempts could be related to a significant increase in gain score.
Transformed to a four-step ordinal variable, with “no test attempts,” “1–4 attempts,”
“5–8 attempts,” and “9 and more attempts,” this variable significantly differentiated
between higher and lower achievement in the pre-course (p < 0.05).
Summarizing pre-course participation from 2014 to 2016, students with no test
attempts on average had the poorest learning gains (gain score = 3.8), and students
with 9 and more attempts had an average gain score of 10.4 (Table 8.5). The effect
of this variable was even stronger when the sample was reduced to the “at-risk”
group (see Table 8.6).
4.3.5 Additional Face-to-Face or e-Tutoring Support
Two additional support programs were provided: a weeklong face-to-face course

and a one-month e-tutoring program. On average, 15% of pre-course participants
enrolled in the face-to-face course, 15% in the e-tutoring course, and 70% studied
alone.
Students who participated in an additional program had below-average pre-test
results. The gain score was significantly affected by the type of course a student had
chosen to attend. In 2014, for example, face-to-face course participants had an aver-
age gain score of 3.5 (n = 91), whereas students who completed the e-tutoring
course had an average gain score of 6.7 (n = 85). The highest learning gains were
Table 8.5 Pre-course gain score of all pre-course participants (2014–2016) by number of test
attempts
Number of test attempts
Total (pre-course participants) None 1–4 5–8 9 and more
n 1750 973 490 132 155
Mean 5.3 3.8 6.0 7.5 10.4
Median 5.6 3.8 6.4 8.1 9.2
Variance 162.3 154.1 142.1 221.0 185.0
ANOVA: F(3, 1746) = 2.8, p < 0.05
132 K. Derr et al.
Table 8.6 Pre-course gain score of “at-risk” group (2014–2016) by number of test attempts
Number of test attempts
Total (pre-course participants) None 1–4 5–8 9 and more
n 906 490 274 64 78
Mean 8.0 6.4 8.2 13.1 13.8
Median 7.3 6.3 7.5 11.3 11.0
Variance 160.1 155.7 131.1 174.2 214.8
ANOVA: F(3, 902) = 11.9, p < 0.001
achieved by students who had participated in both course types, e-tutoring and
face-to-face, with an average gain score of 9.1 (n = 28). While learning gains of
students in the e-tutoring course were highest, the differences between the different
groups were not significant (ANOVA: df1 = 3; df2 = 599; F = 1.578; p = 0.194).
Descriptive analyses suggested that in the face-to-face course students had even
poorer pre-test results and more often had attended vocational schools. Although
these differences were not significant there was a tendency that the e-tutoring
course was more often chosen by higher-performing students. In any case, e-tutoring
participants showed much more online learning activities and submitted significantly
more test attempts.
It is suggested that the more structured design of the e-tutoring course posi-
tively affected students’ activity level and was much more efficient than the shorter
and less binding face-to-face course that had not demanded the submission of
course work.
“…because, you only got a certificate after submitting all exercise sheets. And I thought
that was quite all right because you were somehow forced to do some problems. Because,
in hindsight I guess they do help, even if you’re not always in the mood to do them.” [Ben].
“I did two courses, one e-tutoring course and one in-class. And I was surprised, because I
liked the online course better and I gained much more from it. …In retrospect I would say
that the online course helped me more than the face-to-face course. Despite that, I’m happy
that I did both.” [Anna].
4.3.6 Social Interaction, Help Seeking, and Peer Learning
It had been hypothesized that the number of online interactions as an indicator of

help seeking would be positively correlated with pre-course learning gains
(Macfadyen & Dawson, 2010). The number of forum posts in the e-tutoring course,
however, was unrelated to learning gains. The e-tutoring groups were highly hetero-
geneous regarding communication preferences, and the case numbers were too
small for statistical interpretation. Analysis of single cases, as well, did not suggest
that a high (or low) number of forum posts was related to achievement.
In the interviews it emerged that students had found it more helpful to learn alone
during study preparation. However, social resources and help seeking emerged as
highly relevant for the first weeks and months at university. Studying in groups was
described as important to understand mathematics problems at tertiary level.
“Studying in groups is what helps me the most, solving all kinds of problems, and talking
it through with somebody, discuss it.” [Anna].
“Well, mostly I prefer studying on my own. But especially in maths I find it makes sense
to study in groups. There will always be one person knowing something the others don’t.
And then the next person gets an idea the others would NEVER have. Yes, I really do think
it helps.” [Julia].
It also became apparent that not all students had positive attitudes towards learning
in groups. Frederic, for example, only chose to participate in a study group after he
had failed his first Mathematics I attempt. Ben and Marc found it difficult to benefit
from group learning.
“Well, in general [I prefer studying] alone, because I can concentrate better, because
especially when I study in a group, I have often experienced that you easily let yourself be
distracted, stray away and then in the end you have been sitting there for four hours and
have hardly learnt anything.” [Ben].
4.3.7 Self-Evaluation and Self-Reflection
The significant impact of the variable “test attempts” suggested that taking self-tests at
different points in time positively affected achievement. The evaluation also revealed
that opportunities to practice were highly welcomed by learners. The pre-post-test
design in particular helped students to relate their prior knowledge to their individual
learning gains. In the interviews it was investigated how interviewees interpreted their
test results. Feedback in the form of grades or scores may evoke competitive behavior
(Black, Harrison, Lee, Marshall, & Wiliam, 2003) and thus distract students from
reflection. The qualitative analyses indeed revealed that some students considered
their pre-test result a “negative surprise” [Marc], resulting in feelings of insecurity and
a strong motivation to achieve better post-test scores [Frederic].
“And in the pre-test I think I wasn’t THAT good, I only had 40% or so. And then I just, until
it really got started, up to then I just did all of the exercises. There was always a test after
every exercise and then I just went through everything, and always solved all of the
problems. And then there was this post-test again, here at the university and then I even
managed to get 75%.” [Frederic].
5 Summary and Discussion
5.1 I dentification of Variables that Distinguish

Between Successful and Less Successful Pre-course
Participation of “At-Risk” Students
In order to evaluate the effects of pre-course participation on engineering study suc-

cess, some underlying presumptions had to be confirmed. First, the relevance of
prior knowledge in mathematics for academic achievement in engineering was
established by relating preconditions (educational background, demographic, prior
134 K. Derr et al.
domain knowledge) to tertiary performance. In these analyses, results in a diagnostic

pre-test in mathematics emerged as the most dominant predictor of the first year
exam Mathematics I. In previous analyses of complete cohorts, Mathematics I had
been identified as the best predictor of final year GPA in all five courses.
This study thus contributed to the existing body of literature that placement tests
are good predictors of academic achievement in engineering (Abel & Weber, 2014;
Carr et al., 2013; Ehrenberg, 2010; Faulkner et al., 2010; Greefrath et al., 2016; Zhang
et al., 2004) and that below-average pre-test scores can be considered a risk factor.
To some extent this risk could be reduced by pre-course participation. Pre-course
participants showed better Mathematics I results than nonparticipants, and the gain
score (difference between post-test and pre-test) significantly contributed to a mul-
tiple regression predicting this exam. At the same time, there was a large variance in
the data, suggesting that not all students were able to benefit from the course.
From the set of potentially influential factors only one variable emerged that sig-
nificantly differentiated between successful and less successful pre-course participa-
tion of the “at-risk” group: Students who repeatedly engaged in self-assessments
showed significantly higher learning gains than those who did not. Added to the
multiple regression, this variable could also be related to a small but significant
increase in Mathematics I performance.
Affective and metacognitive variables showed only weak or no correlation with
learning gains of “at-risk” students. Students’ attitude towards the subject, for
example, seemed to be rather a covariate of prior domain knowledge (Richardson
et al., 2012; Robbins et al., 2004) than a factor influencing learning gains. Two
scales addressing the use of time management and organizational strategies were
also unrelated to learning gains, an outcome that was inconsistent with the literature
(Broadbent & Poon, 2015; Entwistle & McCune, 2004; Richardson et al., 2012;
Weinstein et al., 1988). Descriptive analyses revealed interactions with prior domain
knowledge; mainly students with a high pre-test result were likely to make use of
such strategies, whereas for the majority of the sample the data were inconsistent
and lacked linearity. Martin (2012), as well, found that mainly high-performing
students would make use of organizational strategies in an e-learning environment.
Based on similar observations, Eley and Meyer (2004) hypothesized that students’
sometimes contradicting answers to learning strategy items were representative of
the complex and irregular development from an ineffective to a proficient learner.
It is argued that for students with broad knowledge gaps, the “at-risk” group,
effort-related variables might be more relevant than for other learners (Plant et al.,
2005). In our study test attempts showed the strongest impact on this group’s
achievement. As task strategies like rehearsal and repetition are of particular rele-
vance for the acquisition of basic skills such an outcome may not be surprising:
rehearsal helped students to apply mathematical rules more confidently and thus
enabled them to follow their first year lectures (Armstrong & Croft, 1999; Ballard
& Johnson, 2004; Meyer, 2000).
At the same time, taking self-tests is a means to monitor and evaluate the learning
process (Winne, 2004; Zimmerman & Moylan, 2009). Repeatedly engaging in self-
tests thus may also be interpreted as an indicator of a higher level of self-reflection.
The qualitative analyses allowed the cautious interpretation that students who
participated in study groups in their first year at university were more likely to
reflect their learning and to successfully manage the transition to tertiary education.
It is thus argued that social interaction and peer learning are indeed highly relevant
to evoke self-reflection in “at-risk” students, even though such a connection could
not be made based on the quantitative analyses.
5.2 C
ontribution of Data Collected from Web-Based Pre-
courses to the Field of Learning Analytics
It was demonstrated that educational technology is an appropriate way to address

learners with heterogeneous knowledge levels by providing them with tools to cali-
brate and self-monitor their learning (Winne, 2004). Formative self-assessment was
found a fundamental driver of the web-based learning process in mathematics.
Students highly appreciated opportunities to practice and to monitor their learning
(Schumacher & Ifenthaler, 2018; Spector, Ifenthaler, Sampson, & Yang, 2016).
The data collected from the pre-post-test design delivered consistent information
and allowed to relate prior knowledge and learning gains to measures of academic
achievement, thus establishing external validity.
Considering the contribution of pre-course learner data to the field of learning
analytics, the most consistent results were obtained from cognitive variables,
namely test scores (diagnostic pre- and post-test). The number of test attempts was
significantly correlated with pre-course learning gains and thus outperformed other
tracking data like time online, number of page views, or clicks. It is suggested that
this variable is a good indicator of effort and engagement in web-based learning
environments, which is in agreement with previous research on e-learning
(Ledermüller & Fallmann, 2017; Morris et al., 2005; Samson, 2015; Tempelaar
et al., 2015; Zacharis, 2015).
The web-based surveys used to collect information on affective and metacogni-
tive variables showed less consistent results and particularly failed to explain learn-
ing outcomes of the “at-risk” group. It may be hypothesized that the sometimes
skewed answer patterns were influenced by social desirability. It also has been sug-
gested that in web-based environments surveys are answered less conscientiously
(Cook et al., 2000; Fan & Yan, 2010; Nulty, 2008; Tourangeau et al., 2013) and that
high-performing students find it easier to answer metacognitive items, resulting in
interactions between cognitive and metacognitive predictors (Case, 2004; Thiessen
& Blasius, 2008). Concerns thus might be raised regarding the general idea of
“measuring” the use of learning strategies with the help of Likert-scaled items in
e-learning environments; probably more sophisticated ways to evaluate e-learning
are needed to adequately describe the complex construct of self-regulated learning
(Hadwin, Winne, & Nesbit, 2005; Winne & Jamieson-Noel, 2002).
Although some of the data collected from the pre-course were significantly
related to first year performance, this study also showed the limitations of predictive
136 K. Derr et al.
models. The multiple regressions accounted for 35% of the variance in Mathematics
I at most. Thus many students succeeded in spite of a poor pre-test result, and there
also remained a number of students who performed reasonably well in the pre-test
and yet failed their course (Robinson & Croft, 2003). The literature suggests that
indeed many more factors are involved when it comes to study success (Ackerman
et al., 2013; Heublein, Richter, Schmelzer, & Sommer, 2012).
Making individual suggestions based on pre-course data thus does not seem
advisable and even may have counterproductive effects. Students with a positive
prediction might be provided with a false sense of security (Clark & Lovric, 2009),
whereas students with poor prior knowledge might try to avoid the “stigmatization”
of being “at risk” (Case, 2004).
It is suggested that quantitative pre-course evaluations have “blind spots” as they
fail to inform if students’ learning activities remained on the surface or resulted in
deeper understanding or self-reflection. It is argued that the current state of technol-
ogy does not allow to adequately address students who struggle with the learning
process and that human tutoring is needed to identify misconceptions. A claim is
therefore made to not overemphasize the role of predictive modeling for the indi-
vidual student.
At the same time, the outcomes of such course evaluations are considered highly
relevant for practitioners in this area. Students, as well, should be informed of the
outcomes of these analyses and the relevance of basic skills on subsequent achieve-
ment at an early point in time.
5.3 S
uggestions for the Support of “At-Risk” Students
in the Transition Phase Between Secondary and Tertiary
Education
Based on the observations made in this project, we draw the following suggestions
for the design of preparatory courses in mathematics:
1. Raise entering students’ awareness (in this case for the role of basic knowledge
in mathematics) by providing information about the curriculum and tools for
self-diagnosis. The results of pre-course evaluations should be made accessible
for students; reviewing and discussing data from previous cohorts, for example,
can be highly informative and evoke reflection (see example at www.optes.de).
2. Provide an environment to practice and self-monitor learning, fostering the use of
task strategies like repetition, rehearsal, and reactivation of existing knowledge.
3. Provide external guidance like weekly schedules, submission dates, and immedi-
ate feedback by e-tutors.
4. Introduce students to the use of a set of learning strategies. These should include
planning and structuring, self-reflection and analysis and interpretation of test
results, but also social aspects like help seeking and how to benefit from group
work.
Further research should reveal how e-portfolios can be implemented in mathematics

courses and help induce self-reflection (Burks, 2010; McDonald, 2012). Some hands-
on experiments have already been carried out in the optes project, suggesting that it
is quite demanding to meaningfully connect cognitive and metacognitive learning in
an engineering context. It will also have to be explored more deeply how to address
not only different levels of domain knowledge but different needs regarding scaffold-
ing and guidance in e-learning environments (Hannafin & Hannafin, 2010).
Finally, our results also draw attention to the issue of increasing heterogeneity in
first year students’ entry qualifications (Ecclestone, Biesta, & Hughes, 2010; Luk,
2005). Short-term remedial programs like the pre-course described in this study
may help to reactivate school knowledge and thus ease the transition to university,
but they are certainly not sufficient instruments when it comes to broad and funda-
mental gaps in knowledge. For many students a prolonged study preparation would
be needed, providing the domain and meta-knowledge that is required to success-
fully study engineering.
Acknowledgments Support for this publication was provided by the German Federal Ministry
of Education and Research (BMBF) in the context of the Federal “Quality pact for teaching”
(ref. number 01PL17012). Responsibility for the content published in this article, including any
opinions expressed therein, rests exclusively with the authors. Test items and learning materials
are licensed under the Creative Commons Attribution 3.0 Unported and can be provided via
www.optes.de.
References
Abel, H., & Weber, B. (2014). 28 Jahre Esslinger Modell—Studienanfänger und Mathematik. In
I. Bausch, R. Biehler, R. Bruder, P. R. Fischer, R. Hochmuth, W. Koepf, & T. Wassong (Eds.),
Mathematische Vor- und Brückenkurse. Konzepte, Probleme und Perspektiven (pp. 9–19).
Wiesbaden, Germany: Springer.
Ackerman, P. L., Kanfer, R., & Beier, M. E. (2013). Trait complex, cognitive ability, and domain
knowledge predictors of baccalaureate success, STEM persistence and gender differences.
Journal of Educational Psychology, 105(3), 911–927.
Armstrong, P. K., & Croft, A. C. (1999). Identifying the learning needs in mathematics of entrants
to undergraduate engineering programmes in an English university. European Journal of
Engineering Education, 24(1), 59–71.
Artino, A. R., & Stephens, J. M. (2009). Academic motivation and self-regulation: A compara-
tive analysis of undergraduate and graduate students learning online. Internet and Higher
Education, 12, 146–151.
Ashby, J., Sadera, W. A., & Mcnary, S. W. (2011). Comparing student success between develop-
mental math courses offered online, blended, and face-to-face. Journal of Interactive Online
Learning, 10(3), 128–140.
Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning?
The role of self-regulated learning. Educational Psychologist, 40(4), 199–209.
Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate Students'
learning with hypermedia? Journal of Educational Psychology, 96(3), 523–535.
Ballard, C. L., & Johnson, M. (2004). Basic math skills and performance in an introductory eco-
nomics class. Journal of Economic Education, 35(1), 3–23.
138 K. Derr et al.
Bargel, T. (2015). Studieneingangsphase und heterogene Studentenschaft—neue Angebote und ihr

Nutzen: Befunde des 12. Studierendensurveys an Universitäten und Fachhochschulen (Hefte
zur Bildungs- und Hochschulforschung No. 83).
Barnard, L., Lan, W. Y., To, Y. M., Osland Paton, V., & Lai, S.-L. (2009). Measuring self-regulation
in online and blended learning environments. Internet and Higher Education, 12, 1–6.
Barnard-Brak, L., Lan, W. Y., & Paton, V. O. (2010). Profiles in self-regulated learning in the
online learning environment. International review of research in open and distance. Learning,
11(1), 62–80.
Bettinger, E. P., & Long, B. T. (2009). Addressing the needs of underprepared students in higher
education: Does college remediation work? Journal of Human Resources, 44(3), 736–771.
Biehler, R., Fischer, P. R., & Wassong, T. (2012). Designing and evaluating blended learning bridg-
ing courses in mathematics. In Congress of the European Society for Research in Mathematics
Education (pp. 1971–1980).
Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2003). Assessment for learning:
Putting it into practice. Maidenhead, England: Open University Press.
Broadbent, J. (2017). Comparing online and blended learner's self-regulated learning strategies
and academic performance. The Internet and Higher Education, 33, 24–32.
Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in
online higher education learning environments: A systematic review. The Internet and Higher
Burks, R. (2010). The student mathematics portfolio: Value added to student preparation? PRIMUS,
20(5), 453–472.
Carr, M., Bowe, B., & Ní Fhloinn, E. (2013). Core skills assessment to improve mathematical
competency. European Journal of Engineering Education, 38(6), 608–619.
Carson, A. D. (2011). Predicting student success from the Lassi for learning online (LLO). Journal
of Educational Computing Research, 45(4), 399–414.
Case, J. M. (2004). Approaches to learning: A critical examination of inventory responses from
third year chemical engineering students. In A. Buffler, & R. C. Laugksch (Eds.), Proceedings of
the 12th Annual Conference of the Southern African Association for Research in Mathematics,
Science and Technology Education SAARMSTE (pp. 102–110). Retrieved from http://www.
academia.edu
Clark, M., & Lovric, M. (2009). Understanding secondary–tertiary transition in mathematics.
International Journal of Mathematical Education in Science and Technology, 40(6), 755–776.
Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or
internet-based surveys. Educational and Psychological Measurement, 60(6), 821–836.
Cosh cooperation schule:hochschule. (2014). Mindestanforderungskatalog Mathematik (2.0) der
Hochschulen Baden-Württembergs für ein Studium von WiMINT-Fächern. Retrieved from
http://www.mathematik-schule-hochschule.de/images/Aktuelles/pdf/MAKatalog_2_0.pdf
Credé, M., & Phillips, L. A. (2011). A meta-analytic review of the motivated strategies for learning
questionnaire. Learning and Individual Differences, 21(4), 337–346.
Croft, A. C., Harrison, M. C., & Robinson, C. L. (2009). Recruitment and retention of students – An
integrated and holistic vision of mathematics support. International Journal of Mathematical
Education in Science and Technology, 40(1), 109–125.
Derr, K., Hübl, R., & Ahmed, M. Z. (2015). Using test data for successive refinement of an online
pre-course in mathematics. In 14th European Conference on e-Learning ECEL (pp. 173–180).
Derr, K., Hübl, R., & Ahmed, M. Z. (2018). Prior knowledge in mathematics and study success
in engineering: Informational value of learner data collected from a web-based pre-course.
European Journal of Engineering Education, 10(3), 1–16.
Di Pietro, G. (2012). The short-term effectiveness of a remedial mathematics course: Evidence
from a UK university (Discussion Paper Series No. 6358). Bonn.
Dreier, O. (2014). Entwicklung eines Moodle Plug-Ins zur erweiterten Auswertung von Tests:
Bachelor thesis. Duale Hochschule Baden-Württemberg, Mannheim.
Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41(10),
1040–1048.
Ecclestone, K., Biesta, G., & Hughes, M. (Eds.). (2010). Transitions and learning through the
Lifecourse. London, UK: Routledge.
Ehrenberg, R. G. (2010). Analyzing the factors that influence persistence rates in STEM field,
majors: Introduction to the symposium. Economics of Education Review, 29, 888–891.
Eley, M. G., & Meyer, J. H. F. (2004). Modelling the influences on learning outcomes of study
processes in university mathematics. Higher Education, 47(4), 437–454.
Entwistle, N. J., & McCune, V. (2004). The conceptual bases of study strategy inventories.
Educational Psychology Review, 16(4), 325–345.
Fan, W., & Yan, Z. (2010). Factors affecting response rates of the web survey: A systematic review.
Computers in Human Behavior, 26, 132–139.
Faulkner, F., Hannigan, A., & Gill, O. (2010). Trends in the mathematical competency of univer-
sity entrants in Ireland by leaving certificate mathematics grade. Teaching Mathematics and Its
Greefrath, G., Koepf, W., & Neugebauer, C. (2016). Is there a link between preparatory course
attendance and academic success? A case study of degree Programmes in electrical engineer-
ing and computer science. International Journal of Research in Undergraduate Mathematics
Education, 3(1), 143–167.
Hadwin, A. F., Winne, P. H., & Nesbit, J. C. (2005). Roles for software technologies in advancing
research and theory in educational psychology. British Journal of Educational Psychology, 75,
1–24.
Hannafin, M. J., & Hannafin, K. M. (2010). Cognition and student-centered, web-based learn-
ing: Issues and implications for research and theory. In J. M. Spector, D. Ifenthaler, P. Isaias,
Kinshuk, & D. Sampson (Eds.), Learning and instruction in the digital age (pp. 11–23).
Boston, MA: Springer.
Hattie, J. (2009). Visible learning. A synthesis of over 800 meta-analyses relating to achievement.
London, UK: Routledge.
Hell, B., Linsner, M., & Kurz, G. (2008). Prognose des Studienerfolgs. In M. Rentschler & H. P. Voss
(Eds.), Studieneignung und Studierendenauswahl - Untersuchungen und Erfahrungsberichte
(pp. 132–177). Aachen, Germany: Shaker.
Heublein, U., Richter, J., Schmelzer, R., & Sommer, D. (2012). Die Entwicklung der Schwund-
und Studienabbruchquoten an den deutschen Hochschulen: Statistische Berechnungen auf der
Basis des Absolventenjahrgangs 2010. Projektbericht.
Kadijevich, D. (2006). Developing trustworthy timss background measures: A case study on math-
ematics attitude. The Teaching of Mathematics, 17, 41–51.
Karabenick, S. A. (2004). Perceived achievement goal structure and college student help seeking.
Journal of Educational Psychology, 96(3), 569–581.
Kift, S., Nelson, K., & Clarke, J. (2010). Transition pedagogy: A third generation approach to
FYE - a case study of policy and practice for the higher education sector. The International
Journal of the First Year in Higher Education, 1(1), 1–20.
Knospe, H. (2011). Der Eingangstest Mathematik an Fachhochschulen in Nordrhein-Westfalen
von 2002 bis 2010, In Proceedings des 9. Workshops Mathematik für ingenieurwissenschaftli-
che Studiengänge. Wismarer Frege-Reihe (Vol. 2, pp. 8–13).
Krumke, S. O., Roegner, K., Schüler, L., Seiler, R., & Stens, R. (2012). Der Online-Mathematik
Brückenkurs OMB. Eine Chance zur Lösung der Probleme an der Schnittstelle Schule/
Hochschule: DMV Mitteilungen Juni 2012. Retrieved from http://page.math.tu-berlin.
de/~seiler/publications/OMB-eine-Chance.pdf
Lagerlöf, J. N. M., & Seltzer, A. J. (2009). The effects of remedial mathematics on the learning of
economics: Evidence from a natural experiment. The Journal of Economic Education, 40(2),
115–137.
Ledermüller, K., & Fallmann, I. (2017). Predicting learning success in online learning environments:
Self-regulated learning, prior knowledge and repetition. Zeitschrift Für Hochschulentwicklung,
12(1), 79–99.
140 K. Derr et al.
Liston, M., & O'Donoghue, J. (2009). Factors influencing the transition to university service math-
ematics: Part 1 a quantitive study. Mathematics and Its Applications, 28, 77–87.
Luk, H. S. (2005). The gap between secondary school and university mathematics. International
Journal of Mathematical Education in Science and Technology, 36(2–3), 161–174.
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an ‘early warning system' for
educators: A proof of concept. Computers & Education, 54(2), 588–599.
Martin, P.-Y. (2012). Lernstrategien und Umgang mit ICT von Studienanfängerinnen und -anfän-
gern. Zurich, Switzerland: Universität Zürich.
MAXQDA. Berlin, Germany: VERBI Software Consult Sozialforschung GmbH.
McDonald, B. (2012). Portfolio assessment: Direct from the classroom. Assessment & Evaluation
in Higher Education, 37(3), 335–347.
Meyer, J. H. F. (2000). Variation in contrasting forms of 'memorising' and associated observables.
British Journal of Educational Psychology, 70(2), 163–176.
Meyer, J. H. F., & Eley, M. G. (1999). The development of affective subscales to reflect variation
in students’ experiences of studying mathematics in higher education. Higher Education, 37,
197–216.
Michinov, N., Brunot, S., Le Bohec, O., Juhel, J., & Delaval, M. (2011). Procrastination, par-
ticipation, and performance in online learning environments. Computers & Education, 56(1),
243–252.
Morris, L. V., Finnegan, C., & Wu, S.-S. (2005). Tracking student behavior, persistence, and
achievement in online courses. The Internet and Higher Education, 8(3), 221–231. https://doi.
org/10.1016/j.iheduc.2005.06.009
Moss, B. G., & Yeaton, W. H. (2006). Shaping policies related to developmental education: An
evaluation using the regression-discontinuity design. Educational Evaluation and Policy
Analysis, 28(3), 215–229.
Mullis, I. V. S., Martin, M. O., Foy, P., & Arora, A. (2012). TIMSS 2011 International Results in
Mathematics.
Newman, R. S. (2002). How self-regulated learners cope with academic difficulty: The role of
adaptive help seeking. Theory Into Practice, 41(2), 132–138.
Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be
done? Assessment & Evaluation in Higher Education, 33(3), 301–314.
Pardo, A., & Kloos, C. D. (2011). Stepping out of the box: Towards analytics outside the learning
management system. In Proceedings of the 1st International Conference on Learning Analytics
and Knowledge (pp. 163–167).
Parsons, S. J., Croft, T., & Harrison, M. (2009). Does students' confidence in their ability in math-
ematics matter? Teaching Mathematics and Its Applications, 28(2), 52–68.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of
the Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor, MI: University of
Michigan.
Plant, E. A., Ericsson, K. A., Hill, L., & Asberg, K. (2005). Why study time does not predict grade
point average across college students: Implications of deliberate practice for academic perfor-
mance. Contemporary Educational Psychology, 30(1), 96–116.
Polaczek, C., & Henn, G. (2008). Vergleichende Auswertung des Mathematik-Eingangstests.
Retrieved from http://www.fh-aachen.de/uploads/media/Eingangstest_WS_07_08.pdf
Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university Students'
academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138(2),
353–387.
Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychoso-
cial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin,
130(2), 261–288. https://doi.org/10.1037/0033-2909.130.2.261
Robinson, C. L., & Croft, A. C. (2003). Engineering students—diagnostic testing and follow up.
Teaching Mathematics and Its Applications, 22(4), 177–181.
Robson, C. (2011). Real world research: A resource for users of social research methods in applied
settings (3rd ed.). Chichester, UK: Wiley.
Samson, P. J. (2015). Can student engagement be measured? And, if so, does it matter? In Frontiers
in Education Conference Conference Proceedings.
Schiefele, U., & Wild, K. P. (1994). Lernstrategien im Studium: Ergebnisse zur Faktorenstruktur
und Reliabilität eines neuen Fragebogens. Zeitschrift Für Differentielle Und Diagnostische
Psychologie, 15(4), 185–200.
Scholes, V. (2016). The ethics of using learning analytics to categorize students on risk. Educational
Technology Research and Development, 64(5), 939–955.
Computers in Human Behavior, 47, 394–407.
SEFI Mathematics Working Group. (2013). A framework for mathematics curricula in engineering
education. Brussels, Belgium: Author.
Smith, G. G., & Ferguson, D. (2005). Student attrition in mathematics e-learning. Australasian
Journal of Educational Technology, 21(3), 323–334. Retrieved from https://ajet.org.au/index.
php/AJET/article/view/1323
Söderlind, J., & Geschwind, L. (2017). More students of better quality? Effects of a mathematics
and physics aptitude test on student performance. European Journal of Engineering Education,
42(4), 445–457.
Spector, J. M., Ifenthaler, D., Sampson, D. G., & Yang, L. (2016). Technology enhanced formative
assessment for 21st century learning. Journal of Educational Technology & Society, 19(3), 58–71.
SPSS Version 23. Armonk, NY: IBM Corp.
Stake, R. E. (1994). Case Studies. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative
research (pp. 236–247). Thousand Oaks, CA: Sage.
Street, H. (2010). Factors influencing a learner’s decision to drop-out or persist in higher education
distance learning. Online Journal of Distance Learning Administration, 13(4), 4. Retrieved
from http://www.westga.edu/~distance/ojdla/winter134/street134.html
Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for
feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior,
47, 157–167.
Thiessen, V., & Blasius, J. (2008). Mathematics achievement and mathematics learning strategies:
Cognitive competencies and construct differentiation. International Journal of Educational
Research, 47(6), 362–371.
Tourangeau, R., Conrad, F. G., & Couper, M. P. (2013). The science of web surveys. Oxford, UK:
Oxford University Press.
Vuik, K., Daalderop, F., Daudt, J., & van Kints, R. (2012). Evaluation MUMIE—Online math edu-
cation: aerospace engineering and computer science 2011–2012. Reports of the Delft Institute
of Applied Mathematics, 12–13.
Weinstein, C. E., Zimmermann, S. A., & Palmer, D. R. (1988). Assessing learning strategies. The
design and development of the LASSI. In C. E. Weinstein, E. T. Goetz, & P. A. Alexander
(Eds.), Learning and study strategies. Issues in assessment, instruction, and evaluation
(pp. 25–40). San Diego, CA: Academic Press.
Winne, P. H. (2004). Students’ calibration of knowledge and learning processes: Implications
for designing powerful software learning environments. International Journal of Educational
Research, 41(6), 466–488.
Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about
study tactics and achievement. Contemporary Educational Psychology, 27, 551–572.
Yin, R. K. (2009). Case study research. Design and methods. Thousand Oaks, CA: Sage.
Zacharis, N. Z. (2015). A multivariate approach to predicting student outcomes in web-enabled
blended learning courses. Internet and Higher Education, 27, 44–53.
Zhang, G., Anderson, T. J., Ohland, M. W., & Thorndyke, B. R. (2004). Identifying factors influ-
encing engineering student graduation: A longitudinal and crossinstitutional study. Journal of
Engineering Education, 93, 313–320.
Zimmerman, B. J., & Moylan, A. (2009). Self-regulation: Where metacognition and motivation
intersect. In D. J. Hacker (Ed.), Handbook of metacognition in education (pp. 299–316).
New York, NY: Routledge.
Zimmerman, B. J., Moylan, A., Hudesman, J., White, N., & Flugman, B. (2011). Enhancing
self-reflection and mathematics achievement of at-risk urban technical college students.
Psychological Test and Assessment Modeling, 53(1), 141–160.
Chapter 9
Issues and Challenges for Implementing
Writing Analytics at Higher Education
Duygu Bektik
1 Introduction
Most learning analytics applications provide quantitative information about learn-

ers, e.g. based on how many times they have logged in to learning platforms, viewed
a forum post, and replied to it. However, learning analytics can move beyond report-
ing these quantitative logs and provide information on the quality of the contribu-
tions students made. One interest for learning analytics is in its potential for the
analysis of discourse data. This is called ‘writing analytics’ which involves mea-
surement and analysis of written texts, for understanding writing processes in their
educational contexts (Buckingham Shum et al., 2016).
Effective written communication is an essential skill which promotes educa-
tional success for undergraduates. One of the key requirements of good academic
writing in undergraduate higher education courses is that students must develop a
critical mind and learn how to construct sound arguments in their discipline. Thus,
when assessing student essays, educators look for students’ ability to present and
pursue well-reasoned and strong arguments. Yet, critical thinking and argumenta-
tion are not ‘a student’s mother tongue’. Undergraduate students struggle with argu-
mentation: they are either unaware that they are expected to develop an argument in
their essays, have difficulty in arguing, or have difficulty in constructing argumenta-
tive pieces. Thus, students require effective, timely, frequent formative feedback
from their tutors to gain this skill. However, assessing written texts is a labour-
intensive process. Marking and giving detailed feedback and commenting on sev-
eral drafts of essays can be time-consuming, which is why writing analytics
applications can aid this process.
D. Bektik (*)
The Open University, UK, Milton Keynes, UK

https://doi.org/10.1007/978-3-319-64792-0_9
144 D. Bektik
Learning analytics with a focus on the use of discourse to support learning and
teaching is being developed at the intersection of fields such as automated
assessment, learning dynamics, deliberation platforms, and computational linguis-
tics. There exist today some powerful natural language processing systems which
automatically detect writers’ argumentative moves in scholarly/academic texts that
can be adopted as part of writing analytics applications. When assessing their stu-
dents’ essays, educators could benefit from the available automated textual analysis
which can detect students’ argumentative discourse. The studies given in this chap-
ter use a particular language analysis tool, the Xerox Incremental Parser (XIP) as an
exemplar of this type of automated technology.
This chapter reviews student writing at higher education, automated analysis of
written text, and issues and challenges faces in implementing writing analytics
application, together with knowledge from empirical research as well as the theo-
retical considerations.
2 Background
2.1 Writing at Higher Education
Learning in higher education involves new ways of understanding, interpreting, and

organising knowledge. Academic writing in higher education enables professional
advancement for university students as it nurtures thinking and reflection. However,
students’ writing background dates back to school writing, which differs from aca-
demic writing in higher education.
Student academic writing is more than punctuation and grammar. When assess-
ing student writing, academic tutors look for students’ ability to present and pursue
well-reasoned and strong arguments through scholarly argumentation (Sandoval &
Millwood, 2005). Writing a university assignment is a thought-provoking activity
that requires particular skills of critical thinking and argumentation which are not ‘a
student’s mother tongue’ (Lea & Street, 1998). Students, especially those in their
first year at university, are unused to this form of writing, and most of them see
themselves as novices (Lea & Street, 1998).
Undergraduates are expected to adopt higher-order writing skills such as argu-
ment writing and criticality, which are not taught or necessarily practised in second-
ary school. Thus, novice student writers join higher education with partial or
incorrect conceptions about argumentation (Andrews, 2010), they are not familiar
with what they are expected to produce, and they have difficulty in constructing
argumentative pieces. Undergraduate students struggle with argumentation: they
are either unaware that they are expected to develop an argument in their essays or
have difficulty in arguing (Lillis & Turner, 2001; Norton, 1998), often because they
have learned different concepts of argument at secondary school (Hounsell, 1984).
Additionally, argumentation is often not adequately explained by their academic
tutors, who often struggle to provide effective feedback which would prompt good
examples of argumentation (Coffin et al., 2002; Teufel & Kan, 2009), which is why
9 Issues and Challenges for Implementing Writing Analytics at Higher Education 145
adopting an effective writing analytics approach could potentially support to over-

come these problems, yet its implementation brings up new dimension of issues and
challenges.
2.2 Automated Analysis of Argumentation
‘The best way to improve one’s [academic] writing skills is to write, receive feed-
back from an instructor, revise based on the feedback, and then repeat the whole
process as often as possible’ (Simsek, Buckingham Shum, Sándor, De Liddo, &
Ferguson, 2013). This cycle requires tutors to read and provide feedback on student
essays, which can create an enormous workload (Simsek et al., 2013). Assessing
written texts is a labour-intensive process for academic tutors. Marking and giving
detailed feedback and commenting on essays can be time-consuming. This problem
led researchers to study ways of developing applications that can automatically
analyse and evaluate essays for assessment purposes.
Educators expect their students to learn to write in an academically sound way,
specifically to learn to make knowledge-level moves and claims in their essays by
recognising and deploying scholarly rhetoric. Argumentation is articulated by meta-
discourse. Meta-discourse refers to the features of text that provide linguistic cues
which engage the readers and explicitly convey the authors’ intended meaning,
expressing their viewpoint, argument, and claim and signalling their stance (Hyland,
2005). Therefore, when assessing their students’ writing, educators will, among
other features (e.g. spelling and grammar), be looking for scholarly meta-discourse
as an indicator of argumentation.
Natural language processing (NLP) is the automatic processing of natural human
language, such as English, rather than a specialised artificial computer language.
‘NLP is the application of computational methods for the purpose of analysing
language-related characteristics of electronic files of text or speech’ (Shermis &
Burstein, 2013, p. 56). Today, some natural language processing systems which
automatically detect authors’ rhetorical moves in scholarly/academic texts exist.
The archaic definition of rhetoric is the art and study of the use of language with
persuasive effect in any given field (Dawson, 1998), or the art of trickery, a way of
masquerading and obscuring information (Maynard, 1998). A more contemporary
definition of rhetoric refers to the skill to analyse, evaluate, and employ writing
strategies in order to respond to the audience and being aware of one’s own ideo-
logical stance and the audience’s stance (Cook, 2002). Rhetorical ‘move’ refers to a
discoursal unit that performs the communicative purposes of a text (Swales, 1990).
One approach to automatic identification of rhetorical moves is ‘Xerox
Incremental Parser’ (XIP) (Aït-Mokhtar, Chanod, & Roux, 2002), which assigns
rhetorical move labels to rhetorically salient sentences only. The study described in
this chapter adopts the XIP as an exemplar of this type of automated technology. The
framework is implemented as the rhetorical module of the XIP, which detects and
labels rhetorically salient sentences in scholarly writing based on the identification
of meta-discourse conveying the author’s rhetorical strategy. The unit of analysis
146 D. Bektik
XIP uses is at sentence level, and each sentence can have multiple labels. The labels
XIP has are shown in the following table with description and examples:
Label Description Example

Summary Summarising the goals or results of the article The goal of this study…
Emphasis Emphasising the importance of ideas … is crucial for
understanding
Background Describing background knowledge necessary for Recent studies indicate…
understanding the article’s contribution
Contrast Describing tensions, contrasts between ideas, In contrast with previous
models, or research directions hypothesis…
Novelty Conveying that an idea is new New insights provide direct
evidence…
Tendency Describing emerging research directions Growing recognition of the
importance…
Open Describing problems that have not been solved Little is known…
questions
When assessing students’ essays, educators could benefit from the available auto-
mated textual analysis which can detect meta-discourse. This way, academic tutors
could overcome not only limited time they have for providing feedback but also the
issue of providing effective practical examples of what argumentation should look
like with readily available automatic machine output. However, the ethical and pri-
vacy concerns of whether these technologies can be used to analyse student writing
reliably have been an ongoing debate since several decades (Attali, 2013).
2.3 Ethical Concerns Associated with Writing Analytics
Writing analytics and automated marking and feedback of student essays seem to be
useful to overcome the problems tutors experience. Writing analytics can support the
process of improving students’ writing skills, as its results can start to produce mean-
ingful dialogues between students and teachers (Ras, Whitelock, & Kalz, 2015). Yet,
there has been an ongoing tension between the writing teachers, researchers, aca-
demic tutors, and essay markers on the one side and the developers of such auto-
mated technologies on the other regarding the use of automated essay evaluation.
‘There is an inherent suspicion that technology can corrupt the essence of a funda-
mentally human activity’ (Elliot & Williamson, 2013). Since many tutors see auto-
mated technology as a threat, this tension has often appeared in academic literature.
On the one hand, there is significant support for automated essay scoring (AES)
as ‘automated essay scoring and evaluation becomes more widely accepted as an
educational supplement for both assessment and classroom instruction’ (preface in
Shermis & Burstein, 2003). There are several studies showing that AES systems
work well, with studies reporting high agreement rates between AES systems and
human markers (Bridgeman, Trapani, & Attali, 2012; Burstein & Chodorow, 2010;
Landauer, Laham, & Foltz, 2003).
On the other hand, there has been and still is significant opposition to AES, par-
ticularly to the idea, originated by Page, that ‘it might replace human scoring’
(Ericsson & Haswell, 2006; Herrington & Moran, 2012). Harsh criticism comes
particularly from the community of writing researchers. The tension is originated by
the awareness of the limitations and dangers of such automated text analysis sys-
tems and what such systems cannot do. Critiques in Ericsson and Haswell’s (2006)
collection provide the following reasons for this tension. Writing teachers and
researchers are worried because they question:
• Whether such systems can be gamed or fooled and whether students can break
these systems
• Whether machine analysis programmes can fully understand the meaning of
texts
• How students would react when they find out their work has been evaluated
automatically
• How closely such software matches the careful evaluation of writing teachers
• Where automated text analysis leads the teaching profession and would tutors
have greater or less control over courses
It is important that participants trust a machine that is analysing human writing,
and therefore it is important to hear what queries or even doubts they have about
how such a tool works, as well as how similar its output is to their judgement of
quality, and how it can be improved, in the process of implementing writing analyt-
ics. This approach has been adopted to reveal any potential issues and challenges for
implementing writing analytics at higher education, which is explained next.
3 Study
The qualitative data is collected in two parts: one-to-one interviews and a focus
group, in order to understand and explore the opinions of educators on the matter of
how they define the attributes of good student writing and to what degree the auto-
mated text analysis can capture the presence of these attributes.
3.1 One-to-One Interviews
Firstly, individual interviews were conducted with the Open University educators.
Seven one-to-one interviews were conducted in total; each took around 90 min, and
each interview consisted three sections:
1. Section one was a general, introductory part of the interview, in which tutors
briefly set out their views on assessment and what they felt to be good student
writing. This section investigated how these educators defined the quality of
writing and its most valuable characteristics in this context.
148 D. Bektik
2. Section two was the essay-marking exercise. In this section, all tutors were given
a same student essay and their usual marking rubric to mark the essay. They were
specifically asked to highlight the sentences that they thought had a positive
effect on awarding a good mark, not just in terms of writing style but anything
(e.g. spelling, grammar, content accuracy, references, etc.) that they considered
should influence the quality and the essay mark.
3. Section three was a follow-up question-and-answer session on the highlighting
exercise, to discuss specific assessment decisions by the participants and to learn
why they had highlighted particular sentences.
3.2 Focus Group
Following these one-to-one interviews, a focus group with different educators and
senior researchers in the area of academic writing were carried out. The aim was to
observe and comprehend the ideas and the interaction between experts about imple-
menting writing analytics at higher education, to discuss any ethical concerns they
might have, and to discuss possible ways of resolving these concerns. One focus group
discussion was held with six participants (different to the one-to-one interviews) at the
Open University, UK. The study was advertised in the Faculty of Education and
Languages (FELS) since many academics in this faculty have a particular interest in
the area of student academic writing and have experience in teaching and marking
student essays. The participant selection was not confined to these people, however,
and an advertisement was also sent out to all those who carry out research in student
writing and who matched the criteria of experience in teaching and marking.
The focus group discussion was started with a presentation on how the XIP tool
works, what research has been carried out so far, and its results. All focus group
participants were present for the presentation as well as for the focus group study
itself. A question-and-answer session followed the presentation. The focus group
was in three sections. Before the first section began, participants wrote down their
initial thoughts and/or misgivings about the possibility of using a writing analytics
approach in their educational practices. Participants were then given two pages of
student writing, and, without any guidance, they were asked to highlight the sen-
tences that they thought illustrated good-quality writing, with respect to good
critical, argumentative, or analytical statements only. Then, the XIP analysis of the
same writing was shown (note that the machine was not trained with this instru-
ment). After receiving their reactions to the XIP results, participants were then asked
to discuss the sentence category (i.e. summary, background, contrast) that XIP might
have assigned to each salient sentence and whether they agreed with XIP’s choice.
In the second section, after participants were informed about the potential of XIP,
they were asked whether they would consider using XIP to analyse their own stu-
dents’ writing if the XIP tool were to be made publicly available to analyse any form
of student writing. They wrote down three important features that would convince
them to use the tool. After participants had shared their ideas, they discussed what
would be the most important factor in their decisions.
In the final section, participants discussed what might need improvement and
what sorts of change they would make to improve the approach. At the close of the
session, participants were asked to write down their final thoughts and/or doubts
about implementing writing analytics.
3.3 Results
3.3.1 One-to-One Interviews
The essay-marking exercise of the interviews required tutors to highlight the key
sentences that they thought would have a positive effect on the final essay mark.
They were each given the same student essay. Tutors were asked to perform regular
marking activity with reference to the same rubric they use and to talk through their
decisions, which were audio recorded. The essay was five pages long, excluding the
bibliography, and contained around 3000 words. There were 88 sentences in total.
Taking the tools and the tutors’ highlights, and the similarity and overlap between
(a) tutor pairings, and (b) the tool and each tutor, were then measured using Jaccard
similarity index. The Jaccard analysis results showed a highly significant similarity
between the highlights of the tool and the first tutor (p ≤ 0.01) and no significant
similarity between the marks of tool and those of the other six tutors (p ≥ 0.46). The
Jaccard analysis was also performed between tutors to find out whether they agreed
with each other and if their marking was similar. According to Jaccard analysis
results, there are no significant similarities between any of the tutor pairings.
According to the Jaccard analysis results, the highlighting carried out by each tutor
was significantly different to that of the others. The assumption had been that tutors
would share the same understanding about what makes good-quality student writ-
ing, so their highlights would be similar, and the overlap between the XIP and the
tutors could be measured reliably. However, this proved not to be the case. There
could be various explanations for this result. Considering that all these participants
had more than 5-year experience of marking essays using the same marking scheme,
one explanation could be that human marking is not reliable. Human marking is not
always reliable, which supports the assertion that using automated technologies and
writing analytics to support educators’ essay assessment processes could be a good
idea, yet it is a challenge to implement writing analytics taking tutor marking as a
benchmark if it is not reliable.
It is important to underline though in normal circumstances to standardise the
Open University marking would be balanced with the second marker’s decision; and
with the third marker’s in case of a possible disagreement during the coordination
meetings. Therefore, based on this small sample size, it is not credible to generalise
the result that every tutor marks completely differently and unreliably. Yet, it is sig-
nificant to note that human marking and assessment may vary depending on several
factors, whereas automated analysis always provides the same result every time.
This supports the argument that there is a benefit to using an automated technology,
which could support educators’ marking.
150 D. Bektik
Following the essay-marking exercise, tutors talked about some problems with
marking that they experience. One of the problems they raised was about the ‘sub-
jectivity of human marking’. Tutors also mentioned they only have a limited time to
mark an essay; hence they spend too much time marking papers and feel pressured
with hours of grading. Additionally, markers mentioned that they struggle with giv-
ing feedback and commenting and annotating students’ essays, which are even
more time-consuming to make sure they gave a clearer explanation about why they
have given a specific feedback to their students. When tutors raised the problems
they experience with assessment, they were asked whether they would consider
using a computational language technology that might potentially help them to
overcome such problems. They stated the worry that they might be replaced by
technology and felt uncomfortable discussing how technology might be helpful to
overcome their problems.
3.3.2 Focus Group
The focus group session was both audio and video recorded, and a full verbatim
transcription (Poland, 1995) approach was followed. This approach involves noting
down both the non-verbal actions like gestures, mimics, gazes, and nods and verbal
actions signifying hesitations, ignorance, laughs, sarcasm, confusion, and excite-
ment, like confusion in the tone of the voice, murmurs, hums, okays, etc. Adding
observational data like facial expressions made it possible to observe how people’s
ideas had changed and were also influenced by others.
After transcription, qualitative thematic analysis of the qualitative focus group
data was carried out. The responses yielded data for content analysis that permitted
theme creation based on the frequency (number of appearances) and intensity (emo-
tion) of the responses of the six participants. Analysis was undertaken both of the
verbal data and the observational data of facial expressions. From the qualitative
analysis of the focus group data, following overarching themes emerged, which are
given in the next sections.
Theme 1: Belief
When academic tutors and writing researchers came to the focus group, they were
initially not inclined to use an automated technology to analyse student writing and
did not expect to gain any benefit from it. The participants came to the focus group
session with preconceptions about automated technologies. Their belief was that
such technologies were developed for commercial return and that they can never be
as good as human markers; the aim of using automatic technologies is to automate
the marking, not to support assessment processes. This shows in what ways writing
analytics systems can be biased. However, a comparison of their initial and final
thoughts demonstrated a change in the participants’ opinion regarding what they
believe an automated text analysis is capable of. Their initial thoughts were con-
cerned with trusting a machine; their final thoughts focused on motivating its use.
Changes in their opinion occurred at different points. For instance, after the pre-
sentation session on what the tool does, learning about what had been already found
in prior research in the earlier quantitative studies (Simsek et al., 2015) changed the
participants’ opinions positively towards the overall research. As they understood
more, they became more interested. Similarly, after completing the highlighting
activity, when participants examined the tool’s analysis and compared them with
their own highlights, they were impressed. This shows that they did not expect the
analysis of the tool to resemble their own decisions. When the participants found
out the tool’s highlights were congruent with their highlights, their attitude was very
positive which was a shift from their initial opinion.
This study revealed that when writing teachers and researchers are introduced to
how automated technologies work, they are able to gain a better understanding of
such writing analytics tools’ capabilities and limitations. When they are made part
of the process, in other words when it is made transparent to them, their opinion
shifts and their bias can be broken.
Theme 2: Power and Politics
The participants were happier to discuss the weaker points of the tool than they were
identifying its strengths, which implies that they did not want its quality to equal
theirs. The theme power and politics emerged since the participants wanted to feel
superior to automated technologies and ‘harness’ and control them to obtain
benefit.
Emerging from the focus group discussions, the underlying issues of power and
politics were due to the participants’ fear of:
• What might happen to the future of the teaching profession and them losing their
jobs as a result of that
• Being judged by their students who could potentially compare human and
machine results
This finding triangulates with the literature (Attali & Burstein, 2006; Elliot &
Williamson, 2013; Ericsson & Haswell, 2006; Shermis & Burstein, 2013) indicat-
ing the suspicion and tension of writing teachers and researchers towards the use of
automated text analysis. It also triangulates and tallies with the one-to-one inter-
views, where tutors felt uncomfortable discussing how technology might help to
overcome the problems they experience with assessment and stated the worry that
they might be replaced by technology.
The study showed the key element which made participants open to accepting
the idea of using such technology, that is, the ‘power and politics’. They wanted to
feel that they are in control of things and superior over the technology. They wanted
to be the ‘power’ behind such technologies that should be driven and ‘harnessed’ by
them. Eventually, they wanted to decide how and to what extent they would like to
use the automated support. Educators, tutors, and markers wanted to be assured that
they retain the power themselves in any decision.
152 D. Bektik
Theme 3: Problems
The participants talked about some problems that markers and their students experi-
ence which automated support could potentially solve. These were identified as:
• The subjectivity of human assessment and marking.
• The limited time that markers have to assess an essay.
• The possibility that markers do not necessarily notice that their students are actu-
ally making an analytical point, since most of the markers are not linguists.
• Markers need to improve the quality of their feedback and make sure they give a
clear explanation of why they have given a specific feedback to students.
• Markers need to generate discussion with students who are required to reflect on,
critique, and edit their work.
Some of the problems identified tally well with the one-to-one interviews and are
congruent with the literature. The problems of subjectivity in human marking, time
limitations, and the need to provide better feedback and examples to ensure students
understand their reasoning became evident in the one-to-one interviews with tutors.
Earlier research (Lea & Street, 1998) indicating that academic tutors experience
difficulty with providing effective examples and feedback is also supported by the
focus group findings.
These problems, such as the labour-intensive, time-consuming essay assess-
ment problem, could potentially be solved through automated support. Considering
it took around 15 min for the focus group participants to highlight 13 sentences, it
could be time efficient to use the automated support as the XIP analysis, for exam-
ple, took less than a minute for the same piece. Additionally, automated support
potentially could help to overcome the subjectivity of human marking. Participants
were honest about how subjective their marking can be and that there was a mis-
match between the way they interpret and mark things. This is a very critical point
showing the possible inconsistencies between human markers. In line with earlier
findings, human markers can disagree with each other and therefore they do not
necessarily come to the same conclusion as their peers; which is a reliability con-
cern. The automated output on the other hand is always the same, stays the same,
and is not subjective. The writing analytics tool could therefore potentially be
useful to help educators overcome this problem by using it for instance as a self-
reflection tool.
4 Conclusion
This chapter evaluates the issues and challenges in the deployment of writing ana-
lytics to support assessment in higher education. It provides a systematic investiga-
tion of implementing an example writing analytics at higher education and discusses
the issues and challenges revealed during this attempt. The studies conducted with
academic tutors and writing researchers showed that implementing writing analyt-
ics applications can be useful to support feedback process, yet its limitations and
risks should be acknowledged. Although these risks and limitations of using an
automated text analysis were mentioned several times through during the discus-
sions, the final thoughts of the participants were positive towards using writing ana-
lytics, and there was general agreement about the idea that writing analytics could
support both educators and students.
It is true that current AES systems do not mimic human markers’ ability to mea-
sure conceptual reasoning; thus AES measures a narrower range of skills than
human markers (Deane, 2013), though they could measure a lot that human markers
ignore. If human markers are inconsistent and unreliable, then the machine cannot
be trained effectively (Bridgeman, 2013). Therefore, the aim of mimicking human
markers is a difficult task to achieve. However, in order to deploy an AES system by
considering such limitations, this deployment must be sensitive to AES’ own limita-
tions as well. It does not understand the essay, and therefore it is limited to measur-
ing a subset of the written context; therefore, AES should currently be considered as
a ‘complement to human scoring’ (Attali, 2013, p. 194). A ‘division of labour’
approach (Attali, 2013, p. 194) between human markers and machines can be used
to overcome such issues. This study prompts consideration of how human markers
and machines can work well together and mutually complement each other for their
own sake and for their students. Advancing automated support for assessment is key
when the strengths of both sides can be brought together: the speed and reliability
of the machines and the vast capabilities and the knowledge of the human markers.
In summary, this study has proposed that, at the current time, automated text
analysis should not be the sole method of evaluating student writing. Instead, it
should be used in combination with human evaluation. It should be recognised that
machines do not currently fully understand the language itself, the accuracy of the
written material, the content, and the beauty and subtlety of sophisticated argumen-
tation that would be credited by human markers because it flows beautifully
(Whitelock & Bektik, 2018). Machines have limited capacity to understand language
and literacy; this capacity is mostly dependent on the rules that its developers have
written to train them. On the other hand, human language has endless possibilities
of creating and forming new sentences each time. Therefore, machines and human
markers should complement each other, with the aim of providing better feedback
to students.
When adopting writing analytics, the relationship between humans and machines
should be mutually inclusive rather than exclusive considering all stakeholders.
This requires resolving the ongoing tensions between the researchers of writing and
developers of automated essay evaluation tools, which, as this study has empha-
sised, can be achieved through the importance of the ‘decisive power’ that academic
tutors and markers require to overcome their tension and worry about the use of
automated text analysis and writing analytics.
154 D. Bektik
References
Aït-Mokhtar, S., Chanod, J.-P., & Roux, C. (2002). Robustness beyond shallowness: Incremental
deep parsing. Natural Language Engineering, 8(2–3), 121–144.
Andrews, R. (2010). Argumentation in higher education: Improving practice through theory and
research. New York: Routledge.
Attali, Y. (2013). Validity and reliability of automated essay scoring. In M. D. Shermis & J. C.
Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new
directions (pp. 181–199). Oxon, UK: Routledge.
Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of
Technology, Learning and Assessment, 4(3).
Bridgeman, B. (2013). Human ratings and automated essay evaluation. In M. D. Shermis &
J. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new
directions (1st ed., pp. 221–232). Oxon, UK: Routledge.
Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of
essays: Differences by gender, ethnicity, and country. Applied Measurement in Education,
25(1), 27–40.
Buckingham Shum, S., Knight, S., McNamara, D., Allen, L., Bektik, D., & Crossley, S. (2016).
Critical perspectives on writing analytics. In Paper Presented at the Proceedings of the Sixth
International Conference on Learning Analytics & Knowledge, Edinburgh, UK.
Burstein, J., & Chodorow, M. (2010). Progress and new directions in technology for automated
essay evaluation. In R. Kaplan (Ed.), The Oxford handbook of applied linguistics (2nd ed.,
pp. 487–497). Oxford, UK: Oxford University Press.
Coffin, C., Curry, M. J., Goodman, S., Hewings, A., Lillis, T., & Swann, J. (2002). Teaching aca-
demic writing: A toolkit for higher education. New York: Routledge.
Cook, K. C. (2002). Layered literacies: A theoretical frame for technical communication peda-
gogy. Technical Communication Quarterly, 11(1), 5–29.
Dawson, P. (1998). The rhetoric and bureaucracy of quality management: A totally questionable
method? Personnel Review, 27(1), 5–19.
Deane, P. (2013). On the relation between automated essay scoring and modern views of the writ-
ing construct. Assessing Writing, 18(1), 7–24.
Elliot, N., & Williamson, D. M. (2013). Assessing writing special issue: Assessing writing with
automated scoring systems. Assessing Writing, 18(1), 1–6.
Ericsson, P. F., & Haswell, R. H. (2006). Machine scoring of student essays: Truth and conse-
quences. Logan, UT: Utah State University Press.
Herrington, A., & Moran, C. (2012). Writing to a machine is not writing at all. In N. Elliot &
L. Perelman (Eds.), Writing assessment in the 21st century: Essays in honor of Edward
M. White (pp. 219–232). New York: Hampton Press.
Hounsell, D. (1984). Essay planning and essay writing. Higher Education Research and
Development, 3, 13–31.
Hyland, K. (2005). Metadiscourse: Wiley Online Library.
Landauer, T. K., Laham, D., & Foltz, P. W. (2003). Automated scoring and annotation of essays
with the intelligent essay assessor. Automated essay scoring: A crossdisciplinary perspective
(pp. 87–112). Mahwah, NJ: Lawrence Erlbaum Associates.
Lea, M., & Street, B. V. (1998). Student writing in higher education: An academic literacies
approach. Studies in Higher Education, 23, 157–172.
Lillis, T., & Turner, J. (2001). Student writing in higher education: Contemporary confusion, tra-
ditional concerns. Teaching in Higher Education, 6, 57–68.
Maynard, A. (1998). Competition and quality: Rhetoric and reality. International Journal for
Quality in Health Care, 10(5), 379–384.
Norton, L. S. (1998). Essay-writing: What really counts? Higher Education, 20, 411–442.
Poland, B. D. (1995). Transcription quality as an aspect of rigor in qualitative research. Qualitative
Inquiry, 1, 290–310.
Ras, E., Whitelock, D., & Kalz, M. (2015). The promise and potential of e- assessment for learn-
ing. In P. Reimann, S. Bull, M. Kickmeier-Rust, R. Vatrapu, & B. Wasson (Eds.), Measuring
and visualizing learning in the information-rich classroom (pp. 21–40). New York: Routledge.
Sandoval, W. A., & Millwood, K. A. (2005). The quality of students’ use of evidence in written
scientific explanations. Cognition and Instruction, 23, 23–55.
Shermis, M. D., & Burstein, J. (2013). Handbook of automated essay evaluation: Current applica-
tions and new directions. Oxon, UK: Routledge.
Shermis, M. D., & Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspec-
tive. Mahwah, NJ: Lawrence Erlbaum.
Simsek, D., Buckingham Shum, S., Sándor, Á., De Liddo, A., & Ferguson, R. (2013). XIP dash-
board: Visual analytics from automated rhetorical parsing of scientific metadiscourse. In 1st
international workshop on discourse-centric learning analytics. (3rd international conference
on learning analytics & knowledge, 8 April 2013, Leuven, Belgium). Open Access Eprint:
(http://oro.open.ac.uk/37391).
Simsek, D., Sandor, A., Buckingham Shum, S., Ferguson, R., De Liddo, A., & Whitelock, D.
(2015). Correlations between automated rhetorical analysis and tutors’ grades on student
essays. In Proceedings of the fifth international conference on learning analytics and knowl-
edge (pp. 355–359). New York: ACM.
Swales, J. (1990). Genre analysis: English in academic and research settings. Cambridge, UK:
Cambridge University Press.
Teufel, S., & Kan, M.-J. (2009). Robust argumentative zoning for sensemaking in scholarly docu-
ments. In Proceedings of the 2009 international conference on Advanced language technolo-
gies for digital libraries (NLP4DL'09/AT4DL'09).
Whitelock, D., & Bektik, D. (2018). Progress and challenges for automated scoring and feedback
systems for large-scale assessments. In J. Voogt et al. (Eds.), Second handbook of informa-
tion technology in primary and secondary education. Basel, Switzerland: Springer. https://doi.
org/10.1007/978-3-319-53803-7_39-1
Chapter 10
Digital Applications as Smart Solutions
for Learning and Teaching at Higher
Education Institutions
Luisa Seiler, Matthias Kuhnel, Dirk Ifenthaler, and Andrea Honal
1 Introduction
Due to the digital change and the rapid technological development, higher educa-
tion institutions have to cope with upcoming demands. Currently, blended learning
concepts or virtual collaboration via mobile devices had already entered everyday
university practice. With respect to the future, there will be more challenges and in
this context also new opportunities for the academic education. A midterm trend is
seen in a growing interest of making learning processes more measureable and find-
ing innovative learning space (Adams Becker et al., 2017). The usage of mobile
devices and all-time Internet access lead to the fact that our social and professional
lives more and more take place in the virtual world. Due to these circumstances,
large data amounts about individuals and groups are stored and available in quanti-
tative as well as qualitative form. Learning analytics represent a dynamic approach
to evaluate students’ learning almost in real time (Ifenthaler, 2015). By analyzing
and visualizing the individual data, learning deficits can be illustrated, feedback can
be given, and interventions can be implemented. Currently, learning analytics are
already used for making learning more personalized and adaptive. Receiving timely
feedback is a great benefit, because it may lead to better and appropriate design of
L. Seiler · A. Honal
Cooperative State University Mannheim, Mannheim, Germany
M. Kuhnel
D. Ifenthaler (*)
Curtin University, Perth, Australia

https://doi.org/10.1007/978-3-319-64792-0_10
158 L. Seiler et al.
curricula (Ifenthaler, 2017b; Lockyer, Heathcote, & Dawson, 2013). At the same
time, such approaches usually end up in a discussion about privacy and ethical stan-
dards (Ifenthaler & Schumacher, 2016; MacCarthy, 2014). Clear guidelines, mini-
mization of data collection, and transparent procedure can support the implementation
of new learning methods in a positive way (Ifenthaler & Drachsler, 2018). Higher
education institutions regarded as training and developing facility for young aca-
demics also need to consider the labor market situation. With respect to the world of
work, rising trends toward new technologies, virtual collaboration, and intercultural
cooperation need to be considered. Due to this development, more and more
employers require a certain digital skill set when hiring new employees. The so-
called twenty-first-century skills (Griffin & Care, 2015) refer to the idea of develop-
ing viable workers for a modern knowledge society. By summarizing all
aforementioned aspects, this chapter will provide a short historical review of smart
solution in higher education. First, two approaches—mobile learning and learning
analytics—will be considered in detail (Sect. 2). Next, an ongoing research project
of Baden-Wuerttemberg Cooperative State University Mannheim and the University
of Mannheim, which explores the short-term and long-term effects, risks, and ben-
efits of the usage of mobile learning analytics in the students’ daily life, will be
described (Sect. 3). Building on an appropriate implementation of new learning
methods, Sect. 4 summarizes the benefits as well as the challenges for three target
groups—students, lecturers, and higher education institutions. Moreover, two
approaches for a successful implantation of smart solutions into the classroom are
presented (Sect. 5). Last, a conclusion will be drawn, and an outlook will be given
(Sect. 6).
2 Smart Solutions in Higher Education
Digital media and learning found their ways into the higher education systems.
Concepts like serious games, learning analytics, and gamification do no longer
sound as unfamiliar as only a few years ago. Nevertheless, differences of actual
usage of digital media and smart solutions can be identified in various countries. But
what does smart actually mean? When using the concept smart to describe a person,
we mean that someone is intelligent, can deal with difficult situation very quickly,
and is flexible in his or her way of thinking (Cambridge Dictionary, , n.d.). On the
opposite, a not so smart individual will find it difficult to react intuitively to upcom-
ing situations and will not reflect about his or her behavior. So literally, the meaning
is connected with the idea of innovation, certain flexibility, and a 360-degree per-
spective. The framework of smart solutions for higher education institutions assumes
that a learning environment is innovative as well as offers alternatives to learn, col-
laborate, and motivate (Spector, 2014). Therefore, the following sections provide a
closer look on smart learning environments.
10 Digital Applications as Smart Solutions for Learning and Teaching at Higher… 159
2.1 D
efinition and Characteristics of Smart (Learning)
Environments
In this section, the terminology of smart environments will be introduced and char-
acterized. As this paper displays the situation for higher education institutions, we
use smart environments and smart learning environments interchangeable.
According to the literature, the development of smart learning environments is
rooted in the 1980s. During this time, intelligent tutoring systems (ITSs) and adap-
tive learning models were developed (Graf, Kinshuk, & Ives, 2010). These early
attempts of a combination of virtual and pedagogical approaches are by far not
comparable to the current state of various innovative learning and teaching meth-
ods. Particularly, wireless communication and the invention of mobile devices made
a significant contribute to the present state (Sharples, 2013). Similar to the change
of media and technology, humans have aligned to the digital shift. The new genera-
tion of digital natives has grown or rather grows up by using mobile devices natu-
rally (Parment, 2013). Moreover, they place great demands on smart solutions as
well as on the educational system. Hwang (2014) sees the students’ preferences
especially in personalized and adaptive learning. With respect to practical applica-
tions, this means that learning should not only be independent from location and
time but also adaptable to an individual’s personal prerequisites like individual dis-
positions, learning conditions, and personal life. Shaped by the digital change, the
increased requirements, and the omnipresence of new media in daily lives, “context-
aware ubiquitous learning environment (u-learning)” solutions have evolved. These
approaches are able to detect real-world factors (e.g., learner’s status quo, cultural
influence) and offering support (e.g., learning material, feedback) (Hwang, 2014;
Sampson & Zervas, 2013). Furthermore, the learners in a u-learning scenario are
guided under consideration of their individual context. Digital media and new ways
of communication have no special role when talking about adaptive learning but
provide a base for the formation of smart learning environments (SLE). Koper
(2014) mentioned four general features of smart learning environments:
• A physical learning environment, which is enhanced by one or more digital
devices.
• The digital devices that have to be aware of the learners’ status quo (context,
culture, location).
• The digital devices that provide features like the possibility of assessment, vir-
tual collaboration, feedback, or feed forward to the learners.
• Monitoring of the learner’s process and presentation of relevant information to
different stakeholders.
Various benefits for the learner emerge from the creation of SLE. Likewise, to a
“context-aware ubiquitous learning environment,” different real-world and virtual
factors are considered. Furthermore, both approaches are able to provide personal-
ized feedback and interaction through multiple channels and push informal as well
as formal learning (Hwang, 2014). SLEs enable adaption of content, tasks, as well
as interfaces to the learners. Hence, smart learning causes a redefinition of the

classroom through the combination of educational prerequisites (e.g., pedagogic
approaches) and innovative methods (e.g., virtual collaboration, learning analytics,
cloud learning, etc.) (Aguilar, Valdiviezo, Cordero, & Sánchez, 2015). Overall,
SLE offers a learner-centered approach which collects and displays various infor-
mation and leads to a better as well as faster learning by using digital devices (Graf
et al., 2010).
2.2 Mobile Learning and Learning Analytics
This section will extend the idea of learner-centered approaches. According to the
literature, self-regulated and adaptive learnings are well-received concepts when it
comes to academic success (Postareff, Mattsson, Lindblom-Ylänne, & Hailikari,
2016; Vrieling, Stijnen, & Bastiaens, 2017). Mobile learning and learning analytics
fit to the basic concepts of personalized learning which are currently being imple-
mented in the context of higher education. In the following sections, both terms will
be defined, and typical attributes will be presented.
2.2.1 Mobile Learning Definition and Features
Mobile learning or m-learning is related to learning and teaching. Mobile learning

is linked to an appropriate usage of digital media and the possibility of learning
independently from time and location (Traxler, 2009). Due to its focus on flexibility
(i.e., time and location), the concept is clearly differentiated from e-learning
(Gourova, Asenova, & Dulev, 2015). While the borders between private and profes-
sional life get further blurred, learning develops into a ubiquitous process which
inter alia takes place independent from time, location, persons, institutions, and
cultures. Mobile devices provide ideal conditions for such altered life conditions. In
the context of higher education, they meet the requirements of the new generation
of students. According to a meta-analysis investigating the trends of mobile learn-
ing between 2010 and 2015, the results figured out that smartphones are the most
frequent used devices for m-learning. In addition, informal learning is the most
preferred procedure when it comes to m-learning. The omnipresence of smart-
phones provides unlimited access to knowledge, and learners can access informa-
tion in different ways (e.g., informal, formal). With respect to the development of
smart classrooms, digital devices are fundamental requirements. They function as
virtual collaboration/communication tools and provide real-world scenarios for stu-
dents (McQuiggan, Kosturko, McQuiggan, & Sabourin, 2015). Furthermore,
m-learning can help to train a certain digital literacy, which is highly required by
our modern knowledge society. Digital literacy implies the correct handling of
available data as well as an appropriate usage of new technologies. By summarizing
the characteristics of m-learning, it can be concluded that the approach leads to a
more personalized and adaptive learning (Sampson & Zervas, 2013). While
adaptive learning refers to its consideration of learner’s individual prerequisites like

context, social status, and needs, personalization means the possibilities of a per-
sonalized usage of mobile devices and involved applications (different languages
and selection of preferred features).
2.2.2 Learning Analytics Definition and Features
As already mentioned in Sect. 2.2.1, mobile technologies have enhanced and

improved the possibilities of learning and teaching significantly. Particularly, their
contribution to a more personalized and adaptive learning is indispensable for
today’s educational system and for integrated smart solutions. Furthermore, it seems
that personalization is directly linked to a better learning experience, because learn-
ers are able to choose their favorite methods according to their individual conditions
(Ifenthaler, 2015). In this context it is important to mention that personalization is
not only a one-sided process supervised by the individuals. It is also a procedure,
which is built on the interaction between learners and technology. Particular inter-
esting here is the amount of data emerging from the person-computer interaction.
Educational data mining or academic analytics presents good options for analyzing
virtual-produced data (Berland, Baker, & Blikstein, 2014). However, these models
are only able to consider and evaluate information from the past. Learning analyt-
ics, on the other hand, use real-time data about learners like their individual behav-
ior or context (Ifenthaler, 2015). While learning analytics are already used
successfully in English-speaking countries (the USA, UK, Australia), only a limited
number of researchers have implemented the approach in Germany yet (Ifenthaler
& Drachsler, 2018). Besides the possibilities of real-time evaluations, learning ana-
lytics are utilized to visualize information about individuals (Macfadyen & Dawson,
2012; Schumacher & Ifenthaler, 2018). The benefits for involved stakeholders like
students or teachers stem from the simplicity of these analytical tools. By consider-
ing special algorithms, the tools can help teachers to identify “at-risk” students and
to select suitable interventions (e.g., feedback, reasons for poor performance) (Elias,
2011). Other benefits resulting from learning analytics are:
• Recommendations for learners (learning material, learning behavior) (Schumacher
& Ifenthaler, 2018; Verbert, Manouselis, Drachsler, & Duval, 2012).
• Individual reflection of learning behavior (learn tracking) (Verbert et al., 2012).
• Diversity improvement and social collaboration (Gosper & Ifenthaler, 2014).
• Evaluation of influencing factors like emotions, cognition, etc. (Tempelaar,
2017).
• Pedagogical interventions (Ebner, Taraghi, Saranti, & Schön, 2015).
• Adaption of curricula for the target group (Ifenthaler, 2017b; Lockyer et al.,
2013).
• Formative as well as summative evaluation of learning process (Macfadyen,
Dawson, Pardo, & Gašević, 2014).
• Self-regulated and self-controlled learning (Pimmer, Mateescu, & Gröhbiel,
2016; Schumacher & Ifenthaler, 2018).
Nevertheless, the implementation of learning analytics in higher education insti-

tutions demand also trained staff and a well-equipped infrastructure. A current
research showed that the majority educational institutions are only prepared poorly
at this moment (Ifenthaler, 2017a).
Learning analytics include tools which can improve the universities’ routine by
the evaluation of learner’s data. Therefore, the approach is strongly linked to the
development of appropriate learning design (Ifenthaler, 2017b; Lockyer et al.,
2013). The following five steps can be considered for a practical implementation of
learning analytics into the curricula (Elias, 2011) (Figs. 10.1 and 10.2):
Fig. 10.1 Five steps of implementing learning analytics into the curricula
Fig. 10.2 Screenshot of the MyLA app in German

2.3 Interim Conclusion (and Research Gap)
The literature review showed that smart solutions are highly beneficial for different
stakeholders (lecturers, students) and that they play an important role according to
a digital shift. Particularly, the development of wireless communication and innova-
tive collaboration has contributed to on-demand learning. Although mobile devices
and innovative learning environments (virtual, hybrid, blended learning) provide
great potentials, they are rarely implemented to provide full potential for all involved
stakeholders. In addition, a well-developed infrastructure, trained staff, and integra-
tion into the universities’ strategy are indispensable. One step toward the right use
of digitalization in higher education context will be the promotion of a certain digi-
tal awareness and a creation of transparency.
3 Project Mobile Learning Analytics
The project mobile learning analytics is a cooperation project of the Cooperative

State University Mannheim and the University of Mannheim. It is part of the initia-
tive Digital Innovations for Smart Teaching—Better Learning of the Ministry of
Science, Research and the Arts of the State of Baden-Württemberg, Germany.
Within this project, two applications with the brand name MyLA (= My Learning
Analytics) have been developed—an app for students and an additional dashboard
for lecturers. Both applications will be presented in the following sections.
3.1 MyLA App
The MyLA app was built on web technologies, a so-called web application. The
reason for this is a cross-platform usage on almost every smart device. Furthermore,
the utilization with laptops and other personal computers enlarges the possibilities
of this application. MyLA app consists of three main categories: My Profile, My
Learning, and My Progress.
• My Profile: (1) Profile Data: Only anonymous data will be collected, like univer-
sity, course, or lecture. For using the app, a user has to register with an individual
username with a minimum of six characters. (2) Trophies: App users can collect
several trophies for different actions within the application, for example, for
making the first pinboard entry.
• My Learning: (1) Pinboard: This part facilitates the communication between stu-
dents and lecturers. Via the pinboard the app users can express individual needs
or needs occurring on course level. The short messages will be sent to the lec-
turer’s dashboard, where he/she can react on. A special feature is the tagging.
Before sending a pinboard entry, the user can categorize the message into a tag
(e.g., question, exam). Thus, the lecturer can filter all messages by using these
tags. Furthermore, the students receive messages containing text, link, and file
from their lecturers. Hence, the instructors may share exercises with their stu-
dents. (2) Survey: The second part can support the feedback process during dif-
ferent periods (e.g., over a semester at university or a semester abroad). The
student receives the published surveys from their respective lecturer. Now he or
she can anonymously participate to give important feedback to the lecturer.
Based on this exchange, the lecturer can also analyze how the course performs
and if there is room for improvement.
• My Progress: (1) MyTracker: This evaluation feature enables an individual track-
ing of personal data, e.g., learning motivation and learning effort. App users can
evaluate several variables at different time stamps. Based on the feature, students
should be engaged to self-regulate and self-control their personal learning pro-
cess. (2) MyTracker Stats: This section visualizes the MyTracker data using dif-
ferent colored charts. The user can track and analyze their individual development,
where every variable has an own chart.
• Further parts of the app are imprint, privacy, use policies, frequently asked ques-
tions and app history (versions), contact, settings, app statistics (like date of reg-
istration and collected trophies), and several navigation elements (e.g., side menu
and home button).
The aim of MyLA is to improve the communication between students and lectur-
ers. Not only during the shared time at university but also during their physical
absence from the institution. Moreover, the feedback process should be enhanced.
The lecturers can publish surveys and gather feedback on different times (e.g.,
before a semester, while a semester, and after a semester) for starting interventions,
if necessary. By implementing the digital instruments into learning and teaching
processes, the situation for the involved actors can be improved. Additionally, the
tracking function targets progressive feedback to students concerning their individ-
ual learning process. Special concerns, which occur with the usage of individual
data, are privacy issues. Thus, the anonymity and pseudonymity are important fea-
tures of MyLA. The only data that have to be saved are university, course, lecture,
semester, and a self-given username. This should prevent the student’s overcoming
in contacting their lecturer and resultant in using the app.
3.2 MyLA Dashboard
The MyLA dashboard is built on web technologies as well as the app. The default
usage is via personal computer (desktop), but it is also possible to use it on mobile
devices. Generally, tablets are most suitable in comparison to smartphones because
of their screen size and their grid display. MyLA dashboard consists of three grids
on the home page: Pinboard Entries, Survey Centre, and LectureTracker.
Additionally, the lecturer can manage his profile and has almost the identical further
parts (like imprint, etc.) in comparison with the app.
• Short description of Fig. 10.3:

• Left side, Pinboard Entries; Middle, Survey Centre; Right side, LectureTracker.
• My Profile: The profile displays the conjunction between students and their lec-
turer. On the dashboard the lecturer has to select his university and course. Within
the course he/she can register several (at minimum one) lecture/s with the respec-
tive semester. Furthermore, the lecturer has to place an access code for every
registered lecture. The access code and the automatically given lecture number are
important for the students to join a lecture. In MyLA app the students need those
two inputs to successfully register in the lecture and use the app functionalities.
• Pinboard Entries: On this area of the dashboard, the lecturer receives messages
from the students and can also respond to them. Furthermore, the instructor can
mark them just as read. The second option is, e.g., for entries with informative
character. The lecturer can filter the entries by the given filter buttons which are
equivalent to the student’s tagging (see My Learning in Sect. 3.1). Furthermore,
the lecturer can see on the button how many entries of a certain category were
submitted. Additionally, the lecturer can send messages to a certain lecture.
Those lecture messages can contain text, link, and file.
• Survey Centre: This section gives lecturers the possibility to get feedback when-
ever it is wanted or necessary. Lecturers can use default questions or add own
questions with options. After publishing a survey, only the instructor’s selected
lecture sees the survey and can respond to it. Then the lecturer gets the aggre-
gated results of all responds.
• LectureTracker: Students have the possibility to track several variables like
learning motivation or learning success over time (see My Progress in Sect. 3.1).
The aggregation of student values is available on lecture level. The only transmit-
ted data will be the number of participants and the average values of all variables
divided by weeks.
Fig. 10.3 Screenshot of the MyLA dashboard in German

MyLA dashboard’s actual version is a communication and feedback as well as an

analysis tool. With applying these functions, the lecturer can force the exchange
with his students and enhance the feedback process. In further developing of app
and dashboard, additional functions shall be implemented. The added functions
shall be chosen with the feedback from lecturers and students. The aim of both tools
is to improve the situation for both groups—students and instructors.
3.3 First Findings and Outlook
In spring 2017, the app was tested regarding its design, navigation, and further
aspects. One hundred five students from both universities participated in the quanti-
tative study. For the investigation and with respect to meeting general quality criteria
of empirical research, a usability testing instrument was adopted (Pirnay-Dummer,
Ifenthaler, & Spector, 2010). Furthermore, the project team conducted an additional
eye-tracking study to investigate the use of the application. Within the qualitative
testing, the participants had to solve three different tasks (e.g., navigation through
the app) by using the app prototype. The two applications should be tailored to the
potential users (i.e., students and lecturers). Therefore, workshops and information
events had been taken place. The project Mobile Learning Analytics focuses on fur-
ther research regarding mobile learning and learning analytics. With the two appli-
cations—consisting of MyLA dashboard and MyLA app—data will be collected.
Further functionalities shall be identified, students and lecturers evidently need.
Another big challenge of MyLA integration is the conviction especially of lecturers
and also of students. Reasons for that might be the existence of a learning manage-
ment system at the university or the apprehension of an additional time effort.
Challenges have to be processed in a detailed way to reduce existent obstacles.
4 Long-Term Success of Smart Solutions
The future smart solutions in higher education can only work if stakeholders will be
aware of possible benefits and risks. In the following sections, the opportunities as
well as the challenges will be discussed from a perspective of the three main stake-
holders (i.e., students, lecturers, and universities).
4.1 Opportunities for Stakeholders
Smart solutions that integrate mobile devices in learning processes have an important
impact on future learning opportunities and scenarios. The potentials of using mobile
devices in the context of higher education are only rudimentarily exhausted, yet. For
example, a current study with 105 German participants showed that the respondents
use apps for learning on average only 4 days per month. In comparison to this, the
students utilized apps for other reasons almost every day (29 days per month on aver-
age) (Kuhnel, Seiler, Honal, & Ifenthaler, 2017). In this section, the opportunities of
smart solutions will be discussed under consideration of different stakeholders.
For Students Mobile applications are independent from time and location.
Moreover, learning materials and additional options for in-depth studying like vid-
eos are available in portable form via Internet access. Furthermore, the communica-
tion and feedback process can be simplified and improved (Torres, Infante, & Torres,
2015). Additionally, the applications can support collaborative learning like sharing
materials and knowledge or discussing with peers and lecturers. Social media func-
tions like commenting, connecting, and so on are further possibilities. To come
along with this, learners can build a personalized learning experience based on all
inputs and media offers (Gikas & Grant, 2013). Learning analytics (LA), for exam-
ple, use digital traces which are produced while using learning systems. To fulfill the
students’ needs in learning and feedback management, Schumacher and Ifenthaler
(2018) identified several features, which students expect most from LA. For exam-
ple, the majority of the study participants wanted to integrate a self-assessment fea-
ture which is resemble to real exam conditions. Furthermore, they expect valid
feedback and recommendations on learning materials to close their knowledge gaps.
For Lecturers Lecturers can also benefit from the advantages mentioned afore.
Furthermore, instructors ideally detect at-risk students by using learning analytics
technologies and can help their students almost in real time by planning necessary
interventions. LA is also suitable to identify course materials that lead to problems
or do not have any impact to students learning (Ifenthaler, 2017b; Mor, Ferguson, &
Wasson, 2015). Another long-term aspect is digital literacy of the lecturers. This
means the skills concerning the usage of digital technologies. There has been some
positive relationship between digital literacy and the adoption of new technologies.
Additionally, teaching self-efficacy is an important point, which relates to the inte-
gration of technologies in lecturers teaching (Mac Callum, Jeffrey, & Kinshuk,
2014). If those challenges can be coped with success, the lecturers can benefit from
smart solutions regarding their expertise. Using digital technologies can also enlarge
their possibilities in teaching processes.
For Universities Universities as educational institutions benefit from smart solu-
tions. Analyzing and optimizing the impact of internal resources are important
assets to universities. This data should be used to improve processes at institutions
and optimizing the resource input. Further goals are improving learning and teach-
ing processes, lower dropout rate, and higher success rate (Ifenthaler &
Widanapathirana, 2014).
4.2 Challenges for Stakeholders
As already indicated in the introduction, digitalization and advanced technology

have not only advantages for students, lecturers, as well as universities. They will
also occur new challenges for all stakeholders. This section provides some possible
challenges of smart learning environments.
For Students In general terms smart solutions should support the individuals dur-
ing their learning process by analyzing data and providing personalized learning
paths. In this context it has to be mentioned that only data from (recent) past and
from digital sources are used. Hence, they might not be bias-free and lead to wrong
interpretations (Ifenthaler, 2015). The upcoming challenges are dichotomous here:
On the one hand, it has to be focused on the students’ needs and not on what might
be best for the universities images (Ferguson, 2012). On the other hand, transpar-
ency and the selection of data are indispensable aspects for preventing privacy and
ethical issues (Ifenthaler & Schumacher, 2016; MacCarthy, 2014).
For Lecturers The lecturers have a particular role regarding digital enhanced
learning scenarios. They interact between students and institutions. So, apart from
their pedagogical job, they are subordinates under internal regulations (e.g., legisla-
tion and culture). Moreover, it is often criticized that instructors are educated poorly
when it comes to digital media and new technology in the classroom (Gikas &
Grant, 2013). Reasonable factors for this might be a lack of support for further train-
ings and of the encouragement to use innovating approaches by the institutions. The
Horizon Report annually highlights the six trends, challenges, and developments
with respect to digitalization for higher education. In the latest article, the authors
argued that rethinking the role of the instructors and educating a certain digital lit-
eracy to overcome old patterns are serious problems (Adams Becker et al., 2017).
Only if lecturers understand the added value new technology has for students, they
will be able to spend more effort in the didactical realization and their own educa-
tion. Particularly, the correct transfer of technical data into pedagogical interven-
tions is of high importance for the implementation of a holistic approach (Ebner
et al., 2015; Gibson & Ifenthaler, 2017).
For Universities Besides the difficulties to cope with the students, lecturers, and
other involved stakeholders, universities have to face other conditions for imple-
menting smart learning successfully. Primarily, a well-developed digital infrastruc-
ture involving, for example, wireless access to the Internet on the campus or
available technical equipment might be a first obstacle for the institution. Considering
the global situation, it can be derived that not every country has equal preconditions
as the Western-orientated world (West, 2015). Thus, digitalization in the context of
learning and teaching has been discussed under the consideration of social inequal-
ity. For universities this signifies to support, for example, more disadvantaged part-
ner institutions. Working in a globalized world is equivalent to grow together with
also regarding cultural terms. Gosper and Ifenthaler (2014) argued that diversity is
enriching as well as challenging at the same time. In other words, if institutions
want to be attractive for students and thereby competitive on long term, they have to
offer intercultural opportunities for exchange (e.g., semester abroad, virtual collab-
oration, etc.). Hence, the implementation of smart learning is still new in the context
of higher education; a lack of best-practice solutions and profound research is listed.
Furthermore, the majority of initiatives (e.g., projects) to close this research gap
usually end up in silo solutions. Only a strong embedded strategy for implementa-
tion, the determination of interim steps, and an ongoing evaluation are needed for
achieving the objectives.
5 Approaches for Implementation
After discussing the opportunities as well as the challenges for the involved stake-
holders, this section will shortly introduce two approaches for implementation of
digitalization into higher education. Accordingly, a major challenge will be the
avoidance of silo solutions and thus the reinforcement of a long-term success for
smart solutions.
5.1 Technology-Enhanced Learning Complex
Higher education systems have a complex environment where different needs (stu-
dents, lecturers, policy, etc.) are met and where various decisions have to be made.
By ignoring these aspects when realizing innovation into the context of higher edu-
cation, processes often fail. One option for a helpful assistance is seen and discussed
in the technology-enhanced complex (TEL Complex) theory by several researchers
(Laurillard, 2008; Scanlon et al., 2013). The approach covers different influencing
factors, which are presented in the following figure (Fig. 10.4):
The TEL Complex model shows that the vision (e.g., particular example for
including digital media into the learning and teaching process) illustrates the center.
Around the core, the two components of pedagogy as well as technology are placed.
As already mentioned afore, pedagogy is an indispensable parameter for a success-
ful implementation. Apps (e.g., MyLA app) provide good opportunities for a com-
bination of technical and didactical aspects. Moreover, approach complexity results
Fig. 10.4 TEL complex

model (Adapted from
Scanlon et al., 2013)
from the diversity of the involved actors. Thus, Keeley, Pikkel, Quinn, and Walters
(2013) discussed in this content the correct quantity of engagement from students
and lecturers. In addition, transparency and authenticity present dispositive vari-
ables within this context. Besides the human factor, technology constitutes the rec-
ognizable element, which is a prerequisite for the implementation of the shaped
vision and can be seen as supporter for the pedagogical implementation. Furthermore,
technology is also a driver for innovation. Without an ongoing improvement and
development of new technical equipment as well as included applications, no stim-
uli will be given (Laurillard, 2008). The transition in the model is shown by the
component tech context. Additionally, the ecology of practices has to be considered,
too. Scanlon et al. (2013) mentioned in this context that a successful implementa-
tion of technology-enhanced learning formats depends on a variety of factors, which
can differ between cultural, social, and infrastructural aspects. To give an example,
the digital infrastructure in developing countries is usually worse in comparison to
industrialized countries. According to the TLE approach, unequal conditions lead to
another vision and thus to other periodization. Furthermore, there are four commu-
nities, which have to be taken into account when planning the realization of an
innovation. In the literature, communities are frequently related to suppliers and
customers (Keeley et al., 2013). In the context of higher education, the communities
consist of, e.g., learner, lecturers, researchers, and further involved persons.
5.2 The Rapid Outcome Mapping Approach
The rapid outcome mapping approach (ROMA) is a holistic approach originally

developed for the implementation of policy changes (ROMA, online). Likewise, to
afore-presented TEL Model, ROMA considers change not as a linear process, but as
a complex system. By focusing on three main topics (diagnose a problem, develop
a strategy, develop a monitoring and learning plan), the approach represents a man-
ual, which is flexible and dynamic in its usage. Furthermore, the three main areas
are divided into different steps, which help to gain a better understanding of each.
The following figure will give an overview of the approach:
As presented in Fig. 10.5, the first step is to clearly define or redefine the over-
reaching policy goals by mapping the context. The importance of defining precise
problems is supported by the experts’ recommendations for evaluation (ROMA,
online). Besides specifying the strategic purpose, the key stakeholders should be
identified and furthermore set into context with the objectives. By pointing out con-
nections and spheres of influencing persons, assumptions for an appropriate strat-
egy can be made (identify desired behavior changes). The development of a strategy
is a next step. Young and Mendizabal (2009) recommend using a force field analysis
for analyzing supportive as well as obstructive factors. Particular milestones can be
also helpful for the implementation, and they can function as control parameters
(see develop a monitoring and learning system). Another stage of the policy devel-
opment is the inspection of the environment and the analysis of the available
resources (analyze internal capacity to effect change). Only with a suitable infra-
structure, the implementation can be successful on long-term view. Furthermore,
Fig. 10.5 The rapid outcome mapping approach (ROMA) (Adapted from Young & Mendizabal,
2009)
processes should be constantly monitored to contemporary react to changes, make

necessary adjustments, and learn from the interaction (develop a monitoring and
learning system). With respect to ROMA model, the project team had used a similar
procedure (e.g., stakeholder and internal capacity analysis) for the development and
implementation of the two applications.
The represented approaches—TEL complex model and ROMA—indicate two
conceptual models to cope with the implementation of innovation in complex envi-
ronments. Still, there is no standard solution, but the approaches display detailed
and relevant information for higher education institutions. Particularly the develop-
ment of a strategy, which is built under consideration of environmental influences
and internal resources, seems to be a good choice for an appropriate implementation
of smart solutions. Furthermore, the ROMA theory does also control the process
and gives the recommendation for modification if changes occur.
6 Conclusion and Outlook
Smart solutions have great potential of enhancing the teaching and learning pro-
cesses at universities. There are many opportunities in using smart solutions in class
as well as during practical or internship semesters. However, there exist many chal-
lenges that have to be overcome through early interventions. One important aspect
is the improvement of digital literacy among all involved stakeholders. Additionally,

the technical infrastructure has to be strengthened, for example, through stable wire-
less Internet access. Furthermore, rigorous research in the area of mobile learning
and learning analytics is necessary, especially in German-speaking countries. For
instance, German higher education institutions have not reached the level of digita-
lization when compared to other industrialized nations like the USA or England
(Schumacher & Ifenthaler, 2018). Furthermore, universities should implement a
digital agenda in their corporate strategy, particularly to support research in the field
of digital learning and teaching. By doing so, higher education institutions may
actively prevent silo solutions and disorientation. One good example can be seen in
the initiative “Digital Innovations for Smart Teaching—Better Learning,” funded by
the State Ministry of Science, Research and the Arts Baden-Wuerttemberg. Within
this program, the aforementioned research project “mobile learning analytics” and
nine other activities in the field of digital learning and teaching are supported to cre-
ate evidence for innovative higher education learning environments. All projects are
connected for frequent exchange and workshops, providing a platform for collabo-
ration and mutual learning. To sum up, financial and contend-related support (e.g.,
best practices, guidelines, data privacy, etc.) are essential for the implementation of
smart learning environments at higher education institutions. Moreover, higher edu-
cation institutions should be more open for collaborations across boundaries.
References
Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C., & Ananthanarayanan,
V. (2017). NMC Horizon Report: 2017 Higher Education Edition. Austin, Texas: The New
Media Consortium. Retrieved from http://cdn.nmc.org/media/2017-nmc-horizon-report-he-
EN.pdf
Aguilar, J., Valdiviezo, P., Cordero, J., & Sánchez, M. (2015). Conceptual design of smart class-
room based on multiagent systems. In Proceedings of the 17th International Conference on
Artificial Intelligence (ICAI’15), USA.
Berland, M., Baker, R. S., & Blikstein, P. (2014). Educational data mining and learning analytics:
Applications to constructionist research. Technology, Knowledge and Learning, 19, 205–220.
Ebner, M., Taraghi, B., Saranti, A., & Schön, S. (2015). From the field. Seven features of smart
learning analytics – lessons learned from four years of research with learning analytics.
eLearning Papers no. 40, 2. Retrieved from https://www.openeducationeuropa.eu/sites/default/
files/asset/Assessment%20certification%20and%20quality%20assurance%20in%20open%20
learning_From-field_40_3.pdf
Elias, T. (2011). Learning analytics: Definitions, processes and potential. Retrieved from http://
citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.456.7092&rep=rep1&type=pdf
Journal of Technology Enhanced Learning, 4(5/6), 304–317.
Gibson, D. C., & Ifenthaler, D. (2017). Preparing the next generation of education researchers for
big data in higher education. In B. Kei Daniel (Ed.), Big data and learning analytics: Current
theory and practice in higher education (pp. 29–42). New York, NY: Springer.
Gikas, J., & Grant, M. M. (2013). Mobile computing devices in higher education: Student perspec-
tives on learning with cellphones, smartphones & social media. Internet and Higher Education,
19, 18–26.
Gosper, M., & Ifenthaler, D. (2014). Curriculum design for the twenty-first Century. In M. Gosper
& D. Ifenthaler (Eds.), Curriculum models for the 21st century. Using learning technologies in
higher education (pp. 1–15). New York, NY: Springer.
Gourova, E., Asenova, A., & Dulev, P. (2015). Integrated platform for mobile learning. In D. G.
Sampson, P. Isaias, D. Ifenthaler, & J. M. Spector (Eds.), Ubiquitous and mobile learning in
the digital age (pp. 67–92). New York, NY: Springer.
Graf, S., Kinshuk, C., & Ives, A. (2010). A flexible mechanism for providing additivity based on
learning styles in learning management systems. In Proceedings of the IEEE International
Conference on Advanced Learning Technologies (ICALT 2010), Sousse/Tunisia.
Griffin, P., & Care, E. (2015). Assessment and teaching of 21st Century skills: Methods and
approach. Dordrecht, NL: Springer.
Hwang, G. J. (2014). Definition, framework and research issues of smart learning environ-
ments – a context-aware ubiquitous learning perspective. Smart Learning environments – A
Springer Open Journal. Retrieved from https://slejournal.springeropen.com/articles/10.1186/
s40561-014-0004-5
Ifenthaler, D. (2017a). Are higher education institutions prepared for learning analytics?
TechTrends, 61(4), 366–371. https://doi.org/10.1007/s11528-016-0154-0
Ifenthaler, D. (2017b). Learning analytics design. In L. Lin & J. M. Spector (Eds.), The sciences
of learning and instructional design. Constructive articulation between communities (pp. 202–
211). New York, NY: Routledge.
Ifenthaler, D., & Drachsler, H. (2018). Learning analytics. In H. M. Niegemann & A. Weinberger
(Eds.), Lernen mit Bildungstechnologien. Heiedelberg, Germany: Springer.
analytics. Educational Technology Research and Development, 64(5), 923–938. https://doi.
org/10.1007/s11423-016-9477-y
Learning, 19(1–2), 221–240. https://doi.org/10.1007/s10758-014-9226-4
Keeley, L., Pikkel, R., Quinn, B., & Walters, H. (2013). Ten types of innovation: The discipline of
building breakthroughs. Hoboken, NJ: Wiley.
Koper, R. (2014). Conditions for effective smart learning environments. Smart Learning
Environments – A Springer Open Journal. Retrieved from https://slejournal.springeropen.com/
articles/10.1186/s40561-014-0005-4
Kuhnel, M., Seiler, L., Honal, A., & Ifenthaler, D. (2017). Mobile learning analytics in higher
education – Usability testing and evaluation of an app prototype. In Proceedings of the 14th
International Conference on Cognition and Exploratory Learning in the Digital Age (CELDA
2017), Vilamoura/Algarve (Portugal).
Laurillard, D. (2008). Technology enhanced learning as a tool for pedagogical innovation.
Philosophy of Education, 42(3–4), 521–533. https://doi.org/10.1111/j.1467-9752.2008.00658.x
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning
analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.
org/10.1177/0002764213479367
Mac Callum, K., Jeffrey, L., & Kinshuk, N. A. (2014). Factors impacting teachers’ adoption of mobile
learning. Journal of Information Technology Education: Research, 13, 141–162. Retrieved
from http://www.jite.org/documents/Vol13/JITEv13ResearchP141-162MacCallum0455.pdf
MacCarthy, M. (2014). Student privacy. Harm and context. IRIE – International Review of
Information Ethics, 21, 11–24.
Macfadyen, L., & Dawson, S. (2012). Numbers are not enough. Why e-Learning analytics failed
to inform an institutional strategic plan. Educational Technology & Society, 15(3), 149–163.
Macfadyen, L., Dawson, S., Pardo, A., & Gašević, D. (2014). Embracing big data in complex educa-
tional systems: The learning analytics imperative and the policy challenge. Research & Practice
in Assessment, 9, 17–28. Retrieved from https://files.eric.ed.gov/fulltext/EJ1062692.pdf
McQuiggan, S., Kosturko, L., McQuiggan, J., & Sabourin, J. (2015). Changing education with
mobile learning. In S. McQuiggan, L. Kosturko, J. McQuiggan, & J. Sabourin (Eds.), Mobile
learning: A handbook for developers, educators and learners (pp. 1–21). Hoboken: NJ: John
Wiley & Sons.
Mor, Y., Ferguson, R., & Wasson, B. (2015). Editorial: Learning design, teacher inquiry into
student learning and learning analytics: A call for action. British Journal of Educational
Technology, 46(2), 221–229. https://doi.org/10.1111/bjet.12273
Parment, A. (2013). Die Generation Y: Mitarbeiter der Zukunft motivieren, integrieren, führen
(2. Aufl.). Wiesbaden, Germany: Springer Gabler. [The Generation Y: Motivating, integrating,
leading employees of the future].
Pimmer, C., Mateescu, M., & Gröhbiel, U. (2016). Mobile and ubiquitous learning in higher edu-
cation settings. A systematic review of empirical studies. Computers in Human Behavior, 63,
490–501.
Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2010). Highly integrated model assessment
technology and tools. Educational Technology Research and Development, Springer, 58, 3–18.
https://doi.org/10.1007/s11423-009-9119-8
Postareff, L., Mattsson, M., Lindblom-Ylänne, S., & Hailikari, T. (2016). The complex relationship
between emotions, approaches to learning, study success and study progress during the transition
to university. Higher Education, 73(3), 441–457. https://doi.org/10.1007/s10734-016-0096-7
Sampson, D. G., & Zervas, P. (2013). Learning object repositories as knowledge management
Systems. Knowledge Management & E-Learning, 5(2), 116–136.
Scanlon, E., Sharples, M., Fenton-O’Creevy, M., Fleck, J., Cooban, C., Ferguson, R., …
Waterhouse, P. (2013). Beyond prototypes: Enabling innovation in technology-enhanced
learning. Milton Keynes, UK: Open University. Retrieved from http://oro.open.ac.uk/41119/1/
BeyondPrototypes.pdf
Sharples, M. (2013). Mobile learning: research, practice and challenges. Distance Education in
China, 3(5), 5–11. Retrieved from http://oro.open.ac.uk/37510/2/sharples.pdf
Smart. (n.d.). In Cambridge Dictionary. Retrieved from https://dictionary.cambridge.org/
dictionary/english/smart
Spector, M. (2014). Conceptualizing the emerging field of smart learning environments. Smart
Learning Environments – A Springer Open Journal. Retrieved from https://slejournal.spring-
eropen.com/articles/10.1186/s40561-014-0002-7
Tempelaar, D. (2017). How dispositional learning analytics helps understanding the worked-
example principle. In D. G. Sampson, J. M. Spector, D. Ifenthaler, & P. Isaías (Eds.), Proceedings
14th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA
2017) (pp. 117–124). Mannheim, Germany: IADIS Press.
Torres, J. C., Infante, A., & Torres, P. V. (2015). Mobile learning: perspectives. RUSC. Universities
and Knowledge Society Journal, 12(1), 38–49. https://doi.org/10.7238/rusc.v12i1.1944
Traxler, J. (2009). Current state of mobile learning). In M. Ally (Ed.), Mobile learning:
Transforming the delivery of education and training (pp. 9–24). Edmonton, Canada: Athabasca
University Press.
Verbert, K., Manouselis, N., Drachsler, H., & Duval, E. (2012). Dataset-driven research to sup-
port learning and knowledge analytics. Educational Technology & Society, 15(3), 133–148.
Retrieved from http://www.ifets.info/journals/15_3/10.pdf
Vrieling, E., Stijnen, S., & Bastiaens, T. (2017). Successful learning: balancing self-regulation
with instructional planning. Teaching in Higher Education, 23(6), 1–16. https://doi.org/10.10
80/13562517.2017.1414784
West, D. M. (2015). Digital divide: Improving Internet access in the developing world through afford-
able services and diverse content. In Center for Technology Innovation at Brookings. Retrieved
from https://www.brookings.edu/wp-content/uploads/2016/06/West_Internet-Access.pdf
Young, J., & Mendizabal, E. (2009). Helping researchers become policy enterpreneurs: How
to develop engagement strategies for evidence-based policy-making. London: Overseas
Development Institute.
Chapter 11
Learning Analytics to Support Teachers’
Assessment of Problem Solving: A Novel
Application for Machine Learning
and Graph Algorithms
Philippe J. Giabbanelli, Andrew A. Tawfik, and Vishrant K. Gupta
1 Introduction
For many years, education has often been administered using didactic- and lecture-
based methods. In these settings, a teacher often disseminates information, and
learners are tasked with memorization of information. However, many argue this
form of decontextualized education fails to support learning transfer and engender
problem solving. To address this challenge, theorists posit that learning should be
situated within problem-solving contexts (Hmelo-Silver & Barrows, 2006; Jonassen,
1991). Instructional strategies that employ ill-structured problems are often referred
to as inquiry-based learning. In these instructional strategies, learners are often pre-
sented with a problem that is representative of the types of issues that practitioners
face. These problems are often characterized as being ill-structured, that is, the
problems lack defined goals or explicit ways to achieve the predefined goal state
(Herrington, Reeves, & Oliver, 2014). In contrast to well-structured problems that
prescribe a set of correct answers, an ill-structured problem is often assessed on the
viability of the proposed solution given the constraints, perspectives, and standards
embodied in a context (Hung, 2015). The belief is that the ill-structured and
problem-solving approach espoused in inquiry-based learning better generate more
robust knowledge structures (Clariana, 2010; Ifenthaler, Masduki, & Seel, 2011;
Kim & Clariana, 2015).
P. J. Giabbanelli (*)
Computer Science Department, Furman University, Greenville, SC, USA
A. A. Tawfik
Department of Instruction and Curriculum Leadership, University of Memphis,
Memphis, TN, USA
V. K. Gupta
Computer Science Department, Northern Illinois University, DeKalb, IL, USA

https://doi.org/10.1007/978-3-319-64792-0_11
176 P. J. Giabbanelli et al.
This shift in education toward an ill-structured approach has implications for

theory and practice. In contrast with multiple-choice questions that an instructor
may use for assessment of a well-structured problem, ill-structured problems require
learners to identify the relevant elements of the problem space and generate argu-
ments about why a solution is viable and rationale (Ju & Choi, 2017). In addition,
Eseryel, Ifenthaler, and Ge (2013) see an “effective learning process as one that
facilitates transition of problem spaces of learners from the state of preconceptions
or misconceptions to the state of comprehensive, causal explanations” (p. 449). As
such, the multiple-choice approach often employed in the information dissemina-
tion model fails to meet the assessment needs of inquiry-based learning. Therefore,
educators look toward alternative forms of understanding complex problem solving,
such as concept maps (Fig. 11.1) which articulate concepts and their logical ante-
cedents/consequents, because the creation of these artifacts affords learners oppor-
tunities to articulate their understanding of the problem space and the causal
relationships between the concepts (Ifenthaler, 2010). Representing the problem
space using causal maps is a critical cognitive process not only to the success of that
problem solving, but also the refinement of the student’s conceptual knowledge and
problem-solving skills. In doing so, these mapping processes also help students
construct the knowledge acquired into a conceptual framework for that problem
(Jonassen, 2011; Weinerth, Koenig, Brunner, & Martin, 2014).
The shift in student’s representation of knowledge creates challenges for assess-
ment. Whereas well-structured problems are assessed by student’s ability to reiter-
ate the predefined answer, representations of solutions to ill-structured problems
through causal maps necessitate a varied set of cognitive procedures for assessment.
There are two broad categories of procedures. First, a causal map can be assessed
Fig. 11.1 A concept map is a network or graph where relevant concepts are captured as nodes
(depicted as circles) with logical connections known as links to indicate antecedents and conse-
quents (depicted as arrows). The first version of our ITACM software (Giabbanelli & Tawfik,
2018) allowed to compare a student network (left) with an expert network (right)
11 Learning Analytics to Support Teachers’ Assessment of Problem Solving… 177
through a scoring method that favors certain structures. For instance, Kotovsky,
Hayes, and Simon (1985) suggest that one way to understand complexity in prob-
lem solving is through the size of the problem space, measured by the “number of
branches at each node and depth of search to a solution node” (p. 248). A metric can
thus be developed where a student map with longer paths and more branching would
score higher, indicating a more “complex” map. Such scoring methods are known
as referent-free. Structural attributes favored in students’ maps have included “par-
simony, temporal flow, total links, connectedness” (Jeong, 2014, p. 240). Extensions
to causal maps have also been proposed to bring in elements specifically for scor-
ing: weighted concept maps examine the weight of propositions (i.e., links) (Chang,
Sung, Chang, & Lin, 2005), while a “concept map+” distinguishes between types of
links (Passmore, 2004). Second, a student’s map can be assessed by comparison to
a reference expert map. Ifenthaler suggested that providing an expert map and com-
paring it with student maps (e.g., for model-based feedback) can foster a better
understanding of a problem (Ifenthaler, 2011). This potential was confirmed by
experimental studies (Ifenthaler, 2012; Trumpower & Goldsmith, 2004). However,
“hand scoring knowledge maps can be quite time-consuming” and often impractical
for instructors (Trumpower, Filiz, & Sarwar, 2014, p. 229).
Learning analytics may be one way to address the challenge of assessment in
ill-structured problem solving. Indeed, researchers continue to explore how learning
analytics can resolve persistent issues in education, often from the student perspec-
tive. That said, many algorithms have been developed to automatically assess digital
knowledge maps and challenges that teachers face (Trumpower et al., 2014). These
algorithms operate typically at the level of individual nodes and links to find the
ones present in both the expert and the student maps, or present in the expert map
but missing in the student map (Trumpower et al., 2014). In parallel with the grow-
ing interest on comparing maps in education, algorithms have been developed over
several decades at the intersection of pattern recognition and graph theory to address
the related problems of map comparison, network alignment, and graph matching
(Foggia, Percannella, & Vento, 2014; Vento, 2015). However, the uptake of such
approaches in educational research has been limited.
This article explores how learning analytics can be leveraged to address the
assessment challenge that educators face as they implement problem-solving strate-
gies in the classroom. First, we detail how approaches in pattern recognition and
graph theory (e.g., graph kernels, graph editing distance, graph embedding) can be
used to go beyond comparing individual nodes or edges when assessing a student’s
map using a reference expert map. Second, to benefit the practice of problem-
centered instruction and the field of education, we implemented these methods
through a client-server software that supports instructors in comparing maps through
several methods.
The remainder of the manuscript is organized as follows. In the section that fol-
lows, we provide a detailed background on how knowledge structures have been
conceptualized and assessed in the form of causal maps for educational research.
Then, Sect. 3 summarizes and emphasizes the relevance to educational research of
methods for graph comparison from graph theory and machine learning. Sect. 4
presents a new software which implements these methods and provides instructors
with opportunities to collaborate on assessments. Section 5 discusses the implica-
tions and limitations of these methods and implementation.
2 Background
2.1 D
eveloping Knowledge Structures Through Ill-Structured
Problem Solving
Well-structured problems possess all of the necessary information, solution strate-

gies, and criteria for evaluation of the problem. Generally speaking, in these situa-
tions, problem solving often consists of problem representation and then a search
for the correct solution (Ericsson, 2005; Simon & Newell, 1971). These problems
are often employed in modern education because the predetermined nature of the
solution affords efficiency in assessment. Alternatively, inquiry-based learning sug-
gests that learners should be provided opportunities to solve problems that are rep-
resentative of a domain. Specifically, students’ learning is centered around
real-world, ill-structured problems (Lazonder & Harmsen, 2016; Savery, 2006).
These problems provide opportunities to work on complicated or complex cases
that students may later encounter. Complexity is caused in part by a large number of
variables in the problem space, high connectivity among variables, changes over
time (i.e., dynamics), lack of clarity in the goal (i.e., intransparency), or multiplicity
of goals (Clariana, Engelmann, & Yu, 2013). These problems provide students with
a meaningful real-world context to structure their domain knowledge schemata and
effectively retrieve it later when the need arises (Barrows, 1996; Schmidt, Rotgans,
& Yew, 2011).
How learners conceptualize the problem space is a critical issue in inquiry-based
learning. The problem space not only depicts the major concepts (variables) that have
a role in the cause(s) or the solution of the problem, but also provides an underlying
explanation as depecited by the causal relationships among the variables. For exam-
ple, in medical education, problem space should “include[s] all the causal mecha-
nisms that account for the patient’s signs and symptoms” (Hmelo-Silver, 2013,
p. 26), that is, the understanding of the problem is also described by the causal rela-
tionships among the variables that detail the mechanisms for why the problem occurs
and how it can be solved (Eseryel et al., 2013). Furthermore, when students construct
a problem space through causal reasoning, they are practicing scientific problem-
solving process and consolidate their knowledge into a knowledge structure.
Knowledge structures consist of the problem space that explains the mechanism
of how all the variables work together to manifest themselves as the symptom
(“problem”) (Dufresne, Gerace, Hardiman, & Mestre, 1992; Ifenthaler et al.,
2011). Knowledge structures describe the degree to which an individual organizes
information elements from memory. These include an individual’s understanding
of the facts, concepts, and their relationships embedded within the problem space.
Theorists argue that strong knowledge structures facilitate subsequent learning
when new information is presented (Ausubel, 1963; Ifenthaler et al., 2011).
Furthermore, it is posited that retrieval is impacted by the construction of the
knowledge structure, that is, a well-constructed knowledge structure facilitates
efficient pathways when learners need to reference an idea. It is thus hypothesized
that learning can be thought of as the degree to which learners alter their knowl-
edge structures. As it relates to education, theorists contend that the contextualized
nature and emphasis on problem solving in inquiry-based learning strategies facili-
tate the development of robust knowledge structures.
Proponents of inquiry-based learning suggest that the emphasis on ill-structured
problems better bridge the differences that exist between knowledge structures of
experts and novices (Hmelo-Silver, 2013; Jonassen, 2011). Whereas the knowledge
structures of novices are characterized by misconceptions, disconnections, and
surface-level understanding of the problem space, experts include a more complete,
structural-level understanding. Moreover, experts’ knowledge structures are often
defined as a more holistic, schematic organization of information, whereby the con-
cepts in the problem space are organized in a relational and semantic manner
(Jacobson, 2001). In turn, this allows experts to approach problem solving from a
decentralized way that supports causal reasoning (Hmelo-Silver, Marathe, & Liu,
2007). In contrast to experts, studies show that novice knowledge structures often
focus on readily available and most salient concepts while tending to overlook more
foundational concepts that are not as obvious (Ertmer et al., 2008; Hmelo-Silver &
Pfeffer, 2004). In addition, novice explanations are often linear and focused on a
single cause (Grotzer, Kamarainen, Tutwiler, Metcalf, & Dede, 2013; Tawfik, Gill,
Hogan, York, & Keene, 2018a).
2.2 Assessment in Ill-Structured Problem Solving
An important part of problem-solving facilitation is the ability of educators to direct

students toward the most relevant elements of the problem space. Given that net-
works can be depicted through nodes and links (Ifenthaler et al., 2011), concept and
causal maps are becoming increasingly popular forms of assessment (Olney,
Graesser, & Person, 2012). Clariana (2010) suggests there are four unique aspects
as it relates to mapping of the problem space. First, the open-ended nature of ill-
structured problems requires learners to go through the process of recalling and
selecting which concepts (i.e., nodes) to include in a map. Upon completion, stu-
dents must be able to engage in meaning-making about how their concepts are
related. Doing so generates important insights about the structure of causations
(when a directed link goes from a concept into another) and associations (when a
link is undirected or two directed links are reciprocal). Causal maps also elucidate
what learners choose to select from the problem space (extent of knowledge), the
distance of those relationships (proximity), the links between the ideas (lexical
association), and perceived final state of the problem representation (conditional

knowledge) (Clariana et al., 2013).
Although concept maps may serve to understand student’s knowledge construc-
tion, the ill-structured nature of problem solving and design of concept maps creates
a significant challenge for educators. Indeed, research finds that educators cite
assessment as a significant barrier, which precludes their proclivity to implement
inquiry-based learning and ill-structured problem solving in the classroom (Tamim
& Grant, 2013; Wijnen, Loyens, Smeets, Kroeze, & Van der Molen, 2017).
Challenges particularly arise at two steps of the assessment. First, variations in
language (e.g., “cardiac arrest,” “heart attack”) can create an important confound
that make assessment inefficient and impractical. This problem may be prevented
from appearing in the first place when students are limited to using a complete list
of concepts from the problem space, which may include “distracters” or “mislead-
ing” concepts (Ruiz-Primo, 2000). For instance, in participatory modeling studies,
linguistic variability is limited by using a set of terms standardized through indepen-
dent focus groups (Gray, Hilsberg, McFall, & Arlinghaus, 2015). However, impos-
ing such limitations may produce two maps that look more similar than the
individuals’ knowledge structures (Lavin, Giabbanelli, Stefanik, Gray, & Arlinghaus,
2018). The alternative is to place no restriction on the use of concept names and then
to go through an alignment phase in which educators identify equivalent terms
either manually (Giabbanelli & Tawfik, 2018) or through algorithms (Fig. 11.2)
such as recommender systems (Gupta, Giabbanelli, & Tawfik, 2018).
Fig. 11.2 When no restrictions are placed on concept names, linguistic variability can be resolved
by aligning the names used by a student with the names used by the expert. As the alignment pro-
cess can be time-consuming, we recently used interactive visualizations and recommender systems
to support this process (Gupta et al., 2018)
Second, once variations in language have been resolved, educators have to com-
pare maps structurally. Initial assessment approaches have focused on counting ele-
ments that are present in both the student and expert map or only in the expert map.
For instance, such approaches can point out which links are shared and which links
the student may miss (Trumpower et al., 2014). Other recent approaches employed
learning analytics to better address these issues. For instance, HIMATT or AKOVIA
(Ifenthaler, 2014) have treated causal maps as graphs, using structural metrics
derived from graph theory, social network analysis, or network science. For these
earlier approaches, the diameter of the spanning tree was proven to be a retest reli-
able measure (Ifenthaler et al., 2011). More recently, Lavin and colleagues demon-
strated that specific types of centrality indices (i.e., metrics to score the importance
of nodes) could be used to infer that individuals or groups would make similar
predictions (Lavin et al., 2018). Other approaches using structural matching,
semantic matching, overlap measure, and propositional matching have been detailed
in the learning analytics literature (Krabbe, 2014). In sum, assessment approaches
mostly employ tools from graph theory designed to measure structures in one map
(e.g., diameter of the spanning tree), and then two maps were compared with respect
to their individual measures. This is an indirect approach to comparison, using tools
that were not specifically designed for this purpose. In contrast, graph theory (and
particularly as it relates to machine learning) possesses many tools to specifically
compare maps. Therefore, tools exist that can take in two maps and compute the
distance between these maps. The next section seeks to address the paucity of graph
comparison methods in assessment by providing a brief overview of these tools and
their possible uses in learning analytics.
3 Using Graph Comparison Methods for Assessment
3.1 An Intuitive Introduction to Graph Comparison
There are three broad approaches to compare a student map with an expert map.
While they are grounded in graph theory and machine learning, this section pro-
vides an intuitive overview of these approaches. Details and recommendations to
formal specifications as they relate to learning analytics are provided in the follow-
ing sections. Examples from assessment are reinforced with the metaphor of com-
paring houses.
First, we can measure the number of changes necessary to transform one struc-
ture into another. Graph edit distance (GED) accomplishes it by finding an efficient
sequence of transformation. In the case of houses, we can compare their plans side
by side and see what changes are necessary to turn one house into the other, such as
adding a bedroom, removing a bathroom, or relabeling a bedroom as a home office.
In the case of maps, we can add or remove nodes and edges or relabel a node’s
name. The output of the GED is a single number (i.e., the distance), which can be
used for summative assessment. The process to compute this number is also of
interest for formative assessment, as the student can see each difference with the
target map and work along a suggested sequence of operations to bridge this gap.
Similarly to the concept of solution path length (Hays & Simon, 1974), computing
the GED produces a “solution path” (i.e., sequence of operations) to transform a
student map into the expert map, and the final number characterizes this path by
taking into account that some operations may be more significant than others.
Second, rather than doing a possibly long sequence of minute changes, we can
ask whether “the big picture” is similar in two structures. The idea of graph kernels
is to focus on the core of a structure. For instance, the core of a house could be the
size of its rooms. We would thus compute the distribution of room sizes in both
houses and compute the distance between these discrete distributions. Similarly to
GED, the output of a kernel is a single number, but it measures the discrepancy
between distributions of user-defined features. For example, taking the approach of
Kotovsky et al. (1985), we could measure the distribution of the number of branches
per node, as a proxy to a map’s “complexity.”
Third, we could apply a set of metrics to the structures and compare them based
on the results for each individual metric. In the case of houses, one may measure the
surface area and count the number of large open rooms that flow (as a means to
characterize an open floor plan). Each house can then be plotted into a 2D space
where the number of large open rooms is on the x-axis and surface area on the
y-axis. The problem of comparing two houses thus becomes a matter of measuring
the distance between two points in space. Consequently, the final result of a graph
embedding is a single number comparing points in space. For a map, we can mea-
sure N features and plot the map in a space of N dimensions.
Within each approach, we have to state precisely what we value. For instance, in
graph edit distance, is it worst when a student has a link that the expert does not have
or misses a link that the expert has? In graph kernels, which structures are indicative
of learning? For graph embedding, which features should we extract from a map?
Finally, we need to select an algorithm that efficiently accomplishes the computa-
tions. There are dozens of algorithms to compute the graph edit distance, numerous
ways to compare two distributions, and several methods to compare two points in
space. The remainder of this section details each approach, including its implica-
tions for the assessment of digital knowledge maps in education and how to choose
values as well as algorithms in this context.
3.2 Graph Edit Distance (GED)
There are two broad approaches to graph matching. On the one hand, we can require
an exact match, when asking questions such as “are the student and expert maps
identical” or “what is the largest part of the expert map that the student got right.”
However, these questions may not be of practical relevance (e.g., the student and
expert maps are very unlikely to be identical), and the answer may support
summative assessment more than formative assessment. In addition, these questions

are, respectively, known as graph isomorphism and subgraph isomorphism, and
solving them in a reasonable amount of time remains an active area of research
(Arvind & Jacobo, 2005; Carletti, Foggia, Saggese, & Vento, 2018). For all these
reasons, solutions such as graph edit distance (GED) have been developed to handle
an inexact match, in which differences between maps are penalized rather than
forbidden.
In graph edit distance, we look for an edit path, which can be defined as a
sequence of operations that transforms the student map (source) into the expert map
(target). Operations can be performed on edges (deletion, insertion) and nodes
(deletion, insertion, relabeling). It is always possible to transform a map into another
one: for instance, we could delete all of the students’ nodes and edges and then add
all of the expert’s nodes and edges. We are thus interested in finding the best path.
When all operations are viewed as equally important by the instructor, then the best
path minimizes the number of operations (Fig. 11.3).
As the method is flexible, instructors can also state that some operations reveal
more of a misunderstanding from the student than others. For example, studies sug-
gest that students struggle to remove extraneous concepts from a problem space
(Hmelo-Silver et al., 2007; Tawfik, Law, Ge, Xing, & Kim, 2018b). In addition,
studies find that students focus on the surface-level characteristics rather than the
less salient concepts that are more central to the problem (Ertmer et al., 2008;
Fig. 11.3 A student map (top left) can be transformed into an expert map (bottom right) through
sequences of operations, some of which are equally short (green, orange) and some go through
unnecessary steps (red)
Jacobson, 2001). From a learning analytics perspective, the best path in these cases
is to minimize the sum of the operations’ costs. The costs of operations such as
adding/removing nodes are usually set to a positive constant, and the same applies
to edges if they do not have a label. The costs for relabeling have often been
approached from a theoretical perspective (Sole-Ribalta, Serratosa, & Sanfeliu,
2012), where labels are seen as vectors of characters and pair-wise differences are
computed between characters (e.g., “bad” and “sad” differ by only one, whereas
“bad” and “not good” differ on all eight characters). In contrast, assessment is more
focused on the semantic of the words. Methods in the learning analytics literature
exist about how to automatically measure the strength of association between terms
(Sen et al., 2014), but they have not yet been widely applied in the context of com-
paring maps, perhaps due to the paucity of use for graph comparison techniques in
educational research. Our recommendation would be to resolve variations in lan-
guages before computing the GED. In other words, a preprocessing step would
align terms as described in Sect. 2.2, and that would avoid penalizing for variations
in language during the GED. All remaining differences in labels would incur the
same constant cost.
Once the instructor has preprocessed maps (to resolve variations in language)
and identified suitable costs, then the GED can be computed. In terms of supporting
the teacher, this is useful for summative assessment (as the GED is a number sum-
marizing how “close” the student got to the expert), and particularly for formative
assessment as computing the GED shows how to transform the student map into the
expert’s. This is related to the concept of action sequence in educational research.
In contrast to general guidelines or “one-size-fits-all” approaches to identifying
general action sequences leading to accurate maps (Jeong, 2014), computing the
GED can create an entirely personalized action sequence for a given student’s map.
Future research may explore whether these personalized action sequences do cluster
across students and, if so, based on which individual characteristics. Clustering and
sequential pattern mining would provide powerful methods toward this objective
(Perera, Kay, Koprinska, Yacef, & Zaiane, 2008).
3.3 Graph Kernels
A graph kernel considers a graph as being made of an unordered collection of sim-

pler patterns. The matching problem thus consists of extracting and comparing
these patterns. The patterns are user defined; thus, they vary depending on the
emphasis and context of each study. These techniques are particularly used to com-
pare biological networks, and they are summarized in this context by Mueller,
Dehmer, and Emmert-Streib (2013). Patterns have included “graphlets” (i.e., all
subgraphs with 3, 4, or 5 nodes) (Shervashidze, Vishwanathan, Petri, Melhorn, &
Borgwardt, 2009), which are similar to the concept of motifs and triad significance
profile used in social networks and regulatory networks for biology (Juszczyszyn,
Kazienko, & Musiał, 2008; Sanz, Navarro, Arbues, Marijuan, & Moreno, 2011),
trees when comparing the structure of molecules (Mahe & Vert, 2009), and cycles,
which have also been used for molecules (Horváth, 2005).
When it relates to teachers assessment of causal maps, patterns can be used to
examine the level of knowledge construction exhibited by a student. In classifica-
tions of systems, independent nodes are at the lowest level of systems thinking, while
edges can be slightly higher (Malhi et al., 2009; Meadows, 2008). When going even
higher, we start looking at (feedback) loops, also known as cycles. A set of nodes are
in a loop if, starting from any node in the set, we can follow a sequence of edges that
ends at this node. Loops capture a student’s understanding that “a change can be
initiated everywhere in an event circle and after a certain time be read off as either
cause or effect elsewhere in a system” (Skyttner, 2006, from p. 34 specifically).
Studies have shown that loops were absent from many causal maps about a variety
of problems even when they drive the dynamics of these problems in the real world
(Axelrod, 1974). It is thus important when comparing the maps made by individuals
to look at their cycles (Fig. 11.4). That is, a relevant kernel for assessment would be
the distribution of cycles, which counts the number of cycles (y-axis) of each length
(x-axis). In sum, comparing two maps would compare their distributions of cycles.
There are several important differences with using the GED described in the
previous section. The GED takes a very detailed view of all the operations needed
for the transformation: it measures what is different and tells the student how to fix
it. This serves as more knowledgeable peer and source of scaffolding as students are
alerted to gaps in understanding. When students become aware of their knowledge
deficiencies, they are encouraged to reflect and later iterate their problem solving
(Ge & Land, 2003; Hong & Choi, 2011; Jonassen, 1997; Tawfik, Rong, & Choi,
2015). The kernels would not give us an action sequence, but they would tell us the
bigger picture of how a student “thinks” compared to the expert. For instance, the
Fig. 11.4 Comparison of two maps in the software ActionableSystems on the basis of their loops
(Giabbanelli & Baniukiewicz, 2018)
GED may tell the student to add four links because the expert has them. However,
three of those links may be used to close a loop, and it is thus particularly important
that they are present, whereas the fourth one only connects to a peripheral concept.
An action sequence could still be derived from kernels, for instance, by listing all
the loops that the students missed and highlighting which specific edges have to be
added for these loops. Similarly, we can list loops that the students claimed, but that
expert does not endorse. An experimental study may compare whether this action
sequence or the one generated by GED yields either more accurate maps or a better
understanding of the problem.
3.4 Graph Embeddings
Several measures from graph theory have been mentioned in relation to the assess-
ment of causal maps (Krabbe, 2014), including the diameter of the spanning tree,
the number of components (i.e., disconnected parts of a map), or the density (i.e.,
ratio of edges present to the total number of edges that could connect the nodes).
Each one of these measures produces one number. Comparing two maps by look-
ing at each individual measure can be a challenge. For instance, are two maps simi-
lar if both have a single component but one is denser than the other? Would they be
more similar if one was less dense but had a longer spanning tree? A composite
score can be created to provide a single number based on a set of metrics. While
this can be achieved by approaches such as taking the weighted sum of the under-
lying metrics, it raises the question of how to set appropriate weights to each met-
ric and whether we should account for interactions between metrics to avoid
double or triple counting. In contrast, learning analytics that espouse graph embed-
dings provide a mathematical framework. The idea is to “embed” an object (i.e., a
map) into a vector space, where the distance between the embedded objects serves
as proxy for the actual distance between the objects themselves (Hjaltason &
Samet, 2003).
In the case of embedding causal maps for assessment, the instructor decides on
N metrics to use. Each map is then transformed into a vector with N coordinates,
whose values are the graph’s scores on each selected metric. For instance, Fig. 11.5
shows how three maps can be positioned in a three-dimensional space based on their
number of nodes, number of edges, and the average number of edges per node (i.e.,
average degree). Using only two or three metrics allows instructors to conveniently
see each map as a point in space. When using four or more metrics, results can still
be visualized, but the multidimensionality requires the use of techniques such as
parallel coordinates (Inselberg & Dimsdale, 1990), which may be less intuitive.
Once the maps have been transformed into vectors based on selected metrics, the
vectors can be compared. This can inform instructors on whether a student is head-
ing in the same direction as the expert. For instance, a student may have several
nodes, many edges, and a few cycles. If the expert has these elements in similar
proportions but in a larger map, then we suppose that the student’s structure is going
Fig. 11.5 Three maps are embedded as points in a three-dimensional space by measuring three
attributes: the number of concepts/nodes, the number of connections/edges, and the average num-
ber of edges per node (i.e., average degree)
in the right way and will grow with additional problem solving iterations. Conversely,
if the student has few edges and no cycles, then an intervention may be needed to
set the student in the path to success.
4 Implementation
4.1 Influence of Previous Implementations
Our implementation grew out of the Incremental Thesaurus for Assessing Causal
Maps (ITACM) software. The software was first released to assist with reducing
linguistic variability when assessing maps (Giabbanelli & Tawfik, 2018), and the
comparison then consisted of counting the percentage of factors whose names were
equivalent in two maps (Fig. 11.1). The software has three broad limitations as it
relates to supporting teachers during assessment of ill-structured problems. First, it
did not allow for the comprehensive forms of comparison summarized in Sect. 3.
Second, it was a desktop application, forcing users to either work independently or
email files to other specific users. This stands in contrast with previous recommen-
dations to “use software with a web interface or client-server architecture that
allows to retrieve a concept map from different work places through the Internet”
(Krabbe, 2014, p. 279 specifically). Third, the reduction of linguistic variability was
a labor-intensive process, where the computer was only able to recognize whether
two terms were set as equivalent by the instructor previously.
The second release, ITACMv2 (Gupta et al., 2018), addressed the last two limita-
tions. Taking a client/server architecture allows the software to support a commu-
nity of practice. Given that educators struggle with the initial preparation of
inquiry-based learning, allowing users to share maps with their peers allows to save
time in terms of onboarding (Nariman & Chrispeels, 2015; Tamim & Grant, 2013).
Rather than starting with a “blank slate” to redesign the curriculum, sharing
resources using an open education research (OER) format allows educators to lever-
age expertise within their own learning communities, which allows dissemination
of best practices within peer networks. The use of recommender systems also
allowed instructors to identify potential equivalences between terms used by the
student and the expert (Fig. 11.2), which results in a faster alignment process.
Our newest release, ITACMv3, addresses the last limitation by giving access to
all three approaches to comparisons presented in this chapter.
4.2 Methods Selected for Each Form of Graph Comparison
Our implementation for the graph edit distance uses the beam search approach
(Riesen, Emmenegger, & Bunke, 2013; Riesen, Fischer, & Bunke, 2014). Given a
student map with N nodes and an expert map with M nodes, we can start by taking
one of the student’s concepts and identify the expert’s concept to which it can be
mapped (or aligned). There are M possibilities. We can continue the process with
the next concept from the student, and there are still M possibilities as two concepts
from the student may refer to the same in the expert. There are thus in the order of
MN edit paths. Finding the right one is similar to a game of chess, in which a very
large tree is created to compute each possible move, the resulting board configura-
tion, and each possible move within each configuration. Tree-search-based methods
such as the A* algorithm thus estimate the cost of each of the possible branches.
Beam search is an improvement introduced by Riesen (2015) that further prunes the
search tree. Five other alternative implementations are presented in Sect. 4.2 of
Riesen (2015).
Our implementation for graph kernels uses cycles. Cycles are listed using a back-
tracking algorithm with labeling presented by Tarjan (Tarjan, 1973) and imple-
mented in other software such as ActionableSystems (Giabbanelli & Baniukiewicz,
2018). There are alternative algorithms to list all cycles (Bax, 1994), but since
causal maps are typically small graphs, we did not optimize the computation time.
The important choice is on how to handle the distributions. That is, the problem of
comparing two maps has been transformed into comparing the discrete probability
distributions of their cycles. A statistical approach to measuring differences between
these distributions is to use an f-divergence, which is a type of function. Specific
functions include the Hellinger distance and the Kullback-Leibler divergence
(known as KL-divergence). Both are bounded metrics: their output ranges from 0
(similar behavior expected) to 1 (distributions behave very differently). The
KL-divergence is not a distance because it is not symmetric: the KL-divergence
between the student map and the expert map may differ from the KL-divergence
between the expert and the student. This lack of symmetry may prove problematic
in the interpretation of results. Consequently, we used the Hellinger distance, which
is symmetric. We compute it with a four-step process: (1) extract all cycles; (2)
compute the distribution of cycles’ length, where the x-axis is the length of a cycle
and the y-axis is the number of cycles with this length; (3) normalize the distribution
into the range [0, 1] so that it can be treated as probability distribution; and finally
(4) apply the Hellinger distance to the two distributions.
Finally, for graph embeddings, we used three metrics: the number of nodes, the
number of edges, and the graph density. The similarity between two maps is com-
puted using the cosine similarity between their corresponding vectors. The maxi-
mum similarity is 1, obtained when two vectors have the same orientation, intuitively
meaning that the two maps are “thinking in the same direction.” Theoretically, dia-
metrically opposed vectors have a cosine similarity of −1, but this situation cannot
exist here since the metrics produce strictly positive numbers. Thus, the minimum
cosine similarity is 0, when vectors are orthogonal.
Our interface is shown in Fig. 11.6. We recommend that instructors first align the
terms to reduce linguistic variability (first tab), optionally inspect the maps visually
(second tab), and then start to compare maps (third tab). Explanations are provided
for each metric to aid with interpretability of results.
Fig. 11.6 The newest version of the ITACM software includes a “Compare maps” tab, in which
educators have access to an implementation of graph edit distance, graph kernel, and graph embed-
ding. Details of the specific implementation can be accessed by expanding the panel “How does it
work?” within a metric
4.3 Verification
Standard processes in software engineering call for the verification of an implemen-

tation to ensure that it is free of bugs. In this section, we present verification test
cases for the dual purpose of demonstrating that our software was built according to
specifications and to exemplify the methods discussed in the previous section. Two
test cases are presented per method for the sake of brevity, although our implemen-
tation was also subjected to additional cases and passed all of them successfully.
For the graph edit distance (GED), we considered that all operations had the
same cost (0.5). In the first case (Fig. 11.7a), the sequence of operations with the
lowest total cost consisted of three operations (two deletions of causal links and one
addition of a causal link) leading to a total of 1.5. In the second case (Fig. 11.7b),
five operations were required (two node additions and three additions of causal
links), leading to a total of 2.5. The software correctly produced 1.5 and 2.5 in these
two cases.
For graph embeddings, we create the distribution of cycle length. Figure 11.7f
shows the distributions produced by our software for the test case represented by
Fig. 11.7c. These distributions are then turned into proportions, that is, normalized
Fig. 11.7 Test cases and detail of steps for the graph edit distance (a, b). Test cases for graph
kernels and graph embeddings (c–e), with a sample result for the graph kernel (f)
in the range [0, 1] as shown in Table 11.1. For instance, in Fig. 11.7d, the student’s
map has one cycle of length two; hence, each cycle accounts for half (0.5) of the
total cycles in the map. Denoting the proportion of cycles of length i by si and ei for
the student and expert, respectively, then the student and expert maps produce the
discrete distributions S = (s1, …, sn) and E = (e1, …, en) where n is the length of the
longest cycle. The Hellinger distance is:
∑( )
n
1 1 2
H ( S ,E ) = S− E = si − ei
2 2 2 i =1

Applied to the test case shown in Fig. 11.7c, this equation results in:
1
( ) (
 0 + 0 + 0 − 0.16 2 + 0.5 − 0.33 2 + 0.5 − 0.33
) ( ) (0 − ) 
2 2
 0.16
2 
1 0.59
≈ 0.16 + 0.017 + 0.017 + 0.16 ≈ ≈ 0.42
2 2
which is the value displayed in Fig. 11.7f. The value for Fig. 11.7d is 0.54, also
confirmed by the software.
Finally, for graph embeddings, we obtain a vector with three elements for each
map: the number of nodes, the number of edges, and the graph density. If we denote
by A = (a1, a2, a3) and B = (b1, b2, b3) the student and expert maps, respectively, then
we compute their cosine similarity as:
A. B a1b1 + a2 b2 + a3 b3
similarity ( A,B ) = =
A B a + a22 + a32 × b12 + b22 + b32
2
1

In the case of Fig. 11.7c, we have A = (5, 6, 0.6) and B = (5, 8, 0.8) hence
5 × 5 + 6 × 8 + 0.6 × 0.8 73.48

similarity ( A,B ) = = ≈ 0.99
5 + 6 + 0.6 × 5 + 8 + 0.8
2 2 2 2 2 2
61.36 × 89.64

which is confirmed by the software. Similarly, the case in Fig. 11.7e results in 0.97.
Table 11.1 Computing and normalizing the distribution of cycle lengths for graph embeddings
Student Expert
Test case Cycle length Total number Normalized Total number Normalized
Figure 11.7c 2 0 0 1 0.16
3 1 0.5 2 0.33
4 1 0.5 2 0.33
5 0 0 1 0.16
Figure 11.7d 2 1 0.5 0 0
3 1 0.5 1 1
5 Discussion
5.1 Context of the Chapter
Ill-structured problems serve as the foundation of inquiry-based learning. These

problems are defined as having multiple problem elements, unclear goals, and
constraints. A learner must thus articulate multiple solution paths and criteria for the
proposed resolution (Jonassen, 1997, 2011; Loyens & Rikers, 2011). In doing so,
educators are able to go beyond the traditional, didactic forms of learning as they
pursue higher-order learning outcomes (Herrington et al., 2014; Hmelo-Silver et al.,
2007). Indeed, various studies cite the benefits of ill-structured problem solving
afforded by inquiry-based learning when properly supported (Kim, Belland, &
Walker, 2017; Lazonder & Harmsen, 2016; Walker & Leary, 2009).
The aforementioned instructional strategies have given rise to alternative forms
of knowledge representation. Hung (2011) contends that students should be evalu-
ated on their ability to “articulate the critical elements of the problem, their process
for solving it, and the solution proposed and defend their proposed solution and
the rationale, rather than whether they match predetermined answers” (p. 547).
One evidence of how learners solve ill-structured problems is through causal maps,
which require learners to articulate relationships consisting of the concepts’ ante-
cedents and consequents (Jonassen, 2011; Weinerth et al., 2014). Causal maps are
especially beneficial because they provide insight into the connections that learners
make between elements of the problem space, along with affording opportunities to
depict multiple solution paths for ill-structured problems (Clariana et al., 2013;
Jonassen, 2011).
Despite the purported benefits of inquiry-based learning, teachers face unique
challenges in implementation (Hung, 2011; Wijnen et al., 2017). One emergent
confound for educators is how to assess ill-structured problems accurately and
efficiently. In terms of the former, ambiguity of assessment is impractical from a time
management perspective. Research suggests that educators cite assessment as a
primary reason for failing to sustain inquiry-based learning in the classroom (Nariman
& Chrispeels, 2015; Tamim & Grant, 2013). To date, there have been a number of
attempts to resolve the assessment challenge in inquiry-based learning using learning
analytics. One approach is “referent-free,” as the causal map of each student is
assessed independently with respect to certain desired structures such as the total
number of links (Jeong, 2014). The other approach assesses a student map using an
expert map as referent. This is the approach studied in this chapter. Experimental
studies have demonstrated that using a referent can foster a better understanding of
the problem (Ifenthaler, 2012; Trumpower & Goldsmith, 2004), but a manual
comparison of maps is too time-consuming and calls for the development of digital
tools (Trumpower et al., 2014). Several such tools have been developed, but they are
often limited in contrasting low-level features (e.g., number of links that a student
has or misses vis-a-vis the expert) or an ad hoc set of properties (e.g., number of
disconnected components within the maps). In contrast, graph theory and machine
learning have created many algorithms specialized in comparing maps (Foggia
et al., 2014; Vento, 2015).
5.2 New Insights
As noted earlier, the field of education is exploring new ways to leverage learning
analytics to support higher-order learning outcomes in students. While much of the
focus has been on the individual learner, there has been less of a focus on how learn-
ing analytics can be used to support the teacher. Our chapter has introduced three
categories of algorithms for map comparison, with a focus on their applicability for
the assessment of causal maps. We showed that graph edit distance (GED) provides
both an estimation of the difference between maps (for summative assessment) and
an action sequence personalized to assist a specific student in bridging the gaps with
the expert map (for formative assessment). We discussed the benefits of graph ker-
nels to assess systems thinking in students by extracting the distribution of feedback
loops in their maps and comparing it with the expert’s. Lastly, we examined the
potential of graph embeddings to create a composite score and reveal whether a
student is thinking in the same way as the expert. This may be most useful to provide
feedback as the student gradually develops the map. We acknowledge that taking
such approaches to assessing causal maps requires a leap forward, given that current
software in educational research implement none of these approaches. We have thus
provided one implementation (ITACMv3) for all three approaches, which educators
can use within their own learning communities.
5.3 Future Work
While this chapter has detailed the potential for applying machine learning and
graph theoretic techniques to the assessment of ill-structured problems, this applica-
tion raises several questions both across techniques (e.g., how do we select an
approach? can the software be used as it is?) and within each technique. Selected
questions within both categories are now examined in turn.
Having exposed three different techniques to address the same assessment prob-
lem, educators may need to know which one to implement to best support study
success. Despite the different theoretical underpinnings for the three techniques,
one does not offer an obviously superior approach than the others. Consequently,
establishing which one is the best would require an experimental evaluation in the
form of a randomized controlled trial (RCT) where students are assigned to three
groups, each receiving feedback with a different technique. As our software is the
only one that implements all three techniques for the purpose of assessment, it
would provide a de facto intervention tool for this RCT. Nevertheless, the tool itself
should be extensively evaluated and changed where needed before being used.
Indeed, usability testing would ensure that users do not experience barriers when
using the tool for the specific assessment tasks that they face (Giabbanelli, Flarsheim,
Vesuvala, & Drasic, 2016). While verification is about finding bugs and can be
conducted solely by programmers, usability is about evaluating and improving the
users’ experience: it thus requires participants. Consequently, we would recommend
usability testing followed by an experimental evaluation of all three approaches.
This will ensure that instructors have access to the tool that will provide a satisfactory
experience and access to the right method to support study success.
We note that a major limitation to our tool is that students must provide causal
graphs, as produced by tools such as Coggle or Actionable Systems (Giabbanelli
& Baniukiewicz, 2018). The system would have a significantly wider reach if it
could work with essays by first transforming them automatically into causal graphs.
Learning analytics in the automatic analysis of text have demonstrated the growing
possibility of extracting the structures of arguments from narratives or dialogues
(Budzynska et al., 2014). As such techniques mature, they could eventually be inte-
grated in our software.
Argumentation is multifaceted in that it requires learners to justify their claims
using evidence. Moreover, argumentation essays may consist of an initial stance,
counterargument, and rebuttals. Another limitation is that causal graphs do not
include such forms of evidence. It may be that the expert and a student have differ-
ent causal graphs because they reason based on a different set of evidence.
Systematically suggesting that the student should embrace a structure “because the
expert has it” may be less convincing and less flexible than examining how these
structural differences arise from a different evidence base and/or a different use of
the evidence. Approaches have been proposed to used evidential arguments (Bex &
Bench-Capon, 2014) as part of argumentation frameworks, which support a more
much complex analysis of the reasoning than the cause-rule effects employed in
causal maps (Sedki, 2018). Transforming cause-rule effects into argumentation
frameworks that also embed evidence is a long-term possibility for the automatic
assessment of ill-structured problems.
In terms of specific methods, we explained that the graph edit distance can gener-
ate personalized action sequences. An open question is whether these personalized
action sequences tend to cluster and for which student characteristics. This may
assist instructors in better analyzing the learning journeys of groups of students.
It could also assist with the identification of more homogeneous groups of students
with whom the instructor could perform specific activities, as shown in the design
of group-level interventions in health (Giabbanelli & Crutzen, 2014). An open tech-
nical challenge is to estimate the semantic relatedness (Sen et al., 2014) of terms
used by students and the expert, such that the action sequence can deliver better
explanations than merely asking students to change a concept’s name.
While our chapter contrasted three techniques, they may also be used synergisti-
cally. For instance, the action sequence produced by GED would justify operations
such as “add/remove a concept/causal link because the experts has/doesn’t have it.”
Graph kernels can provide a higher-level view on why the student needs to perform
operations such as adding causal links (e.g., because they close a loop and give rise
to certain dynamics). Further exploration into these approaches would allow
researchers to explore additional assessment approaches to assist teachers using
learning analytics.
Acknowledgments Vishrant K. Gupta acknowledges funding support from the College of Liberal
Arts and Sciences at Northern Illinois University. The authors thank Kaspar Riesen for sharing his
implementation of beam search.
Author Contributions PJG designed the project and supervised VKG. PJG and AAT wrote and
revised the manuscript. VKG wrote the software.
References
Arvind, V., & Jacobo, T. (2005). Isomorphism testing: Perspectives and open problems. Bulletin of
the European Association for Theoretical Computer Science, 86, 66–84.
Ausubel, D. G. (1963). Cognitive structure and the facilitation of meaningful verbal learning.
Journal of Teacher Education, 14(2), 217–222.
Axelrod, R. (1974). Structure of decision: The cognitive maps of political elites. Princeton, NJ:
Princeton University Press.
Barrows, H. (1996). Problem-based learning in medicine and beyond: A brief overview. New
Directions for Teaching and Learning, 1996(68), 3–12.
Bax, E. T. (1994). Algorithms to count paths and cycles. Information Processing Letters, 52(5),
249–252.
Bex, F., & Bench-Capon, T. J. (2014). Understanding narratives with argumentation. In COMMA
(pp. 11–18).
Budzynska, K., Janier, M., Kang, J., Reed, C., Saint-Dizier, P., Stede, M., & Yaskorska, O. (2014).
Towards argument mining from dialogue. In Computational Models of Argument: Proceedings
of COMMA 2014 (pp. 185–196).
Carletti, V., Foggia, P., Saggese, A., & Vento, M. (2018). Challenging the time complexity of exact
subgraph isomorphism for huge and dense graphs with VF3. IEEE Transactions on Pattern
Analysis and Machine Learning, 40(4), 804–818.
Chang, K.-E., Sung, Y.-T., Chang, R.-B., & Lin, S.-C. (2005). A new assessment for computer-
based concept mapping. Educational Technology & Society, 8(3), 138–148.
Clariana, R. B. (2010). Deriving individual and group knowledge structure from network dia-
grams and from essays. In Computer-based diagnostics and systematic analysis of knowledge
(pp. 117–130). Boston, MA: Springer.
Clariana, R. B., Engelmann, T., & Yu, W. (2013). Using centrality of concept maps as a measure
of problem space states in computer-supported collaborative problem solving. Educational
Technology Research & Development, 61(3), 423–442.
Dufresne, R. J., Gerace, W. J., Hardiman, P. T., & Mestre, J. P. (1992). Constraining novices to
perform expert like problem analyses: Effects on schema acquisition. Journal of the Learning
Sciences, 2(3), 307–331.
Ericsson, K. A. (2005). Recent advances in expertise research: A commentary on the contributions
to the special issue. Applied Cognitive Psychology, 19(2), 233–241.
Ertmer, P. A., Stepich, D. A., York, C. S., Stickman, A., Wu, X. L., Zurek, S., & Goktas, Y. (2008).
How instructional design experts use knowledge and experience to solve ill-structured prob-
lems. Performance Improvement Quarterly, 21(1), 17–42.
Eseryel, D., Ifenthaler, D., & Ge, X. (2013). Validation study of a method for assessing com-
plex ill-structured problem solving by using causal representations. Educational Technology
Research & Development, 61(3), 443–463.
Foggia, P., Percannella, G., & Vento, M. (2014). Graph matching and learning in pattern recogni-
tion in the last 10 years. International Journal of Pattern Recognition and Artificial Intelligence,
28(1), 1450001.
Ge, X., & Land, S. (2003). Scaffolding students’ problem-solving processes in an ill-structured
task using question prompts and peer interactions. Educational Technology Research &
Development, 51(1), 21–38.
Giabbanelli, P. J., & Baniukiewicz, M. (2018). Navigating complex systems for policymaking
using simple software tools. In P. J. Giabbanelli, V. K. Mago, & E. I. Papageorgiou (Eds.),
Advanced data analytics in health (pp. 21–40). Berlin, Germany: Springer.
Giabbanelli, P. J., & Crutzen, R. (2014). Creating groups with similar expected behavioural
response in randomized controlled trials: A fuzzy cognitive map approach. BMC Medical
Research Methodology, 14(1), 130.
Giabbanelli, P. J., Flarsheim, R. A., Vesuvala, C. X., & Drasic, L. (2016). Developing technol-
ogy to support policymakers in taking a systems science approach to obesity and well-being.
Obesity Reviews, 17, 194–195.
Giabbanelli, P. J., & Tawfik, A. A. (2018). Overcoming the PBL assessment challenge: Design and
development of the incremental thesaurus for assessing causal maps (ITACM). Technology,
Knowledge and Learning. https://doi.org/10.1007/s10758-017-9338-8
Gray, S. A., Hilsberg, J., McFall, A., & Arlinghaus, R. (2015). The structure and function of angler
mental models about fish population ecology. Journal of Outdoor Recreation and Tourism, 12,
1–13.
Grotzer, T. A., Kamarainen, A. M., Tutwiler, M. S., Metcalf, S., & Dede, C. (2013). Learning to
reason about ecosystems dynamics over time: The challenges of an event-based causal focus.
Bioscience, 63(4), 288–296.
Gupta, V. K., Giabbanelli, P. J., & Tawfik, A. A. (2018). An online environment to compare
students’ and expert solutions to ill-structured problems. In Proceedings of the 2018 Human
Computer Interactions (HCI) conference. To appear.
Hays, J. R., & Simon, H. A. (1974). Understanding written problem instructions. In L. W. Gregg
(Ed.), Knowledge and cognition (pp. 167–200). Hillsdale, NJ: Erlbaum.
Herrington, J., Reeves, T. C., & Oliver, R. (2014). Authentic learning environments. In J. M.
Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational
communications and technology (4th ed., pp. 453–464). New York, NY: Springer.
Hjaltason, G. R., & Samet, H. (2003). Properties of embedding methods for similarity searching in
metric spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(5), 530–549.
Hmelo-Silver, C. (2013). Creating a learning space in problem-based learning. Interdisciplinary
Journal of Problem-Based Learning, 7(1), 5. https://doi.org/10.7771/1541-5015.1334
Hmelo-Silver, C., & Barrows, H. (2006). Goals and strategies of a problem-based learning facilita-
tor. Interdisciplinary Journal of Problem-Based Learning, 1(1), 4. https://doi.org/10.7771/1541-
5015.1004
Hmelo-Silver, C., Marathe, S., & Liu, L. (2007). Fish swim, rocks sit, and lungs breathe: Expert-
novice understanding of complex systems. Journal of the Learning Sciences, 16(3), 307–331.
Hmelo-Silver, C., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a com-
plex system from the perspective of structures, behaviors, and functions. Cognitive Science,
28(1), 127–138.
Hong, Y.-C., & Choi, I. (2011). Three dimensions of reflective thinking in solving design problems:
A conceptual model. Educational Technology Research & Development, 59(5), 687–710.
Horváth, T. (2005). Cyclic pattern kernels revisited. Lecture Notes in Computer Science, 3518,
791–801.
Hung, W. (2011). Theory to reality: A few issues in implementing problem-based learning.
Educational Technology Research & Development, 59(4), 529–552.
Hung, W. (2015). Problem-based learning: Conception, practice, and future. In Y. Cho, I. S. Caleon,
& M. Kapur (Eds.), Authentic problem solving and learning in the 21st century (pp. 75–92).
Singapore: Springer.
Ifenthaler, D. (2010). Relational, structural, and semantic analysis of graphical representations and
concept maps. Educational Technology Research & Development, 58(1), 81–97.
Ifenthaler, D. (2011). Bridging the gap between expert-novice differences: The model-based feed-
back approach. Journal of Research on Technology in Education, 43(2), 103–117.

problem-solving scenarios. Educational Technology & Society, 15(1), 38–52.
Ifenthaler, D. (2014). AKOVIA: Automated knowledge visualization and assessment. Technology,
Knowledge and Learning, 19(1–2), 241–248.
Ifenthaler, D., Masduki, I., & Seel, N. M. (2011). The mystery of cognitive structure and how we
can detect it: Tracking the development of cognitive structures over time. Instructional Science,
39(1), 41–61.
Inselberg, A., & Dimsdale, B. (1990). Parallel coordinates: A tool for visualizing multi-dimensional
geometry. In Proceedings of the First IEEE Conference on Visualization (pp. 361–378).
Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between
experts and novices. Complexity, 6(3), 41–49.
Jeong, A. (2014). Sequentially analyzing and modeling causal mapping processes that support
causal understanding and systems thinking. In D. Ifenthaler & R. Hanewald (Eds.), Digital
knowledge maps in education: Technology-enhanced support for teachers and learners. Chapter
13 (pp. 239–251). New York, NY: Springer.
Jonassen, D. H. (1991). Objectivism versus constructivism: Do we need a new philosophical para-
digm? Educational Technology Research & Development, 39(3), 5–14.
Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured
problem-solving learning outcomes. Educational Technology Research and Development,
45(1), 65–94.
Jonassen, D. H. (2011). Learning to solve problems: A handbook for designing problem-solving
learning environments (1st ed.). London, England: Routledge.
Ju, H., & Choi, I. (2017). The role of argumentation in hypothetico-deductive reasoning dur-
ing problem-based learning in medical education: A conceptual framework. Interdisciplinary
Journal of Problem-Based Learning, 12(1), 4.
Juszczyszyn, K., Kazienko, P., & Musiał, K. (2008). Local topology of social network based on
motif analysis. Lecture Notes in Computer Science, 5178, 97–105.
Kim, K., & Clariana, R. B. (2015). Knowledge structure measures of reader’s situation mod-
els across languages: Translation engenders richer structure. Technology, Knowledge and
Learning, 20(2), 249–268.
Kim, N. J., Belland, B. R., & Walker, A. E. (2017). Effectiveness of computer-based scaffold-
ing in the context of problem-based learning for STEM education: Bayesian meta-analysis.
Educational Psychology Review, 30(2), 397–429.
Kotovsky, K., Hayes, J. R., & Simon, H. A. (1985). Why are some problems hard? Evidence from
Tower of Hanoi. Cognitive Psychology, 17(2), 248–294.
Krabbe, H. (2014). Digital concept mapping for formative assessment. In D. Ifenthaler &
R. Hanewald (Eds.), Digital knowledge maps in education: Technology-enhanced support for
teachers and learners. Chapter 15 (pp. 275–297). New York, NY: Springer.
Lavin, E. A., Giabbanelli, P. J., Stefanik, A. T., Gray, S. A., & Arlinghaus, R. (2018). Should
we simulate mental models to assess whether they agree? In Proceedings of the 2018 Spring
Simulation Multi-Conference, Annual Simulation Symposium (SpringSim-ANSS).
Lazonder, A., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance.
Review of Educational Research, 87(4), 1–38.
Loyens, S., & Rikers, R. (2011). Instruction based on inquiry. In Handbook of research on learn-
ing and instruction (pp. 361–381). New York, NY: Routledge.
Mahe, P., & Vert, J.-P. (2009). Graph kernels based on tree patterns for molecules. Machine
Learning, 75(1), 3–35.
Malhi, L., Karanfil, O., Merth, T., Acheson, M., Palmer, A., & Finegood, D. T. (2009). Places to
intervene to make complex food systems more healthy, green, fair, and affordable. Journal of
Hunger & Environmental Nutrition, 4(3–4), 466–476.
Meadows, D. H. (2008). Thinking in systems: A primer. Hartford, VT: Chelsea Green Publishing.
Mueller, L. A. J., Dehmer, M., & Emmert-Streib, F. (2013). Comparing biological networks:
A survey on graph classifying techniques. In A. Prokop & B. Csukás (Eds.), Systems biology.
Dordrecht, The Netherlands: Springer.
Nariman, N., & Chrispeels, J. (2015). PBL in the era of reform standards: Challenges and benefits
perceived by teachers in one elementary school. Interdisciplinary Journal of Problem-Based
Learning, 10(1), 5.
Olney, A. M., Graesser, A. C., & Person, N. K. (2012). Question generation from concept maps.
Dialogue & Discourse, 3(2), 75–99.
Passmore, G. J. (2004). Extending the power of the concept map. Alberta Journal of Educational
Research, 50(4), 370–390.
Perera, D., Kay, J., Koprinska, I., Yacef, K., & Zaiane, O. R. (2008). Clustering and sequential
pattern mining of online collaborative learning data. IEEE Transactions on Knowledge and
Data Engineering, 21(6), 759–772.
Riesen, K. (2015). Structural pattern recognition with graph edit distance. In Advances in computer
vision and pattern recognition. Berlin, Germany: Springer.
Riesen, K., Emmenegger, S., & Bunke, H. (2013). A novel software toolkit for graph edit distance
computation. Lecture Notes in Computer Science, 7877, 142–151.
Riesen, K., Fischer, A., & Bunke, F. (2014). Combining bipartite graph matching and beam search
for graph edit distance approximation. Lecture Notes in Computer Science, 8774, 117–128.
Ruiz-Primo, M. (2000). On the use of concept maps as an assessment tool in science: What we
have learned so far. Revista Electronica de Investigacion Educativa, 2(1), 29–52.
Sanz, J., Navarro, J., Arbues, A., Martin, C., Marijuan, P. C., & Moreno, Y. (2011). The transcrip-
tional regulatory network of Mycobacterium tuberculosis. PLoS One, 6(7), e22178.
Savery, J. (2006). Overview of problem-based learning: Definitions and distinctions. Interdisciplinary
Schmidt, H. G., Rotgans, J. I., & Yew, E. (2011). The process of problem-based learning: What
works and why. Medical Education, 45(8), 792–806.
Sedki, K. (2018). Formalizing arguments from cause-effect rules. In International conference on
industrial, engineering and other applications of applied intelligent systems (pp. 279–285).
Basel, Switzerland: Springer.
Sen, S., Li, T. J.-J., Lesicko, M., Weiland, A., Gold, R., Li, Y., … Hecht, B. (2014). WikiBrain:
Democratizing computation on Wikipedia. In Proceedings of the International Symposium on
Open Collaboration (OpenSym).
Shervashidze, A., Vishwanathan, S. V. N., Petri, T., Melhorn, K., Borgwardt, K. (2009). Efficient
graphlet kernels for large graph comparison. In Proceedings of the Twelfth International
Conference on Artificial Intelligence and Statistics, PMLR 5 (pp. 488–495).
Simon, H. A., & Newell, A. (1971). Human problem solving: The state of the theory in 1970.
The American Psychologist, 26(2), 145–159.
Skyttner, L. (2006). General systems theory. River Edge, NJ: World Scientific.
Sole-Ribalta, A., Serratosa, F., & Sanfeliu, A. (2012). On the graph edit distance cost: Properties
and applications. International Journal of Pattern Recognition and Artificial Intelligence,
26(5), 1260004.
Tamim, S., & Grant, M. (2013). Definitions and uses: Case study of teachers implementing
project-based learning. Interdisciplinary Journal of Problem-Based Learning, 7(2), 3. https://
doi.org/10.7771/1541-5015.1323
Tarjan, R. (1973). Enumeration of the elementary circuits of a directed graph. SIAM Journal on
Computing, 2(3), 211–216.
Tawfik, A. A., Gill, A., Hogan, M., York, C. S., & Keene, C. W. (2018a). How novices use expert
case libraries for problem solving. Technology, Knowledge and Learning, 1–18. https://doi.
org/10.1007/s10758-017-9324-1
Tawfik, A. A., Law, V., Ge, X., Xing, W., & Kim, K. (2018b). The effect of sustained vs. faded
scaffolding on students’ argumentation in ill-structured problem solving. Computers in Human
Behavior, 87, 436–449.
Tawfik, A. A., Rong, H., & Choi, I. (2015). Failing to learn: Towards a unified design approach for
failure-based learning. Educational Technology Research and Development, 63(6), 975–994.
Trumpower, D. L., Filiz, M., & Sarwar, G. S. (2014). Assessment for learning using digital knowl-
edge maps. In D. Ifenthaler & R. Hanewald (Eds.), Digital knowledge maps in education:
Technology-enhanced support for teachers and learners. Chapter 12 (pp. 221–237). New York,

NY: Springer.
Trumpower, D. L., & Goldsmith, T. E. (2004). Structural enhancement of learning. Contemporary
Educational Psychology, 29, 426–446.
Vento, M. (2015). A long trip in the charming world of graphs for pattern recognition. Pattern
Recognition, 48, 291–301.
Walker, A., & Leary, H. (2009). A problem based learning meta analysis: Differences across
problem types, implementation types, disciplines, and assessment levels. Interdisciplinary
Weinerth, K., Koenig, V., Brunner, M., & Martin, R. (2014). Concept maps: A useful and usable
tool for computer-based knowledge assessment? A literature review with a focus on usability.
Computers & Education, 78, 201–209.
Wijnen, M., Loyens, S. M. M., Smeets, G., Kroeze, M. J., & Van der Molen, H. T. (2017). Students’
and teachers’ experiences with the implementation of problem-based learning at a university
law school. Interdisciplinary Journal of Problem-Based Learning, 11(2), 5.
Chapter 12
Utilizing Learning Analytics in Small
Institutions: A Study of Performance
of Adult Learners in Online Classes
Ellina Chernobilsky and Susan Hayes
1 Introduction
The changing landscape of higher education calls for changes in how institutions
use their existing infrastructure, understand and adapt to changing population of
learners, and plan for strategic improvements in their long-term plans. These revised
understandings call for the use of available data that various systems on university
campuses inevitably collect. The use of these data is ever-increasing, and we, as a
field, are still learning to harness and understand how to best use these data. Using
such data to make decisions in postsecondary education, known as learning analyt-
ics, is evolving and becoming embedded in university operations (Picciano, 2012).
While large institutions, with state support or large endowments, have resources
and man power to collect, organize, and then use the data, the situation may be very
different in small, private not-for-profit organizations that cannot easily allocate the
resources for such massive enterprise-level engagements. To stay competitive, how-
ever, this becomes a necessity in small and large institutions alike. In this paper, we
show how one small liberal arts private university uses data to understand its current
academic position and to chart the way to the future.
There are many reasons why higher education institutions decide to use learning
analytics (Shacklock, 2016; van Barneveld, Arnold, & Campbell, 2012). The use of
learning analytics in our university is necessary to better understand the roadblocks
to student success (as defined by grades) and the issues of retention. This is driven
by two specific objectives to help students achieve success and to increase opera-
tional efficiency of the institution. While we are limited in our resources and the
ability to mine the data, nevertheless, we see learning analytics as an opportunity to
increase our understanding of the students we serve and what institutional changes
E. Chernobilsky (*) · S. Hayes

Caldwell University, Caldwell, NJ, USA
e-mail: [email protected]; [email protected]

https://doi.org/10.1007/978-3-319-64792-0_12
202 E. Chernobilsky and S. Hayes
may be required in order to serve the students more successfully. Specifically, we

are interested in student retention, especially, that of adult learners.
Leitner, Ebner, and Khalil (2017) identify short- and long-term trends for the
higher education field. Among those trends is the adjustment to social and economic
factors which afford the changes in what the students can do. Learning analytics is
an important tool to better understand such factors. Learning analytics also helps to
understand what change is necessary and how to impart these changes, especially
for small universities where resources and infrastructure are at a premium.
Since the size of the datasets that the small institutions are generating is not on
the same scale as the sets generated by large institutions in the same time period, we
believe that working with small-scale data sets and growing them over time might
help small schools to develop such models of student success that would be gener-
alizable specifically to small institutions. Understanding what learning analytics can
and cannot do for such schools becomes then particularly useful.
1.1 Learning Analytics and Data Mining
The use of databases to manage student biographical, financial, and academic data,
as well as using learning management systems to manage the instructional and
course information, while also using security card systems to track access to cam-
pus services resulted in proliferation of data that higher education institutions con-
tinuously collect about students and student behavior when learning (Piety, 2013).
These data, if organized and studied properly, can provide a plethora of information
and can help institutions in making decisions, both academic and strategic in nature
(Romero & Ventura, 2013). In the last decade, in order to use the massive amounts
of data, researchers and practitioners began engaging in the use of various tech-
niques collectively known as learning analytics and data mining.
Romero and Ventura (2013) define educational data mining as an interdisciplin-
ary field concerned with the research, development, and application of computer-
ized methods to study large data sets in education. A related concept to data mining
is learning analytics or a set of tools, technologies, and platforms that help shape
understanding of learning and provide information for subsequent decision-making
in order to improve institutions of learning and, ultimately, to help students succeed
(Wagner & Ice, 2012). As such, postsecondary educators and support staff use
learning analytics to collect and study data, both digital and analog, to understand
what data may tell us about student learning and how this understanding might feed
into learning outcomes (EDUCAUSE Learning Initiative, 2011). In an attempt to
understand the level of focus learning analytics takes in postsecondary education,
van Barneveld et al. (2012) propose a definition they adapt from Bach’s (2010)
work. They define learning analytics as “the use of analytics techniques to help
target instructional, curricular, and support resources to support the achievement of
specific learning goals” (p. 8). For the purposes of this chapter, we use this very
definition of learning analytics.
12 Utilizing Learning Analytics in Small Institutions: A Study of Performance… 203
Long and Siemens (2011) differentiate between learning analytics and academic
analytics in that academic analytics is more general in nature and covers all levels
from national to institutional. Learning analytics, in contrast, is more fine-grain,
focusing on the data generated and analyzed specifically for department and course
level improvements, Long and Siemens propose.
All research in learning analytics can be classified into one of the four types
(Boyer & Bonnin, 2017). One such type focuses on recommendations and notifica-
tions to the users of the educational systems; another deals with educational data
mining. The third type is primarily concerned with data visualization, whereas the
last research cluster is focused on statistical analyses in learning analytics. Mattingly,
Rice, and Berge (2012) rightfully note that analytics is a part of a still developing
field and, as such might change noticeably in the future. Mattingly et al. warn that
ethical issues must be carefully considered when engaging in learning analytics
activities. Decisions on who owns and can access and use student data are to be
made for any (big) data research (Slade & Prinsloo, 2013), especially if students can
be identified by name at any point in the mining process. Slade and Prinsloo classi-
fied the ethical issues into three categories: (1) location and interpretation; (2)
informed consent, privacy, and the de-identification of data; (3) the management,
classification, and storage of data. All of these issues are difficult to address at
smaller institutions, where students can be identified easier and in which the
resources, human and financial, dedicated to data analysis and storage, are tight.
In 2011, IBM’s Analytics for Achievement paper outlined eight categories of
instructional applications where using learning analytics can be helpful (IBM,
2011). Of these, the primary interest to us is the ability to disaggregate student per-
formance by specified characteristics such as major, year of study, GPA, and so on
with the goals of improving retention and learning outcomes in our programs and/
or courses. Picciano (2012) reports that student attrition in US colleges and univer-
sities has been an issue in American higher education for decades. Despite signifi-
cant efforts and attention this issue received on all levels, high attrition rates, ranging
from 22% at 2-year public institutions to almost 61% at private nonprofit organiza-
tions, persist (Picciano, 2012).
Chen (2015) argues that theory has been an important concern in the field of
learning analytics. Dawson, Mirriahi, and Gasevich (2015) make an argument that
learning analytics helps advance theories of learning and contributes to pedagogy
and epistemology. Through these contributions, eventually, researchers who work in
the field will devise their own theory where learning analytics is a main tenet.
Currently, there are various frameworks that learning analytics research can rest
upon. One of them, proposed by Scheffel, Drachsler, Stoyanov, and Specht (2014),
offers quality indicators for learning analytics grouped into five broad clusters:
objectives, learning, measures of learning, data, and organization. The first three
indicators deal with educational and pedagogical concepts not directly connected to
learning analytics. These are important to consider, however, because attending to
these concepts affects learning and outcomes. Data-related cluster includes the
issues related to transparency, ownership of data, and data privacy and security.
Organizational cluster of the framework focuses on the stakeholders, implementation
of learning analytics into the organizational processes, and structure and organiza-
tional change factors.
One of our institutional goals is to explore data with a possibility of designing a
notification process that would move the university from information processes to
optimization processes (Boyer & Bonnin, 2017). This will help academic support
staff detect issues early and to alert the student and the instructor if/when necessary.
To approach this strategically, we, as an institution, use learning analytics for three
specific goals outlined in the Horizon Report (Johnson et al., 2016). We use analyt-
ics to diagnose the problems and then lean on it to propose the solution. In our work,
we are subscribing to Scheffel et al.’s (2014) model, and in this paper, we are spe-
cifically looking to understand what the learning analytics can help us see in the
case where the size of data is limited due to small enrollment numbers. The purpose
of this chapter, therefore, is to report on one university’s efforts to use learning ana-
lytics to understand performance of adult learners in online undergraduate courses.
1.2 Online Learning
The roots of online learning can be traced back to earlier forms of distance learning
when students were not always present physically. Correspondence courses were the
traditional approach. Later, with the development of television and other more
advanced technologies, the delivery formats evolved (Casey, 2008). In the mid-
1990s, online course delivery via the Internet began, developing further as learning
management systems were developed (Casey, 2008). Today, online learning is an
integral part of higher education landscape and has a range of formats from MOOCs
to small close access classes in a variety of platforms. During the 2011–2012 school
year alone, about 7.4 million undergraduate students (32 percent) and about 1.3 mil-
lion graduate students (36 percent) took at least one distance education class (NCES,
2015). Means, Toyama, Murphy, Bakia, and Jones (2010) stipulate that a replace-
ment of face-to-face instruction with the online option may be considered successful
only if such instruction provides online learning without sacrificing the achievement
in the course (p. 3). For the institutions of higher learning, offering online courses
frees up space, both classroom and parking, as well as creates flexibility in academic
scheduling and in providing academic support. In addition, it offers faculty the
options of using andragogy approaches in teaching. This makes online learning
attractive to many students, especially working adults, as learning online offers con-
venience and choice that traditional face-to-face classes do not. This is especially
important for those learners who balance work, school, and family (Kauffman, 2015).
Means et al. (2010) identify three key elements that describe online learning:
replacement of face-to-face instruction or its enhancement; different, online-
specific, pedagogical approaches; and the various types of communication that
online teaching and learning can offer. In the current research, we were concerned
with the replacement application, i.e., fully online courses. In our case, however, the
situation is more complex, as traditionally, our institution has accelerated and
semester-long online offerings.
There has been a long debate concerning the effectiveness of online learning.
Recently, Nguyen (2015) explored the effectiveness of online learning by synthesiz-
ing the work done previously. Nguyen pointed out that while different studies see
different results of the effectiveness of online learning, the majority of such studies
find that online learning is at least as effective as the face-to-face education. One
argument for not seeing the differences is that students essentially self-select to be
in online or face-to-face class and that may contribute to the no difference seen in
the results.
Stack (2015) reports on a quasi-randomized study that explored whether delivery
systems (online vs. traditional) resulted in achievement differences. Stack controlled
for supervised conditions during the exam times and for such important factors as
gender, academic ability, effort, and amount of time used for studies by using the
first hour exam as a control variable. The results of Stack’s study indicate that there
were no differences in the course achievement between those students who took an
online section of the course and those who took traditional face-to-face section.
Kemp and Grieve (2014) caution that to consider online learning as a unitary
concept is simplistic when it is examined on its own or when compared to general
face-to-face learning. Kemp and Grieve state that each modality is complex and has
multiple factors affecting success. Nevertheless, looking into the difference between
the two types of offerings helps us, as a small campus, understand what might work
for our students.
One of the flexibilities that makes online education so attractive is an opportunity
to take courses in a variety of formats. One such option that may be particularly
attractive is an accelerated option. Tatum (2010) defines accelerated learning as a
compressed process of education. This can mean shortening the course of study
without changing the total hours or Carnegie units, reducing the number of hours in
a semester, or doing both. Accelerated learning has been around for many years
(Seamon, 2004). The terms “accelerated” can be used in relation to a course in a
program or study (Donaldson, 2001; Mealman & Lawrence, 2000) or the timing the
course is offered as it can refer to summer sessions (Anastasi, 2007), intersessions
(Geltner & Logan, 2001), weekend courses (Messina, 1996), and, currently, more
and more often, accelerated online courses (Collins, Kang, Biniecki, & Favor, 2015;
Millett, Stickler, & Wang, 2015; Shaw, Chametzky, Burrus, & Walters, 2013).
The outcomes in regard to student achievement in semester-long and accelerated
programs and courses seem to be comparable to each other (Anastasi, 2007; Shaw
et al., 2013; Tatum, 2010) or better in accelerated courses (Sheldon & Durdella,
2010). At the same time, Akyol and Garrison (2008) found that accelerated courses
have greater social presence. That is because the courses are shorter and the learners
must interact more with each other and with the instructor (Anastasi, 2007). Other
benefits of condensed or accelerated offerings are that faculty provide faster feed-
back (Lee & Horsfall, 2010), students feel less isolated from peers (Chametzky,
2018), and experience reduced anxiety (Pino, 2008). The combination of these
benefits along with the quicker progression through the program reduces the likeli-
hood of dropping out from the program (Bowen, Chingos, & McPherson, 2009;
Geltner & Logan, 2001). This particular option may be more attractive for older,
mature learners who want to enter and exit the programs quicker to attain educa-
tional and career goals at their own pace.
1.3 Adult Students
Adult learners, often labeled as nontraditional students, are usually identified as

students who are over 25 years old and who have one or more characteristics such
as having dependents, being single parents, working full or part time, and having
life or work experience outside of college or university (De Vito, 2009; Lambert
et al., 2014). Ross-Gordon (2011) suggests that instead of being nontraditional, this
population is slowly becoming a norm in college and university enrollment reports.
Servicing adult learners requires a shift in teaching practices that includes under-
standing of andragogy (Henschke, 2011; Knowles, 1980; Samaroo, Cooper, &
Green, 2013) as well as shifts in scheduling the courses and the availability of sup-
port services (Rhodes, 2001, as cited in Alhassan (2012)).
De Vito (2009) suggests that institutions have responsibility to enhance adult
learning programs. De Vito also acknowledges that one of the barriers to adult learn-
ing is the issue of accessibility in traditional higher education, which addresses such
issues as times and locations of the course offerings. Since most adult students work
and have families, attending school competes with other life obligations and com-
mitments. One way to address this discrepancy is to offer ways for adult learners to
attend classes at the pace comfortable to them which often means asynchronous,
fast-paced courses online.
Dropout rates for adult students attending higher education institutions, whether
face-to-face or online, remain high. In their study of factors that influence adult
learners’ decisions on whether to drop out of the online learning or to persist, Park
and Choi (2009) concluded that there were two significant predictors to the deci-
sions to persist in online learning. These predictors were the perceptions of univer-
sity support and relevance of the courses to the lives and work of the adult learners.
Park and Choi further elaborate that these predictors are linked to the classes for
which adults register. Park and Choi state that learners are more likely to drop out
of an online course if they perceive that the university does not support them or if
they perceive the course as not being related to their own life or work.
1.4 Retention Issues
Despite all these benefits, online classes continue to display serious retention issues,
which need to be addressed (Bawa, 2016). Attrition rates have always been higher
in online learning as compared to face-to-face instruction (Carr, 2000).
Bean and Metzner (1985) proposed that four factors influence the attrition of
nontraditional students. These factors are student background, environmental fac-
tors, and academic variables such as study habits and course availability and aca-
demic and psychological factors. Cochran, Campbell, Baker, and Leeds (2014)
found that the strongest predictor of student withdrawals is academic experience.
Like Levy (2007), Cochran and colleagues saw that seniors are less likely to with-
draw from an online course than freshmen and sophomores. Cochran et al. also saw
that prior course withdrawals from online courses and student GPA are the two
other factors that significantly affect the retention in online courses.
More recently, Bawa (2016) pointed out that attrition of students in online pro-
grams and courses is related to misconceptions about online learning (e.g., it is
easier, it requires less effort, does not interfere with the current lifestyle), financial
factors, issues of motivation, and technological constraints. At the same time, Bawa
indicated that instructors too maybe at fault: the instructors who do not understand
the specifics of online students and who lack technical and pedagogical knowledge
will contribute to student dropout rates from the online programs.
It is clear that the type of course (face-to-face or online) students choose for their
studies may affect their success. When considering the differences between tradi-
tional and adult learners in online settings specifically, research shows that adult
students have higher dropout rates (Park & Choi, 2009). To investigate this phenom-
enon more completely, the authors posed the following research question for this
study: How do adult students perform as compared to the traditional students in
various online course formats? As the data were mined, other more specific ques-
tions began to emerge. Specifically,
• Were students (traditional or adult) excelling in the fast-paced accelerated
courses as compared to the semester-long courses?
• Does taking multiple online courses either at the same time or consecutively
result in better online performance as measured by the final grades earned?
• Is retention in online courses as measured by the withdrawal rate different for the
two populations?
• Do traditional and adult students do equally well in different courses: core, either
foundational or enriched, and other courses offered in online environment?
2 Methodology
2.1 Setting
The university where the study took place is a small, liberal arts and professional
studies university serving 2206 undergraduate and graduate students in 2017–2018
academic years. The school offers traditional education with on-campus residence
halls, learning facilities, extracurricular opportunities, and support services.
All students in undergraduate degree programs are required to take courses in the
core curriculum (45–49 credits in total) in addition to the courses in their intended
majors. The foundational core curriculum (36–40 credits) is intended to introduce
students to the breadth of liberal arts disciplines and to develop the fundamental
skills of writing, critical thinking, and information literacy. The enriched core cur-
riculum (9 credits) is intended to deepen students’ understanding of Roman Catholic
and Dominican heritage, ethics, and global and cultural perspectives. All students,
regardless of the mode of delivery of the degree program, must take the required
core curriculum courses in order to graduate.
In the Fall 2017, 1603 undergraduate students attended the university. The total
number of adult undergraduates (23 years of age or older) dropped steadily over the
last decade; at the same time, the number of traditional undergraduates grew. In
2006, 705 adult students accounted for 42% of the entire undergraduate student
body. By 2012, that number declined to 460 (29%). In an attempt to attract more
adult undergraduate learners to the university and to better ensure they were retained,
the school developed online courses in various programs and within its core curricu-
lum that delivered to all undergraduate majors. The university also added new fully
online undergraduate degree programs. The same year, the university added more
online undergraduate courses in an accelerated 7-week format as opposed to the
original 14-week format, again to better meet the needs of adult students.
Despite the addition of the new programs and formats, the share of adult students
continued to decline, and in 2017, the number of adult undergraduates enrolled was
231 (14%). In order to better retain undergraduate students, the faculty requested
that institutional research staff begin to mine online course performance data to bet-
ter understand the trends, detect problems, and, possibly, propose a solution. This
analysis could help faculty and academic support staff devise strategies to improve
academic achievement and persistence.
2.1.1 Online Courses at the University
The university in this case study defines online courses as those that meet exclu-
sively online without any physical face-to-face time on campus. The instructors
have a choice whether to include any synchronous meeting times but are encour-
aged to offer courses in asynchronous format. At the time the data were collected,
all courses were taught using Blackboard Learning Management System. Regardless
of the length of the course (full semester or accelerated), each course is run in a
module format. Faculty have freedom to identify the number of modules, the num-
ber of learning outcomes, and what each module contains. The faculty are encour-
aged to use Quality Matters best practices when designing their courses. The
university offers support to faculty in course design through the Office of Online
Education. While in the course, students have access to online tutoring, technical
assistance, and the library.
2.2 Data Sources
This cross-sectional research study used course and student data across four semes-
ters during the 2015–2016 and 2016–2017 academic years. Only fall and spring
data were considered for the purposes of equal comparisons, as summer semesters
are shorter. Only fully online courses were included, although the university also
offered one course in a hybrid format during the time period.
The privacy of data was preserved since the researchers did not use any personal
identifiers such as names and student ID numbers. Certain assumptions about the
integrity of the data have been made while working with the data set. Specifically,
the data were assumed to be accurate and independent.
The dataset consisted of data points from 70 online undergraduate course sec-
tions conducted during the 2015–2016 and 2016–2017 academic years. Thirty-nine
of those courses were offered in an accelerated 7-week format, and 31 were the
semester-long 14-week courses. The 70 course sections spanned 16 disciplines,
namely, art history, biology, business, computer information systems, communica-
tions, English, health science, history, mathematics, music, philosophy, political
science, psychology, sociology, Spanish, and theology. Fourteen of the course sec-
tions were noncore courses, 39 were foundational core courses, and 17 were
enriched core offerings. Spring 2017 was the semester with the highest number of
undergraduate online courses offered and the largest enrollments. One explanation
for the increase was a change in the criteria that allowed undergraduates to qualify
for enrollment in online courses. Table 12.1 summarizes the course format and
course registration information by each of the four semesters in the dataset. A total
number of registrations for all the courses in the set was 574.
Most of the courses were offered in a single section during a semester, with the
exception of two sections of introductory biology courses in Spring 2016 and 2017
and introductory Spanish courses in Spring 2016 and 2017 semesters. Interestingly,
all four biology sections were offered as accelerated courses, whereas the two
Spanish sections offered in Spring 2016 were accelerated, and two sections offered
in Spring 2017 were offered in the semester-long format. Furthermore, there were
two disciplines that switched from semester-long to accelerated and vice versa
between semesters within the dataset. These were an introductory course in art his-
tory and an enriched, 300-level course in theology.
Of these 574 course registrations in the dataset, 307 registrations were in acceler-
ated courses and 267 were for semester-long courses. In both formats, traditional
students outnumbered adult students (Table 12.2).
Three hundred thirty unique students made up the 574 course registrations in the
dataset. Of those, 103 were adult undergraduates, and 227 were traditional under-
Table 12.1 Summary of dataset

Course format Course registrations
Semester Accelerated Semester-long Total Adult Traditional Total
1516 FA 8 9 17 31 47 78
1516 SP 13 6 19 79 42 121
1617 FA 5 7 12 31 85 116
1617 SP 13 9 22 166 93 259
Totals 39 31 70 307 267 574
Table 12.2 Course registrations by course format and student type

Student type
Course format Adult Traditional Total
Accelerated format 128 179 307
Semester-long format 127 140 267
Total 255 319 574
Table 12.3 Number of courses by student type

Adult Traditional Total
Single course 40 149 189
2–4 courses 49 76 125
5 or more courses 14 2 16
Total 103 227 330
graduates. Seven adult students in the dataset were enrolled in one of the universi-
ty’s fully online academic degree programs. Of the 330 students who took online
courses in the period studied, 189 students were single course registrants, i.e., these
students took one online undergraduate course over the 2-year period. Forty of these
189 students were adults, and 149 of these students were traditional undergraduate
students. Of the remaining students, 125 took two, three, or four online courses
across the four semesters (regardless of the format), and 16 took five or more online
courses during the four semesters, regardless of the format (Table 12.3).
2.3 Data Preparation
After the raw data were downloaded as a large batch file from the university’s stu-
dent information system, extensive sorting and recoding of several variables were
performed before meaningful analyses could be undertaken. The raw data from the
student records included student type (traditional or adult) and grade received (on
an A–F scale, converted to quality points on a 0.0–4.0 scale). The course informa-
tion included course delivery type (accelerated 7-week or semester-long 14-week),
course discipline, year, and term of course. A variable was created calculating the
total number of online courses completed per student during the time period.
Another variable was created to categorize the type of course, i.e., if the course was
a foundational core course, an enriched core course, or a course outside of the
required core curriculum.
Frequencies and means of final grades were calculated for all course registrations
in the dataset. Withdrawals were not included in the calculation of final grade
means. Once the figures were grouped by subgroups (course format, student type,
number of courses taken, content course type), differences in performance were
noted. T-tests and one-way ANOVAs were conducted to determine if the observed
differences were statistically significant.
3 Results
Initial frequency results indicated that overall, 47% of students earned an A, 9%

earned an F, and 6% withdrew from online undergraduate courses during the 2-year
time period. Students in the accelerated courses had a lower withdrawal rate of 3%
as compared to 10% withdrawal rate for semester-long courses. The share of A

grades and the share of failing grades were similar for both formats.
Additionally, the researchers observed that adult students had a higher failure
rate, as compared to traditional undergraduate students. As 14.5% of adults earned
failing grades in their online courses, just 4% of the traditional students received a
grade F. Throughout the semester, but after the add/drop period, 7% of adult stu-
dents and 6% of the traditional students withdrew from the online courses they were
initially registered to take. Finally, as can be seen in Table 12.4, adults earned fewer
A grades as compared to the traditional students, 38% and 55%, respectively. As
expected after reviewing the frequencies, the mean final grades for adults (2.72)
were lower than the mean of the final grades for traditional students (3.31). This
difference was statistically significant, confirmed by an independent samples t-test,
t(535) = −5.694, p < 0.001. No statistically significant difference was found com-
paring the mean final grade for online courses (3.04) and accelerated (3.07) among
all students.
This important finding, that adult students were not performing as well as the
traditional students did, provoked more analysis. The next logical step was for
researchers to examine the differences of both course type and student type com-
bined on final grades. Table 12.5 provides the summary of these results. Frequency
distributions indicated that adults lagged behind in performance compared to tradi-
tional students in both accelerated courses and semester-long courses. Only 3% of
traditional students earned an F in an accelerated 7-week course, while 16% of adult
students earned an F. Researchers also observed large differences in failures and
withdrawals between adults and traditional students in the semester-long formats.
Researchers analyzed the means for these four groups of students to further
examine the finding that adult students lag behind traditional students in both for-
mats. Results from an independent sample t-test comparing the mean of adult stu-
dents in accelerated courses (M = 2.64, SD = 1.41) to the mean of traditional students
Table 12.4 Frequencies and means of final grades by student type and course format
N A (%) F (%) W (%) Mean Std. deviation
All adults 255 37.6 14.5 7.1 2.72 1.40
All traditional 319 54.9 4.1 5.6 3.31 1.01
All accelerated 307 47.9 8.8 3.3 3.04 1.22
All semester-long 267 46.4 8.6 9.7 3.07 1.24
Total 574 47.2 8.7 6.3 3.05 1.23
Note: Means exclude withdrawal grades
Table 12.5 Frequencies and means of final grades by student type and course format combined
Adult accelerated 128 37.5 16.4 2.3 2.64 1.41
Traditional accelerated 179 55.3 3.4 3.9 3.32 0.98
Adult semester-long 127 37.8 12.6 11.8 2.81 1.39
Traditional semester-long 140 54.3 5.0 7.9 3.30 1.05
(M = 3.32, SD = 0.98) were statistically significant, t(294) = −4.922, p < 0.001.

However, a t-test did not confirm a difference among adult students in semester-long
courses (M = 2.81, SD = 1.39) and traditional students in semester-long courses
(M = 3.30, SD = 1.05). The next step was to perform within-group analysis.
The differences between the four groups, namely, adults who took accelerated
format courses, adults in semester-long courses, traditional students who took
accelerated online courses, and traditional students who opted for semester-long
courses online were tested with a one-way ANOVA. There was a significant effect
of course format on student type at the p < 0.001 level for the three conditions, F(3,
533) = 11.203, p < 0.001. Post hoc comparisons using the Tukey HSD test indicate
that the mean grade for the adults in accelerated courses (M = 2.64, SD = 1.41) was
significantly different than the traditional students in accelerated courses (M = 3.32,
SD = 0.98). Within-group comparisons indicated that there were no differences
between adults taking semester-long courses and accelerated courses. The same is
true for traditional undergraduate students.
Researchers mined further to determine what other variables within the dataset
could impact final grades besides course format and student type. Final grades also
varied between the two types of students when number of online courses, regardless
of format, was included in the analysis, provided in Table 12.6. Adult students who
enrolled in a single online course during the four semesters had a 12.5% failure rate
and 18% withdrawal rate. Traditional students who took a single online course dur-
ing the time period did better than adult students who took just one online course,
with 64.4% earning As and a low failure and withdrawal rate (5% and 3%, respec-
tively). Similar differences in achievement were visible between adult and tradi-
tional students who took two, three, or four online courses in the period of time
studied. Roughly a quarter of adult undergraduate students (21.5%) earned failing
grades. However, those traditional and adult students who took five or more online
courses had similar rates of success.
Mean final grade calculations also highlighted the weaker performance in the
courses detected in the frequency calculations of final grades of adult students as
compared to traditional students. Adult students who took one, two, three, or four
online courses on average did worse as compared to traditional students who took
one, two, three, or four courses online. Those adults who took five or more courses
on average did better than traditional students in the same category.
Table 12.6 Frequencies and means of final grades by student type and number of courses
combined
Adult, single course 40 35.0 12.5 17.5 2.79 1.44
Traditional, single course 149 64.4 4.7 3.4 3.42 1.00
Adult, 2–4 courses 130 36.2 21.5 6.2 2.45 1.54
Traditional, 2–4 courses 158 45.6 3.2 8.2 3.23 0.98
Adult, 5 or more courses 85 41.2 4.7 3.5 3.09 1.03
Traditional, 5 or more courses 12 58.3 8.3 0.0 3.02 1.31
This finding of adults again lagging required further testing. Results from a one-
way 2x2 ANOVA indicated statistically significant differences between final grades
among the six groups. There was a significant effect of online course experience on
student type at the p < 0.001 level for the six conditions, F(3,533) = 10.09, p < 0.001.
The post hoc Tukey HSD test indicated that the mean grade for the adults who com-
pleted 2–4 online courses (M = 2.45, SD = 1.54) was significantly different than the
mean grade for the traditional students who completed 2–4 online courses (M = 3.23,
SD = 0.98). However, the mean grade for the adults who only took one online
course (M = 2.79, SD = 1.44) was not significantly different from the mean grade of
the traditional students who only took one course (M = 3.42, SD = 1.00).
A subset of these data afforded the researchers the ability to compare student
performance measured by final grades in those courses that were offered in both
accelerated and semester-long formats. Two Spanish sections offered in Spring
2016 were accelerated (n = 15 students), and two sections offered in Spring 2017
were offered in the semester-long format (n = 15 students). An introductory course
in Art History was offered as a semester-long course in Fall 2015 (n = 4) and in the
accelerated format in Spring 2016 (n = 6). A 300-level enriched core course in
Theology was offered in Fall 2015 and 2016 in the semester-long format (n = 17)
and in Spring 2016 in the accelerated format (n = 8). Frequency tabulations indicate
a large difference in high academic achievement, as measured by the number of As,
between the two course formats in Theology. Both formats resulted in high failure
rates in the Art History course, 33% (accelerated) and 25% (semester-long).
The Spanish course offered in the accelerated format had a higher failure rate of
33% as compared to the semester-long format (7%). Table 12.7 delineates these
results. Although the Ns for these groups were admittedly low, the researchers ran
independent sample t-tests comparing the mean final grades for each format. The
results were not significant.
Another variable of interest within this dataset that researchers could examine
was to compare the differences across the kinds of courses, not just course formats.
Courses were organized into three categories: foundational core, enriched core, and
courses outside of the core. Table 12.8 depicts the differences in final grades across
these course content areas. It is evident that the courses within the foundational core
curriculum had the highest failure rate (13%) and withdrawal rate (8%), as com-
pared to enriched core courses, where the failure rate was 2% and withdrawal rate
Table 12.7 Frequencies and means of final grades for courses offered in both formats
Spanish 101, accelerated 15 20.0 33.3 6.7 1.68 1.60
Spanish 101, semester-long 15 20.0 6.7 6.7 2.55 1.11
Theology 319, accelerated 8 37.5 0.0 0.0 3.09 0.93
Theology 319, semester-long 17 64.7 5.9 5.9 3.55 0.99
Art history 122, accelerated 6 66.7 33.3 0.0 2.67 2.07
Art history 122, semester-long 4 75.0 25.0 0.0 3.00 2.00
Table 12.8 Frequencies and means of final grades by course content type
N A (%) F (%) W (%) Means Std. deviation
Enriched core courses 168 60.1 2.4 3.6 3.45 0.85
Foundational core courses 344 39.2 12.5 8.1 2.79 1.36
Courses outside of core 62 56.5 4.8 3.2 3.37 1.02
Total 574 47.2 8.7 6.3 3.05 1.23
was 4% and courses outside of the core, with 5% and 3% failure and withdrawal
rates, respectively. The lower grades scored in the foundational courses were also
apparent in the mean calculations. Similar to the other subgroup analyses, means of
final grades were calculated after the frequencies, removing the withdrawals. The
mean final grade for foundational core courses was 2.79 (SD = 1.36), lower than
mean final grades for enriched core courses and courses not within the core curricu-
lum [M = 3.45 (SD = 0.85); M = 3.37 (SD = 1.02), respectively].
To test these differences in means, a one-way ANOVA was conducted. Results
indicated that the difference in the final grades means among the three course con-
tent groups (foundational, enriched, and outside of core) was statistically signifi-
cant. There was a significant effect of course content type on final grade at the
p < 0.001 level for the three conditions [F(2, 534) = 18.981, p < 0.001]. Post hoc
comparisons using the Tukey HSD test indicate that the lower mean grade for stu-
dents in foundational courses (M = 2.79, SD = 1.36) was significantly different than
the grades in both the enriched core courses (M = 3.45, SD = 0.85) and the noncore
curriculum courses (M = 3.37, SD = 1.02).
4 Discussion
These results allow us to answer the research question that we posed at the onset of
the study as well as the sub-questions that emerged as the data were mined.
The major question that was posed at the onset of this research was whether adult
students perform any differently from traditional students in the various online for-
mats. The results suggest that on average, adult students are not succeeding in online
courses at the same rate as traditional undergraduate students. The rates of failure
are higher in both accelerated and semester-long course formats. In addition, adults
do not perform better when they take multiple online courses either simultaneously
or in sequence.
Since it may not be entirely clear why this may be the case, a question of why
this is happening may be a compelling one to ask. However, this question is outside
of the scope of research for this study and is a good future research opportunity for
the university. Despite not offering an explanation as to why the performance is low,
the results may offer an insight into one point of contention. Faculty are often reluc-
tant to offer accelerated courses to students arguing that this online format is harder
for the students, that students need time to get used to the course and to absorb the
material, or that it is impossible for students to do well in the shortened courses. The
results indicate that students perform somewhat equally in either format. Thus, the
answer to the question of the success in the online courses does not lie in the type of
format that students choose but has to do with other factors. This includes but is not
limited to the issues of online pedagogy, faculty readiness and motivation to teach
online, faculty and student engagement, and commitment to online teaching and
learning. These constructs, although hard to measure in online formats, must be the
focus of future endeavors.
One of the sub-questions asked whether the students do better in faster-paced
courses. Success data, as measured by the final grade, indicate that both adult and
traditional students did not perform any differently in the two online formats.
Another question of concern was whether taking multiple classes online, either in
sequence or simultaneously, results in better online performance as indicated by the
final grades. The results indicate that students who take multiple online classes,
whether at the same time or sequentially, on the whole do not perform better than
students who only take one online course. The exceptions are those students who
take more than five online courses in a sequence. One can argue that when students
take online classes sporadically, without a plan, students do not take these courses
seriously. This view aligns with Bawa (2016) who spoke about such misconceptions
as online courses being easier or less demanding. When students sign up for courses
with this mindset and then find out that this view does not hold true, such a discov-
ery might lead to decreased motivation, effort, and engagement in the course, or lead
the students to withdraw from the course. Students who commit themselves to tak-
ing online courses systematically, however, may lack those misconceptions and, as
a result, may be better positioned to succeed by being more motivated and prepared
to engage, and to exert effort to do well.
The final question of interest in this work was to find out whether the retention
rate as measured by the withdrawals is different for the two populations of students
studied. While the results indicate that the overall withdrawal rate of adult students
is somewhat higher than that of traditional students, one interesting point is evident
when the comparisons of students who took a single course are made. These results
indicate that the withdrawal rate for adults who take only one course is much higher
(17.5%) than that of traditional undergraduate students (3.4%). While this might
mean that more research is necessary, an argument that aligns with Cochran et al.
(2014) can be made. Cochran and colleagues found that academic experience and
readiness are strong predictors of persistence in a course of study. One can argue
that traditional students tend to attend school full time, regardless of how many
online courses they take. Adult students are at a disadvantage in this regard. Many
of them have an interrupted academic experience, and many of them take classes on
a part-time basis. This means that their academic experiences and overall academic
baggage, so to speak, may be different. As such, these students may require differ-
ent teaching approaches as advocated by Knowles (1980) and different university
supports as suggested by Park and Choi (2009). One of the arguments Park and
Choi advance is the relevance of content to the learner. It may be the case that adult
learners do not see the relevance of the core courses to their lives and, therefore, do
not put as much effort in them as compared to the classes they perceive as really
important. More studies, probably of qualitative nature, are necessary to understand
this issue further.
While these findings are discouraging, they do provide some food for thought.
More questions were generated as a result of this work than were answered. Future
research could look into the relevance of the core curriculum to the nontraditional
adult population. Such issues as pedagogical approaches within accelerated and
semester-long courses could be explored as well as issues related to student motiva-
tion and commitment to taking online courses, especially multiple online courses at
the same time, should be studied further. Another possible investigation into under-
standing the success of adult learners specifically is to compare the success data as
measured by the final grades in face-to-face formats as opposed to the online
formats.
4.1 Implications
The results of this study were shared with academic affairs leadership, faculty, and
academic support staff on our campus. It was clear to us, as researchers, that the
study, although not perfect, has some clear implications to the institution. In particu-
lar, the following four questions were identified for the institution: (1) what are
some student services decisions that we need to consider to help adult learners suc-
ceed?, (2) what are some enrollment and marketing decisions that we can make
based on these data?, (3) what are some professional development opportunities for
faculty that need to be considered in order to turn the tide?, and (4) what are some
curricula and pedagogical adjustments that should/need to be considered if we are
to continue to serve the adult population? While these questions do not have imme-
diate answers, the stakeholders on campus should begin considering these questions
while also collecting more data and considering other sources of data that may
provide better answers.
The four questions outlined above may serve as long-term guides to the change.
There are also immediate adjustments that have been made based on the data. For
example, to address the issue of readiness, adult undergraduates are now required to
take a quiz to determine how ready they are for online learning. The results of the
quiz are then discussed with their professional advisor who then offers tips on
improving online learning strategies to increase the likelihood of success.
Another finding that was uncovered through these analytics is the performance
of students in foundational core courses. The foundational core curriculum is cru-
cial in the development of writing, critical thinking, and information literacy skills
of students. These skills are necessary for making progress within the degree pro-
gram, as well as in entering and succeeding in the workplace. The finding that the
foundational core courses within this study have high rate of failures and withdraw-
als is important to consider and think about.
4.2 Limitations
This study has a number of limitations that should be considered in future research.
First of all, the study did not take into consideration a number of student character-
istics that may affect performance in online courses. These include but are not lim-
ited to student prior experience with online courses, level of comfort with technology,
amount of time devoted to studies, and student class level. Other variables related to
courses that may also affect performance include faculty type, differences in instruc-
tion, and level of rigor. These too were not considered.
One of the biggest and hardest to overcome limitations in this study is that no
actual student online engagement data were available to us. As researchers, we
would have welcomed the opportunity to study what exactly and how the students
have done in their online classes. Access to actual course data would have allowed
the researchers to study teaching approaches, student responses to prompts, interim
assessment grades, and other data normally stored within a typical Learning
Management System (LMS). We could not use these data however. One reason was
faculty permissions. Another is the limitations of this particular LMS. In addition,
on a small campus, it is much easier to put the names of the students one sees in the
Learning Management System with faces on campus, which feeds the concerns
about privacy and ethics as Slade and Prinsloo (2013) bring up in their work.
Another limitation is the one that concerns the size of the dataset. In this paper,
we argue that working with 574 course registrants constitutes big data for small
institutions. Researchers working in large institutions have access to thousands of
records at a time, and each semester the sets increase, sometimes exponentially. As
Chernobilsky, Jasmine, and Ries (2016) point out, small-scale projects rarely result
in massive data sets that traditionally are used in educational data mining and learn-
ing analytics. Engaging in data explorations, however, is equally important to large
and small institutions. We, thus, argue that for small institutions like ours, the size
of the data set is not a limitation but a reality of life. We believe that the size of the
data set may not be as important and should be considered in relation to the overall
size of the institution as long as the data set helps answer the questions addressed.
Sometimes, working with smaller numbers may be a way to begin understanding an
issue, as our study indicates. As the set grows, mining it further might provide an
opportunity for predictive modeling and other more sophisticated decision-making
tools available in the realm of educational data mining and learning analytics.
5 Conclusions
The researchers in this study sought to understand adult student success in semester-
long and accelerated online courses. Although this research, at its conclusion, posed
more questions than it answered, the researchers argue that learning from small data
sets may be just as valuable as learning from the large sets of data. Small institutions
by virtue of their size cannot collect large data sets easily. Working with smaller sets
for these institutions is just as valuable, however, as working with massive data sets
is for the large institutions when trying to understand issues of importance. Future
investigations should consider a variety of data sources such as face-to-face com-
parisons, other kinds of demographics, prior educational experiences, and student
experiences once in an online course. This case study showed how one small institu-
tion considers available data to learn about online performance of adult students. As
a result of this study, the university has renewed its commitment to using learning
analytics in understanding its student population.
References
Akyol, Z., & Garrison, D. R. (2008). The development of a community of inquiry over time in an
online course: Understanding the progression and integration of social, cognitive and teaching
presence. Journal of Asynchronous Learning Networks, 12(3–4), 3–22.
Alhassan, A. M. (2012). Factors affecting adult learning and their persistence: A theoretical
approach. European Journal of Business and Social Sciences, 1(6), 150–168. http://www.ejbss.
com/recent.aspx
Anastasi, J. S. (2007). Full-semester and abbreviated summer courses: An evaluation of student
performance. Teaching of Psychology, 34(1), 19–22.
Bach, C. (2010). Learning analytics: Targeting instruction, curricula and student support.
Proceedings from the 8th International Conference on Education and Information Systems,
Technologies and Applications: EISTA 2010, Orlando, FL. http://www.iiis.org/cds2010/
cd2010sci/eista_2010/index.asp
Bawa, P. (2016). Retention in online courses: Exploring issues and solutions - a literature review.
SAGE Open, 6(1), 215824401562177. https://doi.org/10.1177/2158244015621777
Bean, J., & Metzner, B. (1985). A conceptual model of non-traditional undergraduate student attri-
tion. Review of Educational Research, 55(4), 485–540.
Bowen, W., Chingos, M., & McPherson, M. (2009). Crossing the finish line: Completing college at
America’s public universities. Princeton, NJ: Princeton University Press.
Boyer, A., & Bonnin, G. (2017). Higher education and the revolution of learning analytics. Oslo,
Norway: International Council for Open and Distance Education.
Carr, S. (2000, February 11). As distance education comes of age, the challenge is keeping the
students (p. A39). Washington, DC: Chronicle of Higher Education, Information Technology
Section. Retrieved from http://chronicle.com
Casey, D. M. (2008). A journey to legitimacy: The historical development of distance education
through technology. TechTrends: Linking Research and Practice to Improve Learning, 52(2),
45–51.
Chametzky, B. (2018). Communication in online learning: Being meaningful and reducing iso-
lation. In A. Scheg & M. Shaw (Eds.), Fostering effective student communication in online
graduate courses (pp. 20–41). Hershey, PA: IGI Global.
Chen, B. (2015). From theory use to theory building in learning analytics: A commentary on
“Learning Analytics to Support Teachers during Synchronous CSCL”. Journal of Learning
Analytics, 2(2), 163–168. https://doi.org/10.18608/jla.2015.22.12
Chernobilsky, E., Jasmine, J., & Ries, E. D. (2016). Action research and data mining. In S. ElAtia,
D. Ipperciel, & O. Zaïane (Eds.), Data mining and learning analytics: Applications in educa-
tional research (pp. 67–78). Hoboken, NJ: Wiley.
Cochran, J. D., Campbell, S. M., Baker, H. M., & Leeds, E. M. (2014). The role of student char-
acteristics in predicting retention in online courses. Research in Higher Education, 55, 27–48.
Collins, R. A., Kang, H., Biniecki, S. Y., & Favor, J. (2015). Building an Accelerated Online
Graduate Program for Military Officers. Online Learning, 19(1), 102–111.
Dawson, S., Mirriahi, N., & Gasevich, D. (2015). Importance of theory in learning analytics in for-
mal and workplace settings. Journal of Learning Analytics, 2(2), 1–4. https://doi.org/10.18608/
jla.2015.22.1
De Vito, K. (2009). Implementing adult learning principles to overcome barriers of learning in con-
tinuing higher education programs. Online journal of Workforce Education and Development,
3(4), 1–10.
Donaldson, J. F. (2001, April). Accelerated degree programs: Policy implications and critique:
What we know about adult learners and its implication for policy. Paper presented at the
Annual Meeting of the American Educational Research Association, Seattle, WA.
EDUCAUSE Learning Initiative (2011, December). Seven things you should know about first
generation learning analytics. Retrieved from https://library.educause.edu/~/media/files/
library/2011/12/eli7079-pdf.pdf. Accessed 23 Jan 2018.
Geltner, P., & Logan, R. (2001). The influence of term length on student success (report no.
2001.4.1.0). Santa Monica, CA: Santa Monica College.
Henschke, J. A. (2011). Considerations regarding the future of andragogy. Adult Learning, 22(1),
34–37.
IBM (2011). Analytics for achievement paper. Ottawa, Ontario. http://public.dhe.ibm.com/soft-
ware/data/sw-library/cognos/pdfs/whitepapers/wp_analytics_for_achievement.pdf
Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC
horizon report: 2016 higher education edition. Austin, TX: The New Media Consortium.
Kauffman, H. (2015). A review of predictive factors of student success in and satisfaction with
online learning. Research in Learning Technology, 23, 1–13. https://doi.org/10.3402/rlt.
v23.26507
Kemp, N., & Grieve, R. (2014). Face-to-face or face-to-screen? Undergraduates’ opinions and test
performance in classroom vs. online learning. Frontiers in Psychology, 5, 1–11. https://doi.
org/10.3389/fpsyg.2014.01278
Knowles, M. (1980). The modern practice of adult education: From pedagogy to andragogy (2nd
ed.). New York, NY: Cambridge Books.
Lambert, C., Erickson, L., Alhramelah, A., Rhoton, D., Lindbeck, R., & Sammons, D. (2014).
Technology and adult students in higher education: A review of the literature. Issues and
Trends in Educational Technology, 2(1), article 3.
Lee, N., & Horsfall, B. (2010). Accelerated learning: A study of faculty and student experiences.
Innovative Higher Education, 35(3), 91–202.
Leitner, P., Ebner, M., & Khalil, M. (2017). Learning analytics in higher education - a literature
review. In A. Pena-Ayala (Ed.), Learning analytics: Fundamentals, applications, and trends:
A view of the current state of the art to enhance e-learning (pp. 1–23). Cham, Switzerland:
Springer.
Levy, Y. (2007). Comparing dropouts and persistence in e-learning courses. Computers and
Long, P., & Siemens, G. (2011). Penetrating fog: Analytics in learning and education. EDUCAUSE
Review, 46(5), 31–40.
Mattingly, K. D., Rice, M. C., & Berge, Z. L. (2012). Learning analytics as a tool for closing the
assessment loop in higher education. Knowledge management and e-learning. An International
Journal, 4(3), 236–247.
Mealman, C. A., & Lawrence, R. L. (2000). You’ve come a long way baby: The roots and resis-
tance of cohort-based accelerated learning. Paper presented at the Adult Higher Education
Association Conference, Chicago, IL.
Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence-
based practices in online learning: A meta-analysis and review of online learning studies.
Washington, DC: U.S. Department of Education.
Messina, R. C. (1996). Power package: An alternative to traditional course scheduling (ERIC
Document Reproduction Service NO. ED 396787).
Millett, C. M., Stickler, L. M., & Wang, H. (2015). Accelerated nursing degree programs:
Insights into teaching and learning experiences (research report no. RR-15-29). Princeton, NJ:
Educational Testing Service. https://doi.org/10.1002/ets2.12078
National Center for Education Statistics. (2015). Distance education in postsecondary institutions.
Retrieved from https://nces.ed.gov/programs/coe/indicator_sta.asp
Nguyen, T. (2015). The effectiveness of online learning: Beyond no significant difference and
future horizons. MERLOT Journal of Online Teaching and Learning, 11(2), 309–319.
on online learning. Educational Technology and Society, 12(4), 207–217.
Picciano, A. G. (2012). The evolution of big data and learning analytics in American higher educa-
tion. Journal of Asynchronous Learning Networks, 16(3), 9–20.
Piety, P. J. (2013). Assessing the educational data movement. New York, NY: Teachers College
Press.
Pino, D. (2008). Web-based English as a second language instruction and learning: Strengths
and limitations. Distance Learning, 5(2), 65–71. Retrieved from http://www.infoagepub.com/
index.php?id=89&i=59
Romero, C., & Ventura, S. (2013). Data mining in education. WIREs Data Mining Knowledge
Discovery, 3, 12–27. https://doi.org/10.1002/widm.1075
Ross-Gordon, J. (2011). Research on adult learners: Supporting the needs of a student population
that is no longer nontraditional. Peer Review, 13, 26–29.
Samaroo, S., Cooper, E., & Green, T. (2013). Pedandragogy: A way forward to self-engaged learn-
ing. New Horizons in Adult Education & Human Resource Development, 25(3), 76–90.
Scheffel, M., Drachsler, H., Stoyanov, S., & Specht, M. (2014). Quality indicators for learning
analytics. Educational Technology & Society, 17(4), 117–132.
Seamon, M. (2004). Short and long-term differences in instructional effectiveness between inten-
sive and semester-length courses. Teachers College Record, 106(4), 852–874. https://doi.
org/10.1111/j.1467-9620.2004.00360.x
Shacklock, X. (2016). From bricks to clicks: The potential of data and analytics in higher educa-
tion. The Higher Education Commission (HEC) report.
Shaw, M., Chametzky, B., Burrus, S. W., & Walters, K. J. (2013). An evaluation of student out-
comes by course duration in online higher education. Online Journal of Distance Learning
Administration, 16(4), 1–33.
Sheldon, C., & Durdella, N. (2010). Success rates for students taking compressed and regular
length developmental courses in the community college. Community College Journal of
Research and Practice, 35, 39–54. https://doi.org/10.1007/s10755-010-9141-0
Stack, S. (2015). Learning outcomes in an online vs traditional course. International Journal
for the Scholarship of Teaching and Learning, 9(1), article 5. https://doi.org/10.20429/
ijsotl.2015.090105
Tatum, B. C. (2010). Accelerated education: Learning on the fast track. Journal of Research in
Innovative Teaching, 3(1), 34–50.
van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education:
Establishing a common language. EDUCAUSE Learning Initiative, 1, 1–11.
Wagner, E., & Ice, P. (2012). Data changes everything: Delivering on the promise of learning ana-
lytics in higher education. EDUCAUSE Review, 47(4), 32.
Part III
Learning Analytics Case Studies: Practices
and Evidence
Chapter 13
Empowering Teachers to Personalize
Learning Support
Case Studies of Teachers’ Experiences Adopting a
Student- and Teacher-Centered Learning Analytics
Platform at Three Australian Universities
Natasha Arthars, Mollie Dollinger, Lorenzo Vigentini, Danny Y.-T. Liu,

Elsuida Kondo, and Deborah M. King
1 Introduction
1.1 Students’ Success and Teachers’ Roles
Ensuring student success is a multifaceted challenge facing higher education insti-

tutions worldwide, particularly in light of pressures such as the massification,
commodification, and diversification of higher education. We adopt Kuh and col-
leagues’ definition of student success as “academic achievement, engagement in
educationally purposeful activities, satisfaction, acquisition of desired knowledge,
skills and competencies, persistence, attainment of educational objectives, and
postcollege performance” (Kuh, Kinzie, Buckley, Bridges, & Hayek, 2006, p. 7).
Student engagement is viewed as a key indicator of student success as the extent
to which students engage in educational activities is likely to determine whether
they will succeed in their studies (Kuh et al., 2006). Key mechanisms that support
student engagement include improving the quality of student-staff relationships,
N. Arthars · D. Y.-T. Liu (*)

DVC (Education) Portfolio, The University of Sydney, Camperdown, NSW, Australia
M. Dollinger
Student Success, La Trobe University, Bundoora, VIC, Australia
L. Vigentini
PVC (Education) Portfolio, The University of New South Wales,
Kensington, NSW, Australia
E. Kondo · D. M. King
Faculty of Science, The University of Melbourne, Parkville, VIC, Australia

https://doi.org/10.1007/978-3-319-64792-0_13
224 N. Arthars et al.
encouraging timely help-seeking behavior, and clarifying expectations (e.g., Kahu

& Nelson, 2018; Krause & Coates, 2008; Zepke & Leach, 2010). A sense of suc-
cess in the early stages of students’ university careers is also critical in building
persistence and downstream academic achievement (Tinto, 2006). Lizzio (2006)
has characterized five ‘senses of success’, including students’ needs to feel capable,
connected, purposeful, resourceful, and competent in terms of navigating academic
culture. While ‘engagement’ and ‘success’ are necessarily broad, what is abun-
dantly clear from the literature is the key role that teachers1 play in fostering the
abovementioned factors.
Krause and Coates (2008) point out that “[s]tudent perceptions of the learning
environment and the commitment of academic staff to supporting student learning
have a profound influence on student satisfaction and sense of belonging” (p. 501).
A teacher’s role in developing relationships and trust with students can powerfully
impact on their engagement and academic achievement (Bryson & Hand, 2007;
Reason, Terenzini, & Domingo, 2006). This has been encapsulated in the idea of
‘relational pedagogy’, which espouses that the “positive messages that are implicit
when academics give time and support to students are significant in helping students
feel that they are both worthy of their place and able to succeed in the university”
(Pearce & Down, 2011, p. 492). Despite the commodification of higher education,
the human relationships that exist at the core of learning and teaching are still criti-
cal but increasingly forgotten.
The unfortunate side effect of burgeoning class sizes and the concomitant sense
of anonymity in today’s higher education contexts undermines this central tenet of
student success (Bryson & Hand, 2007). How students perceive these exchanges
with their teachers has a positive impact on academic achievement, engagement,
satisfaction, and retention (Farr-Wharton, Charles, Keast, Woolcott, & Chamberlain,
2018). Importantly, this conceptualization emphasizes the need for individualized
communications: “Keeping the channels of communication open between the
instructor and student is essential to enhancing the quality of exchanges. Students
need to perceive that there is ready access to the instructor” (Jacques, Garger,
Thomas, & Vracheva, 2012, p. 9). An example of these exchanges is feedback,
known in the assessment literature to be another critical touchpoint between stu-
dents and teachers (Hattie & Timperley, 2007). Timely, specific, goal-oriented feed-
back that clarifies standards and develops positive motivation and self-assessment
strategies help to foster students’ self-regulation and improve learning (Nicol &
Macfarlane-Dick, 2006). The challenge in the context of higher education today is,
of course, to scale these exchanges that provide learning support without losing the
timeliness and personalization aspects. Proposed methods for such scaling increas-
ingly include software (e.g., Pardo, 2017) that allow for teachers to measure student
achievement in greater numbers and inform future teaching designs and approaches.
1
We use “teachers” in this chapter to refer to educators who design and deliver learning experi-
ences for students. This includes coordinators who have additional responsibilities such as broader
curriculum design and ownership, as well as tutors (or teaching assistants) who work under
coordinators.
13 Empowering Teachers to Personalize Learning Support 225
Ostensibly, this is one of the promises of the field of learning analytics (LA),
which purports to have a “[f]ocus on informing and empowering instructors and
learners” (Siemens & Baker, 2012, p. 253). This field typically focuses on big data
available from digital learning systems, algorithmically analyzing behavioral “user
events” in the form of logs of interactions and being concerned with combining logs
from different data sources (Pardo & Dawson, 2016). An important issue here is that
LA can be characterized as taking a computational view of learning, missing out on
its relational and humanistic aspects. Some authors have rightly warned that the
‘learning’ in LA is being forgotten in favor of the ‘analytics’ and that a return to the
root of learning and teaching including considering pedagogical intent and context,
and involving students and teachers as the key stakeholders, is essential (Gašević,
Dawson, & Siemens, 2015).
1.2 T
he Contexts of Teaching and the Learning Analytics
Needs of Teachers
Perhaps symptomatically, reports from around the world suggest laggard adoption
and implementation of LA by teachers. Recent reports on Australasian LA adoption
and implementation have highlighted that, as the primary implementers of any LA
tool, teachers need to be involved in designing LA approaches that “are sensitive to
their environments, meeting and extending their pedagogical requirements, and
ensuring flexibility” (Colvin et al., 2016, p. 19). In this context, and in keeping with
the relational pedagogy outlined above, a key need seems to revolve around actions
that involve personal connections with students, which balances the automation of
computers with the humanistic approach of teaching (West et al., 2015). Notably,
this report highlighted that teachers “still have to make sure that it [communication
and feedback] is personalized and meaningful for students” and that teachers need
LA tools with “some ability to modify it to their own requirements because each
course and each cohort of students may differ” (p. 20).
The learning and teaching landscape in any institution, faculty, and indeed
course2 is unique and influences the uptake of any innovation, especially LA
(Ferguson et al., 2014). Several factors can impact adoption, but some are particu-
larly relevant to teacher- and student-centered LA:
1. Faculty resistance to change and workload issues are examples of social and
cultural context that need to be understood and addressed (Macfadyen & Dawson,
2012), including concerns around needing to adapt to new tools and approaches,
and change existing practices.
2
“Course” is defined in this chapter as an individual component of an academic program that a
student takes, usually lasting a semester. For example, it is referred to as a “unit of study” at the
University of Sydney, a “subject” at the University of Melbourne, and a “course” at the University
of New South Wales.
2. A large proportion of learning and teaching activities typically occurs outside the
online space (not just outside the confines of a learning management system
[LMS]) and often involves human interaction (West et al., 2015), presenting
challenges for capturing and using the right data in the right place.
3. The lack of available tools that properly address the needs of teachers and stu-
dents (Colvin et al., 2016) and a lack of bottom-up support and sharing that is
driven by LA users (teachers and students) who have personally experienced
tangible benefits, potentially causing stalling or retraction of interest (Liu,
Rogers, & Pardo, 2015).
From these challenges, it may be surmised that a potential solution for teacher adop-
tion is LA software that (simultaneously, in one place) assists them in capturing and
working efficiently anywhere and in real time with a wide and flexible range of
meaningful data, addresses their felt needs while reducing workload, and can yield
immediate, shareable benefits. Existing LA tools are predominantly based on dash-
boards or mail merge (Lawson, Beer, Rossi, Moore, & Fleming, 2016; Tanes,
Arnold, King, & Remnet, 2011; Verbert, Duval, Klerkx, Govaerts, & Santos, 2013).
However, these single-purpose LA tools that just present a dashboard or just allow
databasing and creation of mail merge emails may be effective in addressing parts
of the whole ‘data lifecycle’ that teachers must manage through the course of a
semester or year but fail to address its entirety nor the three challenges noted above.
For example, dashboard tools are typically view-only, do not afford direct-to-student
or two-way communications, and even sophisticated reporting outputs may be seen
as a workload imposition with little or no benefit (Macfadyen & Dawson, 2012).
Additionally, mail merge tools typically do not afford data collection processes or
predictive analytics.
In an example of a consolidated LA workflow, Pardo and Dawson (2016) out-
lined a multistep lifecycle for LA which was geared toward improving learning
practice; their process involved (1) data capture and combination from logs and
other sources such as demographics, (2) data visualization and analysis through
reporting interfaces, (3) algorithmic generation of models for prediction of learning
outcomes, and (4) interventions at various levels of the university enterprise ranging
from students and teachers to directors and administrators. Here, we contend that
this conceptualization is still too computational and does not sufficiently involve
teachers throughout the cycle nor address their barriers to adoption identified above.
We therefore propose a reconsideration of this lifecycle that is more humanistic and
meaningfully addresses teachers’ and students’ immediate needs in a wide range of
contexts, in order to shift the conversation from single-purpose LA tools to multi-
functional LA platforms that may address these needs in an integrated way:
1. Data collection—the right data needs to be gathered from both online and face-
to-face learning and teaching environments. This does not necessarily need to
include system logs, nor demographics.
2. Data curation—all relevant data need to be accessible in one place. The teacher,
with their understanding of the pedagogical and pastoral contexts of their course,
should be the one making the informed decisions about what data to curate and
when.
3. Data manipulation and analysis—the ‘raw’ data may need to be transformed or
otherwise manipulated before it can yield a useful representation of information
or be used to inform subsequent action. This does not necessarily need to, but
could, involve any automated or algorithmic processing.
4. Actions enabled by the presence of data—providing learning support to students
needs to occur in a timely way, account for individual student needs, and con-
sider the classroom climate (Hattie & Timperley, 2007). Personalized support
delivered by an electronic system (e.g., via email or a web page) helps to address
this, but empowering the entire teaching team with relevant data when interact-
ing with students face-to-face and online is also important.
5. Closing the loop and evaluation—feeding students’ engagement with, and per-
ception of, personalized support back into the system so that teachers can use it
to improve their approach.
6. Reflection—prompt and guide reflection on teaching and support practices by
providing easy access to relevant representations of data.
Taken together, these needs and challenges speak to the importance of personalizing
the learning environment. The term “personalized learning” encompasses a wide
range of approaches that, broadly speaking, seek to tailor the content, support, and
pathways that students receive based on some information known about each stu-
dent (Alli, Rajan, & Ratliff, 2016). By amplifying the intelligence of human teach-
ers with the agility of software (Baker, 2016), LA can help teachers leverage student
data to provide timely, pedagogically meaningful, and tailored support. In complet-
ing the above LA lifecycle, teachers also change their practices based on data about
students and the impact of support they are provided. We contend here that this
personalized learning is therefore not just about personalization (tailoring) but also
person-alization (humanizing) students’ learning experience by teachers.
We next introduce an LA platform that was developed to address these issues,
followed by the experiences of three Australian institutions—the University of
Sydney, the University of Melbourne, and the University of New South Wales
(UNSW) Sydney—which are at different maturity levels of its adoption and imple-
mentation. As part of this, we discuss the context in which each institution is using
the platform and the rationale behind its adoption. Finally, we synthesize the impact
of the platform on teachers and their students, discuss a series of implications for
practice, and conclude with future research directions.
2 The Student Relationship Engagement System (SRES)
Teachers lack the requisite combination of tools to fully control the personalization
process for their students through the “data lifecycle” outlined above. To this end,
the Student Relationship Engagement System (SRES; www.sres.io) is a unique LA
Fig. 13.1 Schematic of the Student Relationship Engagement System (SRES) as a multifunc-
tional learning analytics platform. Data are collected and curated (left half) into a database that is
unique for each course (center). Data can be manipulated, analyzed, and used in many ways (right
half), some of which feed more data back into the system (thin dashed lines). Students interact with
the SRES through a number of modalities and can also feed data back directly into the system
(double-ended arrows). Teachers are in full control of all these stages of the learning analytics data
lifecycle, accessing the SRES through a web interface
platform, housing a wide range of flexible and highly customizable tools, which has
been developed to give teachers full control over the data lifecycle and to empower
them to use data in ways that suit their specific teaching contexts (Fig. 13.1). The
SRES addresses our proposed LA data lifecycle by providing a platform where
teachers have ownership and control over each stage.
1. Data collection—teachers can import most kinds of electronic data into the
SRES (keyed by a unique student identifier) or set up data synchronizations with
databases or LMSs. Importantly, they can also build simple mobile web app
interfaces within the SRES to collect data from face-to-face learning and teaching
environments (e.g., in-class attendance, participation, or assessment data).

Similar web interfaces (“portals”) can also be presented to students so that they
can enter relevant information directly into the system.
2. Data curation—all collected data can be curated into “lists” (essentially a data-
base, usually one such list per course) within the SRES and made accessible
online to other teaching staff within the course in real time. A key factor here is
that teachers are in full control of the columns (attributes, fields) in their lists.
3. Data manipulation and analysis—spreadsheet-like data manipulations can be
performed directly within the platform, including using rules that can range from
simple to complex depending on teachers’ needs. They can also visualize data by
creating custom dashboards and apply machine learning algorithms (clustering,
decision trees, and association rule mining) to uncover hidden patterns in large
datasets to better understand how students are engaging and succeeding (Liu,
Taylor, Bridgeman, Bartimote-Aufflick, & Pardo, 2016). Predictive models can
also be built and applied using teacher-selected attributes. All of these manipula-
tions and analyses can be performed by teachers without any coding, simply by
selecting relevant columns and operations via a graphical user interface.
4. Actions enabled by the presence of data—teachers can provide personalized
online support to students by sending customized emails or pushing personalized
web page content (“portals”) to a student’s LMS. They can also use data to
inform face-to-face actions, such as presenting relevant data to teaching assis-
tants at the point of contact to contextualize teaching activities to address identi-
fied learning needs. Custom dashboards can be shared with other teachers and
even students. Customizable reports can also be designed to automatically
inform members of the teaching team about students who meet teacher-defined
criteria.
5. Closing the loop and evaluation—teachers can see who has opened emails sent
from the SRES, how many times, and whether links have been clicked. This can
inform the need for further action, such as follow-up communications either
online or face-to-face. Teachers can also capture feedback about whether their
message has been helpful to students by enabling a function that allows students
to vote and provide qualitative feedback explaining how and why.
6. Reflection—by variably combining the custom visualizations, closing-the-loop
information, machine learning insights, and by virtue of having all relevant data
in one place, teachers can evaluate the impact of their actions and better under-
stand the characteristics of their student cohorts. Based on this, they may, for
example, adjust future approaches to student learning support in terms of recipi-
ent pool and messaging.
In stark contrast to other LA approaches and tools, the SRES gives precedence to
teacher intelligence and small (but meaningful) data over predictive algorithms and
big data. It enables teachers to design an LA approach that is contextualized to their
unique learning and teaching situation. This may include collecting and curating
traditional student engagement and performance data such as attendance, LMS use,
tutor feedback, and grades but may also include nontraditional information such as
Fig. 13.2 Key functionality of the SRES mapped to the six stages of the proposed LA data
lifecycle
those that students proffer about themselves, such as their preferred name, photo,
and details such as their background and interests. The flexibility of the SRES
affords teachers the ability to leverage a wide range of data to suit the needs of their
teaching practices and student cohort. Together with the functionality built into the
platform, teachers are given control of the whole data lifecycle (Fig. 13.2), enabling
them to obtain and use contextually meaningful academic engagement and success
data to foster relationships with, and provide support to, their students.
3 Institutional Case Studies
3.1 Methodology
To conduct a cross-institutional study, three Australian universities who currently

have access to the SRES platform were selected: the University of Sydney, the
University of Melbourne, and UNSW Sydney. Between these institutions, the
maturity of adoption of the SRES varied, with the University of Sydney being the
first developers and adopters, giving teachers more time to adjust to and implement
the platform (Vigentini et al., 2017). Across all three cases, however, the research
sought to investigate the following broad questions which applied regardless of the
maturity of adoption:
1 . Why did teachers choose to adopt the SRES?
2. How did teachers use the SRES to support student success? (That is, what data
did they select, how did they use these data, and why?)
3. What are teachers’ perspectives and experiences of the impacts and effectiveness
of the SRES on them and their students? (For instance, on students’ engagement,
satisfaction, and success?)
4. What are students’ perspectives of the personalized support messages received
from teachers via the SRES?
The mixed-methods methodology for the investigation focused primarily on semi-
structured interviews and informal feedback with teachers (including coordinators,
lecturers, and tutors), supplemented with qualitative and quantitative data from the
SRES platform including percentage of emails opened and responses from students
regarding the helpfulness of communications received through the SRES. This
approach was taken because it is often difficult to establish the impact of a platform
which can be applied in many different ways for different purposes; in the context
of LMSs, Coates, James, and Baldwin described this issue as being “not the provi-
sion of features but their uptake and use that really determines [a platform’s] educa-
tional value” (2005, p. 26). The diversity of uses (and indeed teachers and students
and their individual characteristics) also precluded predominantly quantitative mea-
sures of impact, even though they may be possible for more focused programs
(Dawson, Jovanovic, Gašević, & Pardo, 2017).
While these research questions were used across all three institutions, the highly
customizable nature of the platform meant that it was not possible to compare
courses within institutions, nor across institutions. Currently across the three uni-
versities in this study (Fig. 13.3), the SRES houses teacher-selected data for over
43,000 students (2017 count, over 72,000 projected for the entirety of 2018). These
Fig. 13.3 Combined 80000 2000

adoption measures of the Students
Number of faculty/staff
Number of students
70000
SRES at three Australian
institutions. Figures for 60000 Faculty and staff 1500
2018 are projected based 50000
on half-yearly data
40000 1000
30000
20000 500
10000
0 0
2012 2014 2016 2018
10000 500
Lists in use
Number of columns
8000 Columns in use 400
Number of lists
6000 300
4000 200
2000 100
0 0
2012 2014 2016 2018
Fig. 13.4 Combined use measures of the SRES at three Australian institutions. Figures for 2018
are projected based on half-yearly data. The database structure of the SRES allows teachers to cre-
ate “lists” (typically one per course) with students and create relevant “columns” in these lists to
house data of their choosing
data relate to students across 360 lists (2017 count, 470 projected for 2018), each
list typically representing a single course (Fig. 13.4). Across these lists, 5100 col-
umns (2017 count, 9000 projected for 2018) contain data about students, each of
which has been selected by teachers for use. While a large sample size usually lends
itself to more impressive statistical outcomes, in the context of the SRES, this pri-
marily resulted in an overwhelming diversity of variable uses and users. Therefore,
this study sought a more holistic understanding of how teachers were using the
platform and the reported impacts of this on students. The case studies start with a
wide-ranging snapshot of its use at the University of Sydney, the original developers
of the platform where it has been used by an increasing number of teachers since
2012. More focused perspectives are then presented for the University of Melbourne
and particular courses at UNSW Sydney.
3.2 Case Study 1: The University of Sydney
The SRES was developed in 2012 by two teachers in the Faculty of Science respon-
sible for coordinating large first-year units with up to 2000 students per semester.
After being used primarily within the Faculty of Science for the first two years, it
spread organically across other areas including Arts and Social Sciences, Business,
Engineering, Health Sciences, and Medicine. It now reaches over 50% (in 2017,
over 32,000 unique students) of the entire university’s student cohort, almost 5,000
weekly users (students, teachers, and support staff), and over 1,500 registered fac-
ulty and staff users. In these diverse settings, teachers’ adoption of the SRES has
extended from enhancing teaching and learning to streamlining administration and
course coordination.
A recurring theme from interviews was the use of the SRES to capture student
attendance and participation grades. Many reported collecting attendance at
face-to-face sessions such as tutorials and laboratory classes. Compared to tradi-
tional practices of using paper-based rolls that were often lost or sometimes entered
at the end of the semester, the SRES allowed teaching teams to record data and act
upon it throughout the semester. Many teachers viewed attendance as an important
indicator with both their personal experiences and empirical studies indicating a
correlation between student attendance, participation, and performance (Credé,
Roch, & Kieszczynka, 2010; Newman-Ford, Fitzgibbon, Lloyd, & Thomas, 2008).
As one coordinator explained, “it tends to be the case that if you don’t turn up you
just don’t have the opportunity to ask as many questions and really sort of nut out
those ideas. Earlier in the degree attendance is a lot more important because they’re
picking up the basic concepts that they need for the rest of their degree, and possibly
for the rest of their lives.” Having access to attendance data in the SRES allowed
teachers to identify students who were not engaging, with many using this informa-
tion to generate personalized emails, reminding students of attendance requirements
and offering support where required.
Interestingly, others provided additional reasons for collecting attendance data,
noting that the process of collecting this data was in itself an opportunity to engage
with students. The change in tools and process meant that instead of calling names
from a paper-based roll, teachers would move from student to student scanning their
student ID using the SRES mobile web app. Some teachers reported that this pro-
cess allowed them to learn the names and faces of their students more quickly, while
some reported this as an opportunity to provide feedback to students about their
progress. In these situations, the technology caused a change in practice which
helped to strengthen teacher-student relationships.
To achieve a similar goal, some teachers reported directly collecting information
from their students through the SRES at the beginning of semester. They noted the
positive effects of having this information, such as work experience, interests, career
trajectory, and even student photos, with one coordinator reporting, “that really
helped with our tutors because they felt like they had this photo there that they could
immediately see who their students were, and then they also had something interest-
ing about them that they could use to memorize who they were as well. They found
that really useful.” Some used this information to inform their teaching practices as
it allowed them to contextualize these based on the cohort of students. One tutor
with students from a range of majors used this information to structure class discus-
sions based on the different background knowledge and unique perspectives of indi-
vidual students present. They found this to be a more positive experience, stating,
“because I could go directly to a person who I already knew had background—
whether it be through their major or through their work experience—I was able to
generate a much smoother discussion in class.” Reflecting on students’ perspective
of this experience, they reported, “their perception was here’s a teacher who actu-
ally knows all about me already. They’ve taken the time and the effort to understand
me as an individual.”
Coordinators reported using face-to-face data in conjunction with data from a

variety of online platforms, including the LMS and external discussion boards such
as Piazza. For example, having access to last log-in date and discussion board usage
allowed coordinators to identify whether students were actively contributing, pas-
sively viewing, or not engaging at all. Combining multiple data sources in the SRES
allowed coordinators to identify students who were at risk of lower performance.
The focus on this combined dataset was primarily in the early weeks of the semes-
ter, including the weeks leading up to the census date (when students can unenroll
without financial penalty). One teacher reported using this information prior to cen-
sus to send personalized emails to students who were at risk, advising them and
offering support. Interestingly, they reported a reduction in the number of students
who dropped out, stating “we used to have maybe 30, 40 people drop out minimum.
Now we can have a handful” (cohort size 800–1300). Another teacher using the
SRES in a similar way reported a halving of the year-on-year withdrawal rate in a
first-year cohort of 270, without any other changes made to the course except the
introduction of the SRES.
In addition to capturing and acting on face-to-face and online participation data,
the SRES was used heavily for providing personalized assessment feedback. As
with attendance, many teachers increasingly used the SRES to mark and provide
feedback for in-class assessments such as presentations. To do this, course coordi-
nators or learning designers built forms within the SRES, which were then used by
markers on a mobile-friendly SRES web app that enabled live data collection.
Teachers then used these data to build personalized feedback via email, text mes-
sage, or through a web page embedded into the LMS. This involved creating condi-
tional text and piping entered data into templates, as with mail merge approaches.
Some teachers reported grouping students based on performance (such as low,
medium, and high) and sending feedback that was customized to each of these
groups. Others reported much more complex processes in which students were
assessed against a detailed set of criteria and would receive highly personalized
emails with feedback comments tailored to their performance against each criterion.
In addition to receiving feedback, they would also receive suggestions for
improvement.
The use of the SRES for assessment feedback not only allowed for more efficient
marking processes but also resulted in more timely and detailed feedback to stu-
dents. Providing more immediate feedback was seen as important, particularly
where assessments built upon each other or required the application of similar skills.
As one coordinator reflected, “We’ve moved from, I’d have to say, not the best feed-
back mechanisms up to now very prompt feedback on any submitted work. So that
the students, before they have to complete their next submission task, have an
opportunity to improve.” The personalized nature of the feedback was also well-
received by students, with one coordinator noting, “I routinely get emails back from
students who believe I have personally sent them that email. Who believe that I have
taken the trouble to individually engage with them and give them that feedback. I
get thank you emails, and I get questions about the email that I sent, from time to
time. Not a huge number, but enough to give me a good indication that that’s the
way that they seem to be received.” Some coordinators also emphasized the benefit
of having timely access to assessment grades for their course as this allowed them
to quickly identify trends, including areas where further instruction was required for
all students to address learning gaps. From here they were able to communicate this
to tutors and ensure that subsequent classes focused on bridging these gaps.
In addition to using the SRES to provide personalized feedback to students’
assessment tasks, some teachers also utilized the SRES to contact students who
were not performing well and may have needed support. For students who had not
submitted, some coordinators used personalized emails to remind them of the
requirement to submit, to negotiate deadlines, and to remind students of the process
to make arrangements for missed assessments. For students who had submitted but
were not performing to the level required, coordinators often sent emails recom-
mending they attend additional workshops or engage with support services. While
many wanted to provide support and encouragement to other students, time con-
straints were a major challenge which impeded their ability to do so. As one teacher
reflected, “we’re very good at sending complaint emails when things go wrong, but
when things go well we don’t tell people and people need those spontaneous good
emails,” Another teacher noted, “reinforcing the good ones—there’s immense value
in that and we forget that group, often. We don’t give them enough praise and rec-
ognition.” Interestingly, one coordinator did report using it to identify high perform-
ers in order to email them about opportunities for further study.
While many see the value in adopting the SRES, for tutors, the support of the
coordinator was an important factor in being able to implement and use the SRES
within a course. One tutor reported, “I had a whole lot of fights with the [course]
coordinator to get SRES into the [course].” To overcome this resistance, they ini-
tially trialed it with one class of students on a limited basis and then rolled it out
across all classes. The workload generated by sending emails was also a point of
reflection for staff. Some teachers mentioned challenges engaging with students
despite the personalized nature of their emails, reporting that they perceived that
often students did not even open them. In contrast, others reported an increase in the
responses received from students as a result of sending emails through SRES. Despite
the increase in workload, many teachers actually commented on this positively. As
one noted, “[f]rom a workload perspective, yes, it is generating a whole lot more
[email responses from students] that you wouldn’t otherwise have. It’s actually
really quite productive [at engaging students] that way.” Another reflected, “Care
[for students] overrides the [additional] time.”
This sentiment was echoed in student comments, which were volunteered
directly to the platform in response to personalized messages. These allowed the
platform to accurately capture students’ perspectives and therefore allowed teachers
to close the loop and start reflecting on their support approaches:
• “Thank you for the feedback! Understanding the breakdown of marks and feed-
back in such detail really helps prepare for other [assessments] and next year
placement. Thanks for the semester!”
• “Just let me know teachers do concern [sic] about my study and my learning
outcome, and helpful advice to do better.”
• “It is really helpful, thanks for encouraging me onto the rest of the semester, I
was losing it and I thought I might just give up. But thank you very much and I
will keep up the good work!!!!!!!!!!!!!”
• “This message shows me which part I can do better in final exam and makes me
feel the professor is kindness.”
At the University of Sydney, the SRES has provided teachers with a practical
platform to work with student data on attendance, assessment, participation, and
engagement in face-to-face and online environments. This has led to tangible ben-
efits including improved student feedback and engagement as well as administrative
efficiencies, which together has gradually helped to overcome faculty resistance.
Additionally, small communities of practice have formed within and between
departments, where learning designers and faculty worked together to share success
stories and help provide on-demand support. Despite the challenges some teachers
have experienced, the involvement of teaching staff in the design and development
of the SRES over the years has also improved the extent to which it has been adopted
and used across the university. As one participant noted, “the selling point is that it
was made by a teacher. Because teachers know what teachers need.”
3.3 Case Study 2: The University of Melbourne
The second case study involves teachers from the Faculty of Science at The
University of Melbourne, a large research-intensive university, who piloted the use
of the SRES from early 2017 (Vigentini et al., 2017) and as such are still relatively
new adopters of the platform. Melbourne’s adoption began with a conference pre-
sentation piquing interest, leading to the joint initiative between the platform’s
Sydney developers and the academic faculty leadership team at Melbourne.
Melbourne was further supported in the implementation of the platform by an infor-
mation technology specialist, who had her role shifted to help support the SRES.
Participants in the SRES pilot were coordinators from five courses ranging from
Mathematics, Biology, and Chemistry, and all courses had over 300 students
enrolled. The structure of the courses often included multiple lectures each week as
well as additional tutorials and workshops. Assessment typically included a large
examination component (e.g., 70–80%) with the remaining assessment linked to
assignments and laboratory work. Within all the courses that participated in the
study, coordinators noted that they believed student engagement could be improved.
Yet despite these ongoing concerns, coordinators felt they had few routes to improve
engagement or understand what other measures, apart from attendance, could be
designed and implemented to better track student engagement.
Participants were drawn to trial the SRES for a multitude of reasons. One reoc-
curring reason stated by participants to use the SRES in their course was
dissatisfaction with the existing LMS’ functionality. While the SRES was originally
designed to supplement the LMS rather than to replace it, participants noted that
using the SRES decreased the amount of time they spent using the LMS. One coor-
dinator noted, “[t]he systems we’ve got at the moment are a bit old, a bit clunky…
we needed a better system for recording data, [the SRES] seems to be an improve-
ment over what we’ve got at the moment.”
However, the SRES’ appeal was not only a sleeker interface. Participants also
mentioned that they were motivated to participate in a system that could improve
student support. The ability to send personalized emails to students, a function not
available on the current LMS, intrigued participants. One noted, “I wanted the stu-
dents to feel like we were really interested in their progress, so to be able to person-
alize an email to them and point out what support was available to them if they were
struggling, I really like that idea….” Another participant voiced similar motivations
for using the SRES, “It just sounded like a way we could interact with the students
on a more personal level, we’ve heard comments, and we try to reach out to as many
as we could, and [with the SRES] now if there are at risk students we could sort of
go, ‘Hey, you are at risk’.”
Yet despite the appeal of personalized emails, many participants in the pilot did
not fulfil their hope of using the email function available through the platform.
When asked about how they used the platform, many coordinators only used it for
rudimentary functions, such as recording attendance or marks. However, this find-
ing not only did not match participants’ original motivations for using the platform,
which were often far loftier, but also was subsequently modified by many partici-
pants who still wished to use the personalized email function in future semesters.
One participant explained, “[w]hen I started using the SRES the semester was
already underway, and you got to brief tutors about how to use the system and so on,
so by that point it felt a bit too late… also partly because you know I’ve got a hun-
dred other things going on in a semester, and partly because it’s also with so many
tutors it is a bit of an effort to hunt down tutors to get them to input their data.”
Further complicating the research was the ad hoc way that many coordinators
piloted the email function within their teaching design and course delivery. One
teacher noted, “I sent an email to basically the bottom quartile of the class… picked
a point where I thought anything above that seemed reasonable.” He emailed those
students regarding their current mark and reminded them to do their assignments
and attend tutorials and offered consultation times. In response, some students
emailed thanking him for letting them know or for noticing, although he also
received some emails saying that students’ assignment marks were missing from
tutors who had yet to input them.
In fact, a common issue that arose in interviews about the SRES was the lack of
engagement from tutors, rather than students. As the courses had such large student
enrolments, some coordinators had close to 40 tutors during the semester. For the
data to stay up to date, all the tutors needed to take attendance using the SRES web
app (and if by hand, they would later input it into the system) and enter all student
grades in a timely manner. However, this goal was difficult to achieve and cumber-
some to enforce. As one participant said, “When you have 40 tutors in a [course],
there’s always going to be some that haven’t entered their data on time, no matter
how many times you drum it into them.”
Implementation challenges related to tutor responsibilities and ad hoc emailing

were further compounded by the lack of available time for coordinators to familiarize
themselves with the system. Despite these challenges, students who did receive mes-
sages were generally positive about the extra support and care. Students who were
asked to comment on whether the email was helpful or not wrote comments such as:
• “It was personal and gives me faith in the care our lecturers and coordinators
have in us. It also included additional helpful information just in case.”
• “Great appreciation to that, like a hopeless person just found a guiding star!
Thanks.”
• “It was highly encouraging and sends a positive message to the student about the
staff’s commitment to their success.”
Additionally, the use of the SRES may have contributed to changing teachers’
perspectives on student engagement, teaching design, and possibly motivate teach-
ers’ attitudes and future behaviors about student engagement and LA. For example,
in one course, the tracking of marks and attendance made the coordinator realize
that students were unconcerned about marks and attendance as long as the mini-
mum requirement was achieved so they could sit for the exam. This revelation led
the teacher to rethink his approach and use the email function in the future to help
understand why students may not come to lectures and/or tutorials.
The use of the SRES also motivated some coordinators to more deeply consider
how student engagement could be measured to improve their teaching design in the
future. For example, one participant, when asked about how the SRES impacted
them, noted that “[t]he SRES has made me think ‘wouldn’t it be good if we knew
this?’,” while another participant mentioned, “[y]ou know, you need to do some
analysis to have evidence for making a change… and you need data for that.” One
participant, who noted that they did not really utilize the platform to its fullest
potential, further mentioned, “I’d like to know more about the platform, I think we
can improve, I think we can do more.”
Teachers at the University of Melbourne adopted the platform to provide more
personal support for students and to ease the process of data collection and curation
primarily from face-to-face environments. These two purposes were interlinked,
with data availability affording targeted support; an unforeseen but encouraging
effect was that this also triggered further reflection on selecting and applying rele-
vant data to enhance learning and teaching. Despite only having used the SRES for
a short period of time, teachers started to see positive impacts in terms of workload
efficiencies and improved student feedback, although there were issues around
compliance by more diverse teaching teams.
3.4 Case Study 3: The University of New South Wales Sydney
UNSW Sydney also started piloting the SRES in early 2017. There were three ele-
ments that catalyzed its implementation here: (1) a fertile landscape incentivizing
the personalization of student experience, supported by an ambitious strategic plan
(the “2025 Strategy,” grounded on four key domains: “Communities,” “Feedback

and Dialogue,” “Inspired Learning through Inspiring Teaching,” and “Being
Digital”); (2) a certain flexibility to support innovations in learning and teaching
afforded by a strategic and systematic review of over 800 courses over 5 years; and
(3) a forward-looking team in the portfolio of the Pro-Vice-Chancellor Education,
with the expertise to support early adopters of educational technologies.
Yet, a cautious and thorough approach was developed in order to provide effective
support for teachers involved in the project. The starting point of the implementation
was the invitation of several coordinators to participate in the project, focusing on large
first-year courses. In the initial round, four large courses expressed interest, but only
two decided to continue; these two were characterized by a higher level of resilience
to uncertainty and innovation (with a potential to accept and learn from failure and
suboptimal processes): a first-year Marketing course (800+ students) and a first-year
Biology course (250+ students), both repeated over the two semesters in the year).
An important element behind the choice of courses is the belief at this university
that the focus on the first-year experience departs from the traditional transactional
model of education delivery, instead of offering a multicomponent model with mul-
tiple value creators that focus on student experience. In this sense, it is envisaged
that personalized learning pathways and communications are customized using LA
and iteratively inform learning design. This is intended to address critical concerns
that are particularly salient for first-year students, including interaction in group
work, ambiguity in communications, and assessment anxiety. By scaffolding stu-
dents in personalized ways, the strategic aim is to enhance students’ educational
experience and improve performance.
With this backdrop, the two courses adopted very different approaches: the
Biology course took a simplistic path, adding the SRES as the tool to enable more
detailed feedback after the mid-semester exams. In a sense, this established a base-
line for the implementation without disrupting the normal running of the course but
gave an opportunity for the teachers to identify data to offer students a more detailed
account of their performance which they would not normally get for exams. In this
case, only the two course coordinators were involved in the process: they negotiated
the scope of the implementation and selected the metrics of interest, keeping in
focus only the provision of better feedback to students.
The personalized report received in students’ inboxes after the mid-semester
exams gave specific details about the areas requiring improvement and additional
targeted resources, enabling them to adjust their modes of learning. The student
response (in semester one 2017, emails were sent to 1005 students; 81.4% opened
the emails with some up to 30 times) was overwhelmingly positive with 99.8% of
the students indicating via a survey link at the bottom of the email that it was help-
ful. Similar results were obtained in the second mailing. The following comments
exemplify their views:
• “This email was helpful in highlighting specific areas of weakness and will allow
me to fill in gaps in my knowledge!”
• “It told me exactly where I went wrong, now I can improve in that area, thanks
[teacher name]!”
• “The feedback was detailed and constructive- advised on what areas could be
improved on instead of a generic feedback relating to the entire cohort.”
• “Thank you for providing me feedback on the areas that I am weak on, please
continue to do this. I will use this to revise and improve in these areas.”
The coordinators were surprised by the response from students and were quickly
convinced about the effectiveness of the approach: “I could not believe that students
would open and go back to the email 30 times! …even if the amount of feedback
provided is limited, the students are appreciating the fact that the message is directed
to them.”
The Marketing course adopted a more holistic and systematic approach, integrat-
ing several tools in the course including the use of an external resource from the
textbook publisher (McGraw-Hill Connect and LearnSmart), a unique approach
focusing on individual characteristics for personal development and team forma-
tion, and the SRES as an essential component to provide logistic support in the
collection and curation of key behavioral and performance attributes during the
course (including attendance, class participation, and team presentation outcomes).
Although the main focus of the integration and adoption of the SRES was an admin-
istrative one, because of the nature of the discipline (marketing), the coordinator
was convinced about the potential benefits of the SRES for the running of the course
and for the use of data. In this case, all the tutors as well as the course coordinator
contributed by using the SRES in their daily activities. For example, all teachers
logged attendance and class participation in the same place via the SRES web app,
saving much time from manually aggregating separate spreadsheets.
In the first run of the course using the SRES, the coordinator praised the simplic-
ity of being able to visualize a snapshot of what happened in the course by the end
of each week. Using the SRES visualization tools, the coordinator could easily gen-
erate a real-time report. Further, the fact that information about engagement with
the external tool was brought back into the SRES meant that she could also appreci-
ate how students valued the resource. The ability to see what students and tutors
were doing in near real time also meant that she started to question the importance
of attendance at lectures and of engagement with the ecosystem in the course. This
sort of reasoning, partly prompted by the disciplinary context, reflects the effective-
ness of marketing channels in the consumer journey to purchase and draws a paral-
lel to the student journey (Bucic, Vigentini, & King, 2018). This thinking drew the
course team to experiment with the modes and level of “nudging” (i.e., the fre-
quency and timing of messages) in order to test whether there was a perceived dif-
ference in “teacher’s presence.” Comparing the way in which messages were sent
(high frequency, about once per week in semester one vs low frequency, at the start
and around key assessment points in semester two) showed that students receiving
more frequent and consistent relevant messages rated their satisfaction with the
course to be much higher than when they were just prompted occasionally.
Both cases provided strong evidence that students appreciated the teacher’s
presence or simply the fact that their teachers cared about them. This was associ-
ated with higher satisfaction with the courses and, at least in the Marketing course,
was also associated with an improvement in performance compared with the pre-
vious instances of the course without the SRES. Combined with the ability to col-
lect and curate face-to-face and electronic data (e.g., assessment outcomes,
attendance, participation, and online tool use) as well as visualize and act on this
data, all in a single platform, this helped teachers overcome resistance to change.
As seen in the other case studies, teacher engagement with the SRES also enabled
some reflective practice.
4 Discussion
4.1 E
mpowering Teachers to Personalize Support for Student
Success
The work presented here sought to investigate one LA solution to a sector-wide

issue: maintaining the personalization of higher education in the face of massifica-
tion, commodification, and diversification. As student numbers grow, along with
tuition prices, and emerging new cohorts of students, higher education is pressured
to find new ways to support student engagement and success. The platform dis-
cussed in this chapter, the SRES, seeks to enable teachers to provide personalized
and timely support and feedback to students which would not be feasible at scale
using traditional approaches.
The case studies coordinated across three very different institutions provided
systematic data on the adoption, implementation, and use of the SRES, showing
how the platform offered teachers the ability to collect, curate, analyze, and act
upon data that was meaningful to their specific teaching context, as well as close the
loop for reflection on changes to practice. To teachers, the two most important dif-
ferences between the SRES platform and other extant LA tools have been (1) the
ability to precisely select and use data that is relevant to them and (2) being able to
efficiently perform operations at scale on the data from a single software platform.
Many teachers at the three universities placed strong emphasis on collecting atten-
dance and assessment data, and many also used the SRES to curate data from other
sources including online systems and richer metrics from face-to-face classroom
interactions. The unique web interface also afforded both students and teachers the
ability to input information directly into the SRES, such as allowing students to
enter information about themselves and teachers to efficiently enter attendance, par-
ticipation, grades, and feedback.
In contrast to existing (often manual) practices, the SRES has empowered teach-
ers to engage in more systematic and targeted support actions throughout the semes-
ter. The most common actions from the platform have been to personalize messages
to students for a wide range of purposes including offering support to students con-
sidered “at risk,” reminding students of attendance and assessment requirements,
providing tailored feedback, and alerting high achievers to the possibility of
advanced study. Other actions have included customizing face-to-face teaching and
learning activities to the backgrounds and interests of learners by leveraging data

curated in the SRES. Interestingly, many teachers commented that their workload
when using the platform was not necessarily reduced (at least not at first), although
their time was more “productive”; this has helped to alleviate a key barrier to adop-
tion (Macfadyen & Dawson, 2012).
Another contributing factor to overcoming change resistance has been the posi-
tive outcomes for both teachers and students from the personalized, person-alized,
and timely nature of actions taken by teachers empowered by the SRES. These out-
comes included more open channels of communication, increased help-seeking
behaviors from students, feedback that allowed students to improve performance on
subsequent assessments, and increased retention rates. The impact on student satis-
faction has been reflected in the positive feedback teachers have received from their
students. Although the diversity of ways in which this LA platform was used across
the three institutions precluded a typical quantitative impact study, the mixed-
methods data including teacher interview responses and student perception data
together suggested that the SRES positively impacted students’ outlook on the level
of personal support provided for them, and perhaps even their engagement with the
material and eventual academic performance. The SRES has therefore helped to
enhance teachers’ “relational pedagogy” (Pearce & Down, 2011) to promote stu-
dent success. Of course, effect is hard to generalize as the use of the platform was
sometimes part of a range of changes made to courses by teachers, and the cohorts
were different. At the same time, the case studies started to show that the platform
not only provided an opportunity to enhance the student experience, but given inter-
est, time, and effort from teachers, there is great opportunity to delve into action
research of how students learn and engage with support.
4.2 Implications for Practice
This chapter set out to investigate the experiences of teachers at three Australian
universities implementing a humanistic LA platform. A number of ideas from the
LMS implementation and adoption literature are instructive here to help frame
implications for LA practice.
First, it is important to keep in mind that learning technologies (including LA) are
not neutral technologies but rather can impact teacher’s expectations, desires, behav-
iors, and, thus, their teaching design (Coates et al., 2005). The technology itself has
a powerful role in influencing and shaping teaching practices. Second, even though
an LMS may provide various functions for enhancing online learning and teaching
beyond the transmission of textual content, the way teachers use the technology may
be mismatched with students’ expectations or needs (Lonn & Teasley, 2009). Third,
these varied functions open the possibility for teachers to reconsider their practices
as their use of (and comfort with) the technology progressively evolves (West,
Waddoups, & Graham, 2007). Using these three perspectives in the context of the
case studies, we highlight three general implications for LA practice.
4.2.1 Learning Analytics Needs to Address Actual Needs
Adoption of the SRES has spread throughout the University of Sydney where it was
developed, to the University of Melbourne, UNSW Sydney, and to teachers in other
Australian institutions. In contrast to many top-down implementations of LA, the
bottom-up nature of the SRES has assisted with its widespread adoption primarily
due to being designed by teachers for teachers. By helping teachers to collect and
use meaningful data relevant to their context to provide timely learning support to
students, the SRES addresses pedagogical and pastoral needs of personalization and
relationship building (Kahu & Nelson, 2018; Zepke & Leach, 2010) and removes
some of the usual barriers to LA adoption such as one-size-fits-all approaches,
opaque predictive algorithms, and a disconnect between analysis and action (Liu,
Bartimote-Aufflick, Pardo, & Bridgeman, 2017). Other barriers removed have led
to greater veracity and workload efficiency (Macfadyen & Dawson, 2012) in data
entry, analysis, and communication with students. It has not only empowered teach-
ers with the capability to increase the number and quality of exchanges with their
students (West et al., 2015), but more importantly it has allowed them to humanize
these exchanges and also support nonelectronic interactions. This has helped to
mitigate the sense of anonymity (Bryson & Hand, 2007) that is associated with large
cohorts of students. At the same time, the SRES has streamlined some of the most
burdensome administrative aspects of unit coordination, allowing teachers to focus
more attention on pedagogical and pastoral care for students.
Together, these contribute to the critical “usefulness” factor highlighted in the
literature on LA (Ali, Asadi, Gašević, Jovanović, & Hatala, 2013) and LMS (West
et al., 2007) adoption and address the need for innovations to present a “relative
advantage” to existing approaches as argued by Rogers (2003). In the face of insti-
tutional culture and workload pressures contributing to the lack of LA adoption by
teachers, being able to demonstrate the relative advantage of LA and its compatibil-
ity with their needs is crucial (Macfadyen & Dawson, 2012). Bringing data and a
range of tools together into one platform for academics who are interested in their
students empowers them to reflect on how their practice affects students but also
enables them to reflect on the effectiveness of their practice.
4.2.2 Start Small but Provide for Growth
Addressing elements of teachers’ immediate perceived needs helps to underpin

strong teacher buy-in. However, it is also important to negotiate the implementation
space to ensure that the technology does not get in their way but rather blends in
with their practice facilitating the running of courses. The case studies suggested
that teachers’ contexts and approaches were nuanced, needing an LA system that
was customizable and flexible and could support multiple learning designs and
teacher perspectives. For example, while the SRES platform is capable of quite
complex data collection and analysis, many teachers included within the study
found its most simple functions to be the most meaningful for them: collecting
attendance and sending personalized messages. This is not to say that these are
purely perfunctory; indeed, many studies have suggested a close association
between these and student success (Credé et al., 2010; Newman-Ford et al., 2008;
Pearce & Down, 2011). Also, this does not indicate that the other functions of the
platform are not useful or meaningful, although it does affirm that the system design
needed to include both simple and advanced functionality to allow teachers to apply
functions as they chose and felt comfortable with. This concords with LMS adop-
tion literature where instructors would start to use simple features that addressed
their immediate goals, and then experiment with other functionality as they grew
more comfortable with, and confident in, the platform (West et al., 2007).
A risk here is that teachers will not progress from rudimentary applications of the
technology, and the technology becomes a shackle rather than an enabler. In the
LMS space, this may present as teachers continuing to use the largely textual plat-
form to didactically transmit written content to students and codify learning in terms
of achievement in preprogrammed positivist quizzes (Coates et al., 2005). This
potential shackling was clear in some interviews where, although teachers had every
intention to explore richer functionality beyond attendance tracking, they did not
have the workload capacity or the knowledge to be able to experiment in this way.
Part of this involves teachers overcoming the initial learning curve with all technol-
ogy such that this investment can pay time dividends and permit further experimen-
tation. Another part is gaining an understanding of how the technology may be
better applied. When combined with initial rudimentary applications, this may be
sufficient to drive teachers to richer uses; as one interviewee noted, this is a “…foot
in the door. Because once you realize that you can efficiently keep track of people
and just use it almost as an administrative tool, then you start to see what else can
be done with it… Once I see that it can do that, then oh, I can also – and then you
start to get into the more sophisticated functions. There’s that classic thing about
any piece of software and the user interface – you want it to be accessible to your
new user, to your basic level.” The same interviewee emphasized that “sharing suc-
cess stories” as part of his community of colleagues was an integral part of this, as
discussed in the next section.
4.2.3 Foster Communities
Combining top-down support with bottom-up adoption may assist to expand the
uptake of LA throughout higher education institutions. In our case studies, teachers
were empowered by having the customizable platform, although there was scope to
further empower them by fostering communities of practice through which they
could share different ways in which the SRES might helpfully personalize support
and feedback. Learning by members of an organization is often informal, relying on
colleagues who are geographically close or in similar roles (Boud & Middleton,
2003). This is particularly true for university teachers, who primarily rely on infor-
mal conversations with peers to grumble about teaching issues and share solutions
for improving teaching practices (Thomson, 2015). As there is a strong tendency for
teachers to interact and share ideas just with others who are similar to them, top-
down support may provide the opportunity to foster a more heterogeneous commu-
nity of users. As the type of user expands from innovators to early adopters and the
use of the SRES moves from ad hoc trials to more strategic and systematic use, the
need for support is also likely to expand. Communities of practice that form around
interested teachers and support staff can then aid the sharing of information and the
spread of innovation (Wenger, 1998). At the University of Sydney, these communi-
ties are starting to form within faculties where early adopter teachers and learning
designers are promoting the platform, training their peers, and self-organizing sup-
port from the central learning and teaching unit. At UNSW Sydney, both course
coordinators were awarded teaching excellence awards from their respective facul-
ties for the ability to lead teaching in their disciplines and experimenting with digi-
tal tools capable of improving the student learning experience; this serves to raise
the profile of such innovations and pique colleagues’ interest.
The importance of a learning and teaching support unit (either within a faculty or
centrally) was an understated feature in all three case studies. In the three institu-
tions, this unit variably comprised staff with learning design, educational technol-
ogy, and/or software development expertise, which was crucial in supporting
academics in using the platform (Vigentini et al., 2017). In some cases, these units
were also instrumental in connecting various data sources to the SRES so that rele-
vant student data was available. Because these institutions shared a common open-
source codebase for the platform, the designers and developers in these units formed
an informal cross-institutional community of practice. This allowed not only the
sharing of practice but also the development of new approaches and software func-
tionality, which in turn benefited all involved.
4.3 Conclusion and Future Directions
Traditional LA, with its focus on single-purpose tools such as dashboards, visual-
izations, or mail merge, may not only stifle the richness and depth of support and
relationship building that is integral to effective teaching but may also inadvertently
suppress the development of teachers’ collection and application of student data.
Although the SRES is a more holistic platform, it is not immune: a fixation on cap-
turing and tracking attendance (even though it may be pedagogically and contextu-
ally meaningful) may limit teachers’ conceptualization and the use of student data.
However, at least at the University of Sydney where SRES adoption is more wide-
spread, we are observing a subtle progression from rudimentary to richer applica-
tions, which has been afforded by the flexibility of the platform and communities of
teachers sharing success stories. In future work, we seek to analyze this progression
of sophistication, consider how teachers’ aspirations compare with their actual
usage, and determine the factors that lead to evolving uses of student data by teach-
ers to continue to personalize the learning experience.
Personalized messages were one of the intermediary steps in this progression of

complexity, requiring teachers to apply data to tailor this form of support and feed-
back, which itself required teachers to collect and curate the right data. Previous
research has suggested that there may be some discrepancies between how teachers
compose message-based support and what is impactful for students in terms of con-
tent (e.g., motivational vs informational, summative vs formative) and nature (e.g.,
tone and orientation toward performance or outcomes) (Tanes et al., 2011). With the
SRES allowing teachers to be more nuanced and granular in the triggers and content
for each personalized message, it will be interesting to explore these in terms of
their nature and content, and the extent to which they are being personalized. Some
future analyses will also explore the interaction between students’ personal charac-
teristics (e.g., personality, emotional intelligence, and learning approaches) and
behavioral observations (from both engagement and performance) with the mes-
sages received.
Beyond direct student-facing impacts such as personalized electronic communi-
cation and data-augmented face-to-face interactions, our interviews also revealed
how using the SRES to enhance student engagement and success could prompt
teachers to reconsider their broader teaching approaches and learning design. This
seemed to be related to their use of the platform indirectly increasing their aware-
ness of the measures of student engagement and success, and the implications for
their existing practices. At the micro level, the affordances of the technology have
led to changes in how teachers interact with students during face-to-face classes,
nurturing positive teacher-student relationships. These impacts warrant further
investigation, such as identifying archetypes of users and uses, investigating how
teachers’ own learning may be associated with each of these, and examining appro-
priate ways to measure impact that are specific to the type of use. Further studies are
also planned at the course level to identify the impact of specific SRES affordances,
such as the impact on student belonging of using the SRES web app to capture stu-
dent attendance. After all, LA as a human activity is intensely contextualized, and
its ultimate goal is to optimize learning and the environments in which it occurs,
through empowering teachers’ human judgment (Siemens & Baker, 2012).
Acknowledgments The authors wish to thank the designers, developers, and directors who help
to build and assist academics with the SRES, including but certainly not limited to Kevin Samnick,
Melissa Makin, Joshua Lilly, Melanie Keep, Adam Bridgeman, Ruth Weeks, and Uli Felzmann.
References
Ali, L., Asadi, M., Gašević, D., Jovanović, J., & Hatala, M. (2013). Factors influencing beliefs
for adoption of a learning analytics tool: An empirical study. Computers & Education, 62,
130–148.
Alli, N., Rajan, R., & Ratliff, G. (2016). How personalized learning unlocks student success.
Educause Review Online, March/April, 12–21.
Baker, R. S. (2016). Stupid tutoring systems, intelligent humans. International Journal of Artificial
Intelligence in Education, 24(4), 470–497.
Boud, D., & Middleton, H. (2003). Learning from others at work: Communities of practice and
informal learning. Journal of Workplace Learning, 15(5), 194–202.
Bryson, C., & Hand, L. (2007). The role of engagement in inspiring teaching and learn-
ing. Innovations in Education and Teaching International, 44(4), 349–362. https://doi.
org/10.1080/14703290701602748
Bucic, T., Vigentini, L., & King, M. E. (2018). Disruptive innovation in the marketing class-
room for a personalized learning journey. In Paper presented at the Marketing Management
Association Fall Educators’ Conference, Kansas City, MO.
Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning man-
agement systems on university teaching and learning. Tertiary Education and Management,
11(1), 19–36.
Colvin, C., Rogers, T., Wade, A., Dawson, S., Gašević, D., Buckingham Shum, S., … Fisher,
J. (2016). Student retention and learning analytics: A snapshot of Australian practices and a
framework for advancement. Canberra, Australia: Australian Government Office for Learning
and Teaching.
Credé, M., Roch, S. G., & Kieszczynka, U. M. (2010). Class attendance in college: A meta-analytic
review of the relationship of class attendance with grades and student characteristics. Review of
Educational Research, 80(2), 272–295.
Dawson, S., Jovanovic, J., Gašević, D., & Pardo, A. (2017). From prediction to impact: Evaluation
of a learning analytics retention program. In Paper presented at the Proceedings of the Seventh
International Learning Analytics & Knowledge Conference, Vancouver, British Columbia,
Canada.
Farr-Wharton, B., Charles, M. B., Keast, R., Woolcott, G., & Chamberlain, D. (2018). Why lectur-
ers still matter: The impact of lecturer-student exchange on student engagement and intention
to leave university prematurely. Higher Education, 75(1), 167–185. https://doi.org/10.1007/
s10734-017-0190-5
Ferguson, R., Clow, D., Macfadyen, L., Essa, A., Dawson, S., & Alexander, S. (2014). Setting learn-
ing analytics in context: Overcoming the barriers to large-scale adoption. In Paper presented at
the Fourth International Conference on Learning Analytics and Knowledge, Indianapolis, IN.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1),
81–112.
Jacques, P. H., Garger, J., Thomas, M., & Vracheva, V. (2012). Effects of early Leader–Member
Exchange perceptions on academic outcomes. Learning Environments Research, 15(1), 1–15.
https://doi.org/10.1007/s10984-012-9100-z
Kahu, E. R., & Nelson, K. (2018). Student engagement in the educational interface: Understanding
the mechanisms of student success. Higher Education Research and Development, 37(1),
58–71. https://doi.org/10.1080/07294360.2017.1344197
Krause, K. L., & Coates, H. (2008). Students’ engagement in first-year university. Assessment &
Evaluation in Higher Education, 33(5), 493–505.
Kuh, G. D., Kinzie, J. L., Buckley, J. A., Bridges, B. K., & Hayek, J. C. (2006). What matters to
student success: A review of the literature (Vol. 8). Washington, DC: National Postsecondary
Education Cooperative.
Lawson, C., Beer, C., Rossi, D., Moore, T., & Fleming, J. (2016). Identification of ‘at risk’students
using learning analytics: The ethical dilemmas of intervention strategies in a higher education
institution. Educational Technology Research and Development, 64(5), 957–968.
Liu, D. Y. T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2017). Data-driven person-
alization of student learning support in higher education. In Learning analytics: Fundaments,
applications, and trends (pp. 143–169). Zürich, Switzerland: Springer.
Liu, D. Y. T., Rogers, T., & Pardo, A. (2015). Learning analytics - are we at risk of missing the
point? In Paper presented at the 32nd ASCILITE Conference, Perth, WA.
Liu, D. Y. T., Taylor, C. E., Bridgeman, A. J., Bartimote-Aufflick, K., & Pardo, A. (2016).
Empowering instructors through customizable collection and analyses of actionable informa-
tion. In Paper presented at the 1st Learning Analytics for Curriculum and Program Quality
Improvement Workshop.
Lizzio, A. (2006). Five senses of success: Designing effective orientation and engagement pro-
cesses. Griffith University First Year Experience Project. Griffith University. Retrieved
from www.griffith.edu.au/__data/assets/pdf_file/0008/51875/Alfs-5-Senors-Paper-FYE-
Project-2006.pdf
Lonn, S., & Teasley, S. D. (2009). Saving time or innovating practice: Investigating perceptions
and uses of Learning Management Systems. Computers & Education, 53(3), 686–694. https://
doi.org/10.1016/j.compedu.2009.04.008
Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed
to inform an institutional strategic plan. Journal of Educational Technology & Society, 15(3),
149–163.
Newman-Ford, L., Fitzgibbon, K., Lloyd, S., & Thomas, S. (2008). A large-scale investigation into
the relationship between attendance and attainment: A study using an innovative, electronic
attendance monitoring system. Studies in Higher Education, 33(6), 699–717.
Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning:
A model and seven principles of good feedback practice. Studies in Higher Education, 31(2),
199–218. https://doi.org/10.1080/03075070600572090
Pardo, A. (2017). A feedback model for data-rich learning experiences. Assessment & Evaluation
in Higher Education, 43(3), 1–11. https://doi.org/10.1080/02602938.2017.1356905
Pardo, A., & Dawson, S. (2016). Learning analytics: How can data be used to improve learn-
ing practice? In P. Reimann, S. Bull, M. Kickmeier-Rust, R. Vatrapu, & B. Wasson (Eds.),
Measuring and visualizing learning in the information-rich classroom (pp. 41–55). New York,
NY: Routledge.
Pearce, J., & Down, B. (2011). Relational pedagogy for student engagement and success at uni-
versity. The Australian Educational Researcher, 38(4), 483–494. https://doi.org/10.1007/
s13384-011-0037-5
Reason, R. D., Terenzini, P. T., & Domingo, R. J. (2006). First things first: Developing academic
competence in the first year of college. Research in Higher Education, 47(2), 149–175.
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York, NY: Free Press.
Siemens, G., & Baker, R. S. (2012). Learning analytics and educational data mining: Towards
communication and collaboration. In Paper presented at the 2nd International Conference on
Learning Analytics and Knowledge, Vancouver, Canada.
Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using signals for appropriate feed-
back: Perceptions and practices. Computers & Education, 57(4), 2414–2422.
Thomson, K. (2015). Informal conversations about teaching and their relationship to a for-
mal development program: Learning opportunities for novice and mid-career academics.
International Journal for Academic Development, 20(2), 137–149. https://doi.org/10.1080/1
360144X.2015.1028066
Tinto, V. (2006). Research and practice of student retention: What next? Journal of College Student
Retention: Research, Theory and Practice, 8(1), 1–19.
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dash-
board applications. American Behavioral Scientist, 57(10), 1500–1509.
Vigentini, L., Kondo, E., Samnick, K., Liu, D. Y. T., King, D., & Bridgeman, A. J. (2017). Recipes
for institutional adoption of a teacher-driven learning analytics tool: Case studies from three
Australian universities. In Paper presented at the ASCILITE 2017.
Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge,
England: Cambridge University Press.
West, D., Huijser, H., Lizzio, A., Toohey, D., Miles, C., Searle, B., & Bronnimann, J. (2015).
Learning analytics: Assisting universities with student retention, final report (part 1). Australian
Government Office for Learning and Teaching.
West, R. E., Waddoups, G., & Graham, C. R. (2007). Understanding the experiences of instruc-
tors as they adopt a course management system. Educational Technology Research and
Development, 55(1), 1–26.
Zepke, N., & Leach, L. (2010). Improving student engagement: Ten proposals for action. Active
Learning in Higher Education, 11(3), 167–177.
Chapter 14
Predicting Success, Preventing Failure
Using Learning Analytics to Examine the Strongest
Predictors of Persistence and Performance in an Online
English Language Course
Danny Glick, Anat Cohen, Eitan Festinger, Di Xu, Qiujie Li,

and Mark Warschauer
1 Introduction
Given the increasing recognition of English as the lingua franca for business and
communication in the global economy, a growing number of developing countries
are making significant investments to improve English language learning (ELL).
Yet, often constrained by both economic and human capital resources, national
efforts to develop students’ English language proficiency in developing countries
often remain ineffective. Typical challenges include a shortage of qualified
English teachers, lack of teacher certification or training, limited access to high-
quality language learning resources, and poorly designed English curricula
(Ministério de Educación Perú, 2016a; OECD, 2015). While policy frameworks
in many Latin-American countries (e.g., Peru, Ecuador, Costa Rica, and Mexico)
require teachers to have an English proficiency level of B2 (upper intermediate) or
better on the CEFR1 (Common European Framework of Reference for Language)
scale, many English teachers perform well below these standards (see, e.g.,
Cronquist & Fiszbein, 2017).
1
The CEFR is an international standard for describing language ability ranging from A1 (basic)
up to C2 (proficient).
D. Glick (*)
Edusoft, a Subsidiary of ETS, and UC Irvine’s Digital Learning Lab,
University of California, Irvine, CA, USA
A. Cohen · E. Festinger
Tel-Aviv University, Tel Aviv, Israel
D. Xu · Q. Li · M. Warschauer
University of California, Irvine, CA, USA

https://doi.org/10.1007/978-3-319-64792-0_14
250 D. Glick et al.
In view of these challenges, online language learning has increasingly been

viewed as a possible way to remove the barriers associated with traditional English
language learning by providing low-cost, quality education (e.g., Glick et al., 2016;
Sife, Lwoga, & Sanga, 2007). Recent studies suggest that technology-mediated lan-
guage learning implemented in developing countries has a significant, positive
impact on students’ course grades and course completion rates compared to tradi-
tional face-to-face instruction (Bai, Mo, Zhang, Boswell, & Rozelle, 2016;
Banditvilai, 2016; Glick et al., 2016; Rios & Cabrera, 2008). While online learning
is a promising approach to deliver more desirable learning outcomes, many students
struggle in such settings, especially students lacking certain personal attributes –
such as being goal-oriented and self-disciplined—that are essential for successful
online study (Appana, 2008). Therefore, to improve persistence and performance in
online courses, it is important to have a better understanding of student online learn-
ing behavior and student engagement in online courses.
This study employs a learning analytics approach framed in Ryan and Deci self-
determination theory (SDT) (1985, 2000) to examine the strongest predictors of
persistence and performance in an online English language course. Deci and Ryan’s
SDT argues that all humans have intrinsic needs to be self-determining or
autonomous (i.e., experiencing a sense of agency or control), as well as to feel
competent and connected in relation to their environment. SDT states that
environmental conditions that support perceptions of autonomy and social
relatedness improve motivation, thereby positively influencing learning outcomes.
Thus, the purpose of this study is to investigate how autonomy, competence, and
relatedness, as defined by Deci and Ryan, may predict student persistence and
performance in an online English language course by analyzing a large dataset
collected from a pre-course student readiness survey, the course learning
management system (LMS) log files, as well as data collected from the course
Facebook page.
Previous research with regard to student persistence and performance in tradi-
tional learning environments has found that learner autonomy, competence, and
sense of relatedness have a positive effect on student performance and persistence
(e.g., Black & Deci, 2000; Vansteenkiste, Lens, & Deci, 2006). However, research
examining the SDT with regard to students’ persistence and performance in online
courses is scarce.
Based on a sample of 716 Peruvian university students enrolled in an online
English language course, we present evidence on the effects of an online ELL
course on university students’ learning outcomes. The aim of this study was to use
learning analytics framed in SDT to examine the strongest predictors of persistence
and performance in an online ELL course.
The results of our study provide important implications for policymakers, online
teachers, administrators, and instructional designers in the postsecondary arena in
developing countries as they consider whether and how to implement online learn-
ing to better reach their goals of promoting English language proficiency.
14 Predicting Success, Preventing Failure 251
2 Research Background
2.1 Challenges in Peru’s English Language Education
Peru has made considerable efforts to improve English language learning through
policies and programs, resulting in more students having access to ELL. However,
despite considerable efforts and investments in English language education, the
quality of Peru’s English language system remains low. The English First English
Proficiency Index (EF EPI), which ranks 72 countries and territories based on
online English proficiency tests taken by more than 950,000 test takers, put Peru
in the bottom half – ranking it 45th out of 72 countries (EF English Proficiency
Index, 2016).
One major barrier to improving ELL is the quality of Peru’s English teachers.
A UNESCO study assessing the proficiency of 3356 English teachers in Peru
found that about 33% of the teachers have an English proficiency level of A1 (low
beginner) on the CEFR (Ministério de Educación Perú, 2016b). Another area of
concern is the lack of accreditation of Peru’s pre-service English teaching programs.
While there are many options throughout Peru to pursue an English teaching
degree, a growing number of nonaccredited programs continue to enroll and train
pre-service teachers. Specifically, only 34 out of 285 pedagogical institutes of
higher education (11.9%) that offer English teaching programs were accredited in
2016 (Cronquist & Fiszbein, 2017).
Recent reports show that a serious shortage of high-quality English teachers in
Peru (Cronquist & Fiszbein, 2017) constitutes a serious bottleneck for ELL programs.
Even more worrying is the fact that Peru’s demand for English teachers will continue
to grow over the next 5 years. English is one of the subject areas receiving more time
in the new school day in Peru (Ministério de Educación Perú, 2017), which has
increased the need for qualified English teachers. The Ministry of Education, in its
national English plan, aims to recruit and train over 30,000 additional English
teachers over a period of 8 years (Ministério de Educación Perú, 2016b), a goal
which seems very ambitious for a country where nearly 90% of its pre-service
training programs failed to gain accreditation in 2016. Given the fact that nearly two-
thirds of Peru’s teachers are unlicensed or have low English proficiency levels, online
learning programs may provide an attractive alternative.
2.2 E
nhancing English Instruction in Developing Countries
Using Online Courses
Online learning is a growing trend and derived from efficiency, economic, organiza-
tional, pedagogical, and operational considerations (Bakia, Shear, Toyama, &
Lasseter, 2012; Lee, 2016; Massengale & Vasquez, 2016). Online courses provide
access to a wide range of audiences and, in some cases, improve teaching and
252 D. Glick et al.
learning processes (Macfadyen & Dawson, 2010; Roby, Ashe, Singh, & Clark, 2013).
They may offer an enjoyable and effective learning environment if they emphasize
solid content, interaction, and clear structure (Driscoll et al. 2012). Studies have
found that students perceive the ease of access to varied, high-quality, and up-to-
date learning materials as benefiting their learning in an online course (Palmer &
Holt, 2010). In addition, integrating social networks can contribute to a sense of
community among learners and develop pedagogical values (Erdem & Kibar, 2014)
as well as reduce loneliness and motivate students to persevere in their studies (Yuan
& Kim, 2014). Furthermore, online courses provide more autonomy and enable
students to learn at their own pace, as they allow flexibility of time and location
(Lim, 2016; Rodriguez, Rooms, & Montañez, 2008).
Prior research has shown that online learning is particularly beneficial to learners
in developing countries where language learning resources are limited. In a large-
scale study in Mexico, it was found that, compared to traditional face-to-face
instruction, blended learning has a significant, positive impact on students’ course
grades and course completion rates (Glick et al., 2016). Rios and Cabrera (2008)
found that students who used an online environment improved their language skills
more than the students who learned face to face. Bañados (2006) found similar find-
ings with Chilean students who significantly improved their speaking, listening,
pronunciation, grammar, and vocabulary. Similarly, Iranian students who learned in
online environments outperformed students taking the same course face to face
(Barani, 2011; Marzban, 2011; Vahdat & Eidipour, 2016).
Online courses offer many benefits for students (Baker & Yacef, 2009; Cohen &
Nachmias, 2006; Zakrzewska, 2009); however, there is growing concern regarding
the persistence and engagement of students in such courses, as well as with the high
dropout rates compared to face-to-face courses (Cheng, Kulkarni, & Klemmer,
2013; Clay, Rowland, & Packard, 2009; Levy, 2007; Nistor & Neubauer, 2010;
Otter et al., 2013; Park & Choi, 2009; Willging & Johnson, 2009). Moreover, previ-
ous studies indicate that lack of persistence which is reflected in low engagement
and poor self-regulation is an important factor leading to attrition among students in
online courses and inadequate academic achievements (Angelino, Williams, &
Natvig, 2007; Otter et al., 2013; You, 2016). Learning analytics, therefore, seems to
have the potential to overcome the abovementioned challenges associated with
online learning by understanding the learners’ behavior during the learning pro-
cesses (Gašević, Dawson, & Siemens, 2015).
2.3 L
earning Analytics for Predicting Success in Online
Courses
The development of online courses has led to enormous amounts of information

regarding the learning process. Students’ interactions with online learning activities
as well as with their peers and instructors are captured automatically in the log file
records of the course LMS (Levi-Gamlieli, Cohen, & Nachmias, 2015), without
interfering with the learning process (Cohen & Nachmias, 2012). These log data can
be retrieved and analyzed in order to identify patterns of learning behavior and
provide insights into education practice. This process of mining and analyzing data
on learning has been described as learning analytics (Gašević et al., 2015). Siemens
and Gašević (2012, p. 1) defined the study of learning analytics as the “measurement,
collection, analysis and reporting of data about learners and their contexts, for
purposes of understanding and optimizing learning and the environments in which
it occurs.”
Learning analytics has been used to study teaching and learning processes
(Gašević, Dawson, Rogers, & Gasevic, 2016); diversity among learners and usage
patterns, which could be used to predict achievement (Ai & Laffey, 2007; Lu, Yu, &
Liu, 2003; Romero & Ventura, 2007); the utilization of information accumulated in
log files to identify students at risk of dropping out with low achievement (Santana,
Costa, Neto, Silva, & Rego, 2015); and the provision of identification tools for
instructors (Cohen, 2017). In addition, intervention programs (Campbell et al.,
2007; Dietz-Uhler & Hurn, 2013; Romero, López, Luna, & Ventura, 2013) and
warning systems to identify at-risk students have been developed using learning
analytics (Lykourentzou, Giannoukos, Nikolopoulos, Mpardis, & Loumos, 2009;
Macfadyen & Dawson, 2010).
Predicting student success or failure is a challenge due to the many factors that
may affect student performance, including demographics, pace of progress, amount
of content viewed, activities in discussion groups, and task performance and achieve-
ments (Johnson, Becker, Estrada, & Freeman, 2015). Some claim that learning
behaviors that reflect self-regulation are the primary predictors of student success or
failure in courses. You (2016) found that students’ regular study, late submissions of
assignments, number of sessions (the frequency of course logins), and evidence of
reading the course materials significantly predicted their course achievement. In
addition to self-regulation, collaborative learning activities have been found to be a
relevant predictor of student performance as well. For example, the cooperative
activity level of students in the first weeks of a course was found to predict course
drop out (Nistor & Neubauer, 2010). Gašević et al., (2016) found that the number of
logins and the number of operations performed in discussion forums and resources
were significant predictors of academic performance.
The literature shows that predicting student learning success has been one of the
most frequent tasks associated with learning analytics (Dawson, Gašević, Siemens,
& Joksimovic, 2014). However, there remains a significant absence of theory in the
research literature that focuses on LMS variables as key indicators of interaction
and success (Gašević et al., 2016). Furthermore, learning analytics needs to build on
and connect better with the existing body of learning and teaching research knowl-
edge (Gašević et al., 2015). In this study, we address these issues, and in what
appears to be a first attempt to frame data analytics methods in a motivational theory
in an ELL context, we link learning analytics (based on observed behavior in the log
files) to a theoretical model based on the motivation to learn (SDT). In addition, a
holistic approach that combines data collected from three different sources—a
student readiness survey, LMS log files, and Facebook logs – is applied.
254 D. Glick et al.
2.4 Self-Determination Theory
As in traditional learning environments, in online courses, student engagement and

persistence are strong predictors of learning outcomes (Campbell et al., 2007;
Goldstein & Katz, 2005; Mandernach, 2009). One of the main factors influencing
persistence and success in a course, whether it is conducted face to face or online, is
student motivation (Allen & Bir, 2012; Hartnett, George, & Dron, 2011; Järvelä,
Volet, & Järvenoja, 2010; Miltiadou & Savenye, 2003; Muilenburg & Berge, 2005).
Motivation is a central factor in learning. Students are motivated by past and cur-
rent reinforcements, which are related, among others, to purpose, self-efficacy, val-
ues, and areas of interest (Schunk, 2012). Motivation is a presumed internal force
that stimulates an action and determines its direction and influences what we learn,
how we learn, and when we choose to learn (Schunk, 2012). According to SDT, a
learner’s motivation is determined by the degree of responsiveness to three basic
needs: autonomy (the degree of learner control over the learning process),
competence (sense of ability and efficacy), and relatedness (the perception of social
belonging).
Autonomy Autonomy is the feeling of choice or the sense of control that learners
have over the learning process (Ryan & Deci, 2000). The autonomous learner is
self-regulating (Driscoll et al., 2012), takes responsibility for his/her learning, and
defines the goals, content, and pace of his/her learning (Holec, 1981). Active and
autonomous involvement of students in their own learning increases motivation to
learn and improves learning effectiveness (Dickinson, 1995) and thus can have a
positive impact on success and achievement (Giesbers, Rienties, Tempelaar, &
Gijselaers, 2013). In an online learning course, learners have freedom to determine
their learning pace and pathway (Rienties, Tempelaar, Van den Bossche, Gijselaers,
& Segers, 2009) by utilizing a range of learning strategies such as time management
and goal setting. The literature shows that the ability to manage one’s time, to work
out contingency plans, and to set goals is a strong indicator of autonomy (Andrade
& Bunker, 2009; Ho & Crookall, 1995).
Competence Competence is the feeling that one has the relevant skills to achieve
a goal. Competence is measured by the learner’s ability to cope with the tasks of the
course (Ryan & Deci, 2000) and the degree to which he/she experiences feelings of
efficacy and a sense of achievement (Baard, 2002). In a learning environment, learn-
ers can express competence by challenging tasks in a way that enables them to
examine and expand their academic abilities and by allocating time for reflection to
improve their learning process. Teacher feedback and assessment can also be used
to measure competence.
Relatedness Relatedness is the sense of belonging and connectedness to the per-

sons, group, or culture disseminating a goal. In a traditional learning environment,
students feeling respected and cared for by the teacher is essential for their willing-
ness to accept the proffered classroom values (Ryan & Deci, 2000). The sense of
belonging is also important in an online learning environment and can be expressed

in activities such as online discussion groups. Studies have shown that the more the
learners find online discourses to be valuable, interesting, and enjoyable, the greater
the frequency and the quality of their participation in discussions (Järvelä et al.,
2010; Rienties et al., 2009; Xie, Debacker, & Ferguson, 2006).
When these three needs are met, there is an optimal environment for functioning
and growth (Deci & Ryan, 1985). Thus, conditions supporting individuals’ basic
psychological needs for autonomy, competence, and relatedness are argued to foster
high-quality forms of motivation and engagement in activities, including enhanced
performance, persistence, and creativity (Giesbers et al., 2013; Ryan & Deci, 2000).
3 The Study
Based on a sample of 716 Peruvian university students, the primary aim of this
study is to employ a learning analytics approach framed in Deci and Ryan’s self-
determination theory to examine the strongest predictors of persistence and achieve-
ment in an online English language course.
3.1 Research Variables
This study utilizes data collected from three different sources: a pre-course student
readiness survey, course LMS log files, and activity reports from the course
Facebook page.
3.1.1 Independent Variables
We classified the independent variables into two categories: variables related to

students’ online learning behavior, which we labelled behavioral variables, and
variables related to students’ learning strategies and online readiness, which we
labelled readiness variables.
Seven behavioral variables emerged from the LMS log files and Facebook activ-
ity reports. Description of these variables is presented in Table 14.1.
Eighteen readiness variables emerged from the student readiness survey. These
variables were classified into three categories. The first category, competence,
includes items that describe students’ learning strategies and skills (V1–V5, V12).
The second category, autonomy, includes items that describe activities related to
students’ sense of control over the learning process (V6–V11, V14–V15). The third
category, relatedness, includes variables that describe different aspects of social
interaction (V13, V16–V18) (Table 14.2).
256 D. Glick et al.
Table 14.1 Description of student behavioral variables

Variable name Description Range
V1-Behav: Average score on unit Average score on quizzes in the learning units 0–100
quizzes
V2-Behav: Midterm exam score Student grade on the midterm exam 0–100
V3-Behav: Facebook engagement Number of messages, comments, and likes 0–18
rate posted on the class Facebook page
V4-Behav: Email activity Number of emails sent by each student 0–23
V5-Behav: Time spent in course Total time each student spent in the course in 0–2316
minutes
V6-Behav: Time spent trenda Trend line of time spent studying during the −22–12
course (slope)
V7-Behav: Quiz score trend in Trend line of quiz scores in the learning units −19–6
course units during the course (slope)
a
Trend is calculated according to the values along the units from the first unit to the last. Its value
is the slope value of the progress according to the LINEST function which calculates the statistics
for a line by using the least squares method to calculate a straight line that best fits the data
Table 14.2 Description of student readiness variables

Variable Survey statement/question
V1-readiness I am certain I will be able to master the skills taught in this course
V2-readiness I am certain I can figure out how to learn even the most difficult course material
V3-readiness I believe I can do almost all the work in this class if I don’t give up
V4-readiness What was your high school grade point average?
(Decoded)
V5-readiness What grade do you expect to get in the online English proficiency course that
you are about to take?
(Decoded)
V6-readiness When I make a schedule for my coursework, I stick to it
V7-readiness I never give up even if the course material is difficult
V8-readiness I can ignore distractions around me when I study
V9-readiness I keep a record of what my assignments are and when they are due
V10-readiness I plan my work in advance so that I can complete all my assignments on time
V11-readiness Planning the order of class tasks and following a schedule is easy for me
V12-readiness I am quick to get caught up with my coursework if I start falling behind
V13-readiness When studying for this course, I plan to set aside time to discuss the course
material with a group of students from the class
V14-readiness I plan to work hard to do well in this class even if I don’t always like what we
are doing
V15-readiness On average, how many hours per week do you plan to spend on all aspects of
this course?
V16-readiness I am comfortable asking my instructor and classmates to clarify concepts I
don’t understand well
V17-readiness I believe that participating in a course Facebook page would help me learn
V18-readiness I believe that participating in a synchronous online class would help me learn
3.1.2 Dependent Variables
Three dependent variables were defined for this research: student persistence,
achievement, and motivation. Persistence was measured using the behavioral vari-
able Average Completion of Unit Materials, which is the amount of course materials
completed by the student during the course. The second outcome variable, achieve-
ment, was measured using students’ grades on the final exam, which is an online test
covering the teaching points from all the units in each course. The test was devel-
oped by a leading learning and assessment company specializing in creating and
administering high-stake exams. The test covers four skills (listening, reading,
speaking, and writing), and it reflects grammar and vocabulary covered in the
course. The test is scored on a 100-point scale. Finally, motivation is an index calcu-
lated as the mean score of ten items measuring student motivation in the pre-course
student readiness survey. Table 14.3 presents these items. The reliability analysis
revealed high scores for this motivation sub-questionnaire (N = 10, α = 0.953).
3.2 Research Context
This study was conducted at a private university in North Peru. The university
enrolls roughly 13,500 students annually in 20 degree programs, 10 of which are
delivered online. About 88% of the enrollees come from low or middle socioeco-
nomic backgrounds. Each student is required to take 20 courses of 1 month each to
fulfill their degree requirements: five courses at pre-beginner level (Levels 1–5), five
at beginner level (Levels 6–10), five at pre-intermediate level (Levels 11–15), and
five courses at intermediate level (Levels 16–20). The courses are moderated
by certified EFL teachers, who receive training in online teaching methodologies.
Table 14.3 Motivation questions from the readiness survey

VariableSurvey question
Motiv1 I’m really looking forward to learning more English
Motiv2 I enjoy learning English
Motiv3 I think learning English is very interesting
Motiv4 I think that learning is important
Motiv5 English courses are important for my future
Motiv6 I think what we are learning in the English course is important
Motiv7 The language skills we are going to acquire in this course are really important
Motiv8 Having a good level of English is important
Motiv9 Being able to communicate fluently in English is an important aspect of who I want to
be in a few years
Motiv10 I would classify myself as someone who is generally self-motivated and always gets
things done on time
Note. Cronbach’s alpha = 0.953, N = 10 items
258 D. Glick et al.
The student-to-teacher ratio is approximately 32 online students for every 1 teacher.

Students can choose between regular and intensive courses. Typically, students in
the regular track finish the course requirements within 2 years.
The online course, which is the focus of the current study, is an interactive learn-
ing environment developed by a leading technology-mediated English language
learning and assessment provider. The course combined 4 hours a week of online
instruction with monthly Skype meetings, class discussion forums, email
communication, and a Facebook page. Each course consists of eight units that include
learning material and quizzes or written assignments. Students who completed less
than 30% of the course assignments were classified as dropouts. Only students who
completed a minimum of 70% of the assigned course materials were allowed access
to the final test. The final grade in the online course was based on the student’s score
on the final test. The final grade was assigned on a 100-point scale, with a grade of
75 required to pass and continue on to the next course.
3.3 Sample Description
The course participants were 716 Peruvian undergraduate students enrolled in

online English language courses offered during the fall semester in 2017. Over two-
thirds of the enrollees came from low or middle socioeconomic backgrounds. 22.9%
of the students felt that online courses were more difficult than face-to-face courses.
16.8% of the students had slow Internet connections, and 13% of them were first-
time online learners. 7.1% of the students were not using social media regularly.
3.4 Method and Procedure
This study employs quantitative methods to analyze data collected from three dif-
ferent sources: a pre-course student readiness survey, LMS log files, and Facebook
activity reports, which were exported to an Excel spreadsheet.
–– The student readiness survey was a mandatory prerequisite for all students
enrolled in the online English language course. The online readiness survey used
in this study was adapted from Wladis and Samuels’ (2016) e-learning readiness
survey questions, which were developed by faculty and staff at a large community
college in the USA. The items on this survey are based on instruments used at
other community colleges and those identified in the research literature. Each
item on the survey was assessed for content validity by e-learning staff and
faculty. Additionally, a number of items on the student readiness survey used in
this study were adapted from Pintrich, Smith, Garcia, and McKeachie’s (1991)
Motivated Strategies for Learning Questionnaire (MSLQ2), which was developed
The MSLQ is in the public domain; therefore, permission to use this instrument is not needed.
2
to measure the types of learning strategies and academic motivation used by

college students. The survey used in this study includes 18 five-point Likert-type
scale items. Sixteen out of eighteen items (items 1–4, items 6–14, and items
16–18) were adapted from Wladis and Samuels’ (2016) and Pintrich et al. (1991)
pre-course surveys. One item in our survey (item 15) was adapted from Maki and
Maki’s (2003) student readiness survey. Finally, one item (item 5) was included
in our survey to reflect a large body of research which has shown that students’
self-efficacy beliefs may predict academic achievement (Bandura, Barbaranelli,
Caprara, & Pastorelli, 2001; Komarraju & Nadler, 2013).
–– Student log files were exported from the course LMS to an excel spreadsheet in
a tabular, student-level view with student behavioral data (V1–V7 in Table 14.1).
Student data were stored in a password-protected LMS. Special attention was
given to information security and privacy aspects.
–– The data from the Facebook page were retrieved through a designated code,
written in C#, which utilizes the Facebook graph API and retrieves data from
the Facebook feed into CSV files. It contains the following data: Number of
likes, number of comments, messages, and replies posted on the course
Facebook page.
An additional program was developed in C# to merge the information retrieved
from the three different data sources and cross-reference the student names to asso-
ciate data to students. To compensate for naming variations, a designated algorithm
was developed to match the student names by comparing text strings in the student
names. The LMS consists of 3–4 text values for each of the students enrolled and
was used as the baseline for building up the complete students’ grid. The algorithm
in the program used a comparison method to check the other data sources, so that if
at least two text strings in the complete name were matching, the reference would
be made. A manual review was conducted afterward to verify correct matching.
Finally, the merged data were entered into SPSS version 24.
3.5 Statistical Analyses
To examine the strongest predictors of persistence and achievement in an online

ELL course, three steps of statistical analysis were carried out. The first step
involved conducting factor analysis using the 7 behavioral variables (see Table 14.1)
and the 18 readiness variables (see Table 14.2). This enabled us to identify common
themes in the data and to examine whether the emerged factors would fit into the
three motivational attributes in Deci and Ryan’s SDT. The second step in the
statistical analysis involved running a Pearson correlation among the emerged
factors and the three dependent variables: persistence, achievement, and motivation.
Finally, stepwise multiple regression was used to determine the overall fit of the
model and the relative contribution of each of the predictors to the total variance
explained. It is hypothesized that the emerged factors will affect student persistence,
achievement and motivation.
260 D. Glick et al.
4 Findings
This study includes 25 independent variables classified into 2 groups: 7 behavioral

variables and 18 readiness variables related to students’ learning strategies and
online readiness. Descriptive statistics of the behavioral variables are presented in
Table 14.4. As shown, average unit quiz score (V1) and midterm exam score (V2)
means were quite high (> 80). Interestingly, although the mean of average unit quiz
score (V1) was high, the average quiz score trend (V7) was negative. This may sug-
gest that student motivation was higher during the first few weeks of the course and,
as the course progressed, student motivation dropped slightly, which may have led
to decreased scores during the second half of the semester. This may also suggest
that the level of course difficulty increases as the course progresses, which may
explain the negative average quiz score trend.
Descriptive statistics of all readiness variables are presented in Table 14.5.
Results of the competence-related variables (V1–V5, V12) show that, on average,
students have a fairly strong belief that they have the ability to master the skills
taught in this course (M = 4.08, SD = 0.79) and are fairly certain they can learn even
the most difficult course material (M = 4.12, SD = 0.81). Results of the autonomy-
related variables (V6–V11, V14–V15) indicate that students, on average, believe
they have a fairly good sense of control over the learning process. If students make
a schedule for their coursework, they tend to stick to it (M = 3.82, SD = 1.0), and
they do not usually give up even if the course material is difficult (M = 4.31,
SD = 0.81). Finally, results for the relatedness variables (V13, V16–V18) show that
students are fairly comfortable asking their teacher and classmates to clarify con-
cepts (M = 3.9, SD = 0.94) and believe that participating in the course Facebook
page will help them learn (M = 3.89, SD = 0.99).
To identify common themes, a factor analysis with a varimax rotation using the
standardized values (z-scores) of the 7 behavioral variables and 18 readiness vari-
ables was conducted. The results of the factor analysis are presented in Tables 14.6
and 14.7. All the correlation coefficients, except one, are greater than 0.9, indicating
that there is no problem of multicollinearity or singularity in the data. The results of
the behavioral variable analysis are presented in Table 14.6. The variables that load
highly on Factor 1 (unit score trend, average unit score, and time spent trend) relate
to unit score. Therefore, this factor was named unit learning outcomes. The vari-
ables that load highly on Factor 2 (Facebook and email activity) relate to the number
Table 14.4 Descriptive statistics of behavioral variables (N = 716)

Variable name Range Mean SD
V1-Behav: Average unit quiz score 0–100 80.83 25.22
V2-Behav: Midterm exam score 0–100 83.69 15.074
V3-Behav: Facebook engagement rate 0–18 1.96 3.260
V4-Behav: Email activity 0–23 2.51 4.233
V5-Behav: Time spent in course 0–2316 521.73 445.449
V6-Behav: Time spent trend −22–12 −3.88 5.85
V7-Behav: Quiz score trend in units −19–6 −1.88 4.29
Table 14.5 Descriptive statistics of readiness variables

Variable Survey question Mean SD N
V1-readiness I am certain I will be able to master the skills taught in this 4.08 0.794 387
course
V2-readiness I am certain I can figure out how to learn even the most 4.12 0.807 383
difficult course material
V3-readiness I believe I can do almost all the work in this class if I don’t 4.31 0.769 384
give up
V4-readiness What was your high school grade point average 2.90 0.901 384
(Decoded)
V5-readiness What grade do you expect to get in the online English 2.35 0.687 393
proficiency course that you are about to take?
(Decoded)
V6-readiness When I make a schedule for my coursework, I stick to it 3.82 1.008 387
V7-readiness I never give up even if the course material is difficult 4.31 0.814 388
V8-readiness I can ignore distractions around me when I study 3.63 1.092 387
V9-readiness I keep a record of what my assignments are and when they 3.85 1.043 387
are due
V10-readiness I plan my work in advance so that I can complete all my 3.87 0.990 387
assignments on time
V11-readiness Planning the order of class tasks and following a schedule 3.97 0.915 381
is easy for me
V12-readiness I am quick to get caught up with my coursework if I start 4.2 0.856 386
falling behind
V13-readiness When studying for this course, I plan to set aside time to 3.41 1.153 384
discuss the course material with a group of students from
the class
V14-readiness I plan to work hard to do well in this class even if I don’t 3.96 1.142 391
always like what we are doing
V15-readiness On average, how many hours per week do you plan to 3.06 1.346 392
spend on all aspects of this course?
V16-readiness I am comfortable asking my instructor and classmates to 3.90 0.943 383
clarify concepts I don’t understand well
V17-readiness I believe that participating in a course Facebook page 3.89 0.989 348
would help me learn
V18-readiness I believe that participating in a synchronous online class 4.15 0.938 388
would help me learn
Note. Responses were made using a response scale of 1–5
Table 14.6 Results of factor analysis of behavioral variables

1 2 3 4
V7-Behav: Unit score trend .883 0.050 0.030 −0.147
V1-Behav: Average unit score .807 0.136 0.232 −0.016
V6-Behav: Time spent trend .629 −0.393 −0.410 0.229
V3-Behav: Facebook activity 0.226 .851 0.040 −0.051
V4-Behav: Email activity 0.234 −.562 0.242 −0.346
V5-Behav: Time spent in course 0.112 −0.077 .899 0.086
V2-Behav: Midterm score −0.034 0.045 0.085 .929
262 D. Glick et al.
of messages sent or received via email and number of messages and likes posted
on the class Facebook page. This factor, therefore, was named social engagement.
The variable that loads highly on Factor 3 (time spent in course) seems to relate to
one’s ability to allocate time and effort; therefore, we named this factor time spent
learning. Finally, the midterm score is the only variable that loads highly on Factor 4;
therefore, this factor was named midterm. These factors explain 76.2% of the total
variance in the behavioral variables.
The results of the statements that assess student readiness are presented in
Table 14.7. The statements that load highly on Factor 5 (e.g., when I make a s chedule
for my coursework, I stick to it) seem to all relate to processes that make up activities
such as planning, monitoring, and regulating. Therefore, this factor was named self-
regulation. The statements that load highly on Factor 6 (e.g., I believe I can do almost
all the work in this class if I don’t give up) seem to relate to one’s belief that learning
outcomes are contingent on one’s own efforts; therefore, this factor was named
learning beliefs. The statements that load highly on Factor 7 (e.g., I believe that par-
ticipating in a course Facebook page would help me learn) relate to one’s belief that
interaction with peers and the virtual instructor via the course c ommunication tools
can help the learner clarify course material. Therefore, we named this factor peer
learning. The two questions that load highly on Factor 8 (what was your high school
GPA and what grade do you expect to get in the Online English Proficiency course
that you are about to take) seem to relate to two aspects of academic success, past
success and performance expectations, and were named academic success. Finally,
the statements that load highly on Factor 9 relate to effort regulation (e.g., I plan
to work hard to do well in this class even if I don’t always like what we are doing).
Table 14.7 Results of factor analysis of readiness variables

5 6 7 8 9
V6-readiness .780 0.111 0.201 0.043 0.066
V9-readiness .765 0.106 0.110 0.142 −0.168
V10-readiness .699 0.198 0.060 −0.098 0.084
V8-readiness .681 0.181 0.151 0.173 0.049
V13-readiness .664 0.224 0.203 −0.040 −0.018
V7-readiness .635 0.264 0.244 0.004 0.340
V12-readiness .625 0.154 0.179 0.162 0.383
V11-readiness .619 0.452 0.071 −0.130 −0.154
V1-readiness 0.181 .817 0.215 0.228 −0.002
V2-readiness 0.294 .809 0.187 0.252 −0.027
V3-readiness 0.339 .736 0.071 0.091 0.173
V17-readiness 0.199 0.106 .841 −0.044 0.056
V18-readiness 0.172 0.236 .790 −0.011 −0.021
V16-readiness 0.475 0.103 .523 0.310 0.008
V4-readiness −0.063 0.107 0.016 .771 0.096
V5-readiness 0.102 0.217 −0.010 .687 −0.098
V14-readiness 0.331 −0.001 0.258 0.225 .654
V15-readiness 0.228 −0.038 0.290 0.281 −.611
This factor, therefore, was named effort regulation. These factors explain 64.8% of the
total variance in the readiness variables.
The nine factors fit well into the three categories in Deci and Ryan’s SDT (see
Table 14.8). Three factors fall into the first category—autonomy. They include one
behavioral-related factor (Factor 3: time spent learning) and two factors related to
the student readiness survey (Factor 5, self-regulation, and Factor 9, effort
regulation). Four factors fall into the second category—competence. These factors
include two behavioral-related factors (Factor 1, unit learning outcomes, and Factor
4, midterm) and two factors related to the student readiness survey (Factor 6, learn-
ing beliefs, and Factor 8, academic success). Finally, there are two factors that fall
into the third category—relatedness. They include one behavioral-related factor
(Factor 2: social engagement) and one factor related to the student readiness survey
(Factor 7: peer learning).
Table 14.9 presents the descriptive statistics for the dependent variables: persis-
tence, achievement, and motivation. As shown, the mean scores for persistence
(M = 85.02), achievement (M = 81.53), and motivation (M = 4.33) were all high.
Table 14.8 Fitting the emerged factors into SDT

Autonomy Competence Relatedness
Behavioral Factor 3: Time spent Factor 1: Unit learning Factor 2: Social
data learning outcomes engagement
V5- Behav: Time spent in V1-Behav: Average unit V3-Behav: Facebook
course score activity
V7-Behav: Unit score V4-Behav: Email
trend activity
V6- Behav: Time spent
trend
Factor 4: Midterm
V2-Behav: Midterm score
Readiness Factor 5: Self-regulation Factor 6: Learning beliefs Factor 7: Peer learning
survey V6-readiness V1-readiness V17-readiness
V9-readiness V2-readiness V18-readiness
V10-readiness V3-readiness V16-readiness
V8-readiness Factor 8: Academic
V13-readiness success
V7-readiness V4-readiness
V12-readiness V5-readiness
V11-readiness
Factor 9: Effort regulation
V14-readiness
V15-readiness
Table 14.9 Descriptive frequency statistics of the dependent variables

Variable Range Mean SD N
Persistence 1–100 85.02 27.530 668
Achievement 13–100 81.53 13.739 574
Motivation 2–5 4.33 0.646 392
264 D. Glick et al.
Figure 14.1 presents the nine factors that emerged from the factor analysis
along with the dependent variables—persistence, achievement, and motivation.
These variables were used in this study to quantify student autonomy (the degree of
learner control over the learning process), competence (sense of ability and efficacy),
and relatedness (the perception of social belonging), as well as student persistence,
achievement, and motivation.
Pearson correlation coefficients, which include the nine factors of persistence,
achievement, and motivation, are presented in Table 14.10. As shown, unit learning
outcomes are significantly and strongly correlated with persistence (r = 0.619,
p < 0.01); time spent learning is moderately correlated with persistence (r = 0.297,
Fig. 14.1 The emerged factors and the outcome variables
Table 14.10 Correlations between predictor factors and the dependent variables
Measure Persistence Achievement Motivation
1. Midterm score 0.253a 0.452a 0.008
2. Unit learning outcomes 0.619a 0.129a −0.047
3. Time spent learning 0.297a −0.171a 0.088
4. Social engagement 0.220a −0.047 0.075
5. Self-regulation 0.058 0.018 0.474a
6. Effort regulation 0.100 −0.015 0.307a
7. Learning beliefs 0.020 −0.054 0.670a
8. Academic success −0.024 0.109b 0.330a
9. Peer learning −0.004 −0.140a 0.409a
Correlation is significant at the 0.01 level (two tailed).
a
Correlation is significant at the 0.05 level (two tailed).

b
p < 0.01). In addition, the analysis revealed positive, moderate-to-high significant

correlations between motivation and the five readiness factors: learning beliefs
(r = 0.670, p < 0.01), self-regulation (r = 0.474, p < 0.01), peer learning (r = 0.409,
p < 0.01), academic success (r = 0.330, p < 0.01), and effort regulation (r = 0.307,
p < 0.01). Additionally, a moderate-to-strong correlation is found between midterm
score and student achievement (r = 0.452, p < 0.01). The results show that three
behavioral factors (midterm score, unit learning outcomes, and time spent learning)
are significantly correlated with either achievement or persistence. Interestingly, no
significant correlation was found between any of the behavioral factors (Factors
1–4) and motivation. On the other hand, the readiness factors (Factors 5–9) were all
significantly correlated with motivation.
To test the validity of the emerged factors in predicting student persistence, we
ran a stepwise multiple regression model with average completion of course materi-
als (persistence) as the dependent variable. The results can be seen in Table 14.11.
Model 3 significantly predicts student persistence (p < 0.001). The combination of
three factors (unit learning outcomes, time spent learning, and midterm) accounts
for 50% of the variance in persistence scores (adjusted R2 = 0.50). These results
suggest that average unit score and unit score trend (i.e., unit learning outcomes),
time spent learning, and midterm score each uniquely predict how well a student
will do in the course.
To test the validity of the emerged factors in predicting student achievement, we
ran a stepwise multiple regression model with final exam score as the outcome
variable. The results can be seen in Table 14.12. Model 4 significantly predicts
student achievement (p < 0.001). The combination of four factors (midterm, peer
learning, self-regulation, and time spent learning) accounts for 28.2% of the variance
in final exam scores (adjusted R2 = 0.282). These results suggest that one’s midterm
score combined with processes such as planning, monitoring, and regulating
(i.e., self-regulation) may predict student score on the final test. Having said that,
the proportion of the explained variance is relatively low (28.2%), which suggests
that there are other variables which account for 71.8% of the variance.
Table 14.11 Multiple regression model for student persistence by emerged factors
Unstandardized
coefficients Standardized coefficients
Model B SE Beta t P
1 (constant) 89.218 0.732 121.809 0.000
Unit learning outcomes 17.505 0.971 0.687 18.026 0.000
2 (constant) 88.936 0.727 122.370 0.000
Time spent learning 2.473 0.728 0.128 3.398 0.001
3 (constant) 88.937 0.717 124.047 0.000
Time spentlearning 2.635 0.719 0.137 3.663 0.000
Midterm 2.534 0.764 0.126 3.315 0.001
Dependent variable: persistency
a
266 D. Glick et al.
Table 14.12 Multiple regression model for student achievement by emerged factors
Unstandardized
coefficients Standardized coefficients
Model B SE Beta t P
1 (constant) 81.041 0.610 132.903 0.000
Midterm 7.661 0.708 0.508 10.813 0.000
2 (constant) 88.341 3.204 27.571 0.000
Midterm 7.553 0.705 0.500 10.707 0.000
Peer learning −1.835 0.791 −0.108 −2.320 0.021
3 (constant) 84.116 3.637 23.125 0.000
Midterm 7.464 0.701 0.494 10.640 0.000
Peer learning −3.168 0.962 −0.187 −3.292 0.001
Self-regulation 2.448 1.021 0.136 2.397 0.017
4 (constant) 84.619 3.628 23.324 0.000
Midterm 7.277 0.704 0.482 10.337 0.000
Peer learning −3.160 0.958 −0.187 −3.300 0.001
Self-regulation 2.364 1.017 0.131 2.324 0.021
Time spent learning −1.289 0.625 −0.096 −2.062 0.040
Dependent variable: achievement
a
5 Discussion
Online language learning has been recognized as a promising approach to increase

students’ English language proficiency in developing countries where high-quality
language learning resources are limited (Bai et al., 2016; Glick et al., 2016).
Identifying factors that predict students’ performance in online courses can inform
institutions and instructors of interventions to improve learning experiences and
success in online courses (Ai & Laffey, 2007; Romero & Ventura, 2007). Based on
the SDT and on data from a pre-course readiness survey, LMS log files, and
Facebook activity reports, this study identifies several important predictors of per-
sistence and achievement in online language courses in a low-income developing
country.
We first used factor analysis to identify latent factors from behavioral variables
and readiness variables and classified these latent factors into three categories of
measures based on the SDT. Nine factors emerged from the factor analysis.
Specifically, four latent factors (i.e., learning beliefs, academic success, unit learn-
ing outcomes, and midterm score) captured students’ competence and were indica-
tive of the degree to which students were confident about their performance in the
course. It is worth noting though that this may not be an exhaustive list, and addi-
tional factors may influence students’ sense of competence, such as prior knowl-
edge and students’ perceived value (Eccles & Wigfield, 2002). Two factors – peer
learning and social engagement – measured students’ relatedness and reflected how
much students valued social interaction at the beginning of the course and the level
of social interaction each student actually engaged in the course. The other three
latent factors – time spent learning, self-regulation, and effort regulation – focused
on the extent to which students could control their own learning and were classified
as measures of autonomy.
In exploring the relationship between each of the three categories and course
performance, we found that factors under the categories of competence and auton-
omy were predictive of students’ course persistence and achievement, which aligns
with the SDT (Ryan & Deci, 2000). In particular, the factors of midterm score and
self-regulation were strong and positive predictors of students’ final test score. We
also found that time spent on the course was a significantly negative predictor of the
final test score, suggesting that students who spend more time in a course may have
difficulties understanding the course content or completing the assignments.
Interestingly, we also identified a negative relationship between the extent to which
a student valued peer learning at the beginning of the course and their course
achievement, suggesting that students who value peer-to-peer interactions may
encounter particular difficulties in online learning environments, which typically
include limited human interactions, compared to in traditional face-to-face
environments.
These results have several theoretical and practical implications. First, prior
studies on the SDT mainly focused on traditional face-to-face classrooms. Our
study extends this line of research on SDT to the online learning environment,
which differs substantially from face-to-face classrooms in terms of time and space,
methods of communication, and the roles of teachers and students (Moore &
Kearsley, 2011). The SDT proposes that students’ sense of autonomy, relatedness,
and competence improves motivation, thereby positively influencing learning out-
comes (Ryan & Deci, 2000). Our findings provide suggestive evidence that the SDT
also applies to online learning and that measures related to autonomy and compe-
tence are predictive of course performance. Moreover, we took advantage of the
nuanced behavioral data in online learning to measure students’ behaviors and per-
formance during the course, which reduces possible bias and inaccuracy related to
self-reported measures used in previous research in face-to-face classrooms (Baker
& Inventado, 2014). As proposed by the SDT, we found that most of the behavioral
measures were moderately and positively correlated with persistence and achieve-
ment, indicating the validity of these measures. Overall, these results provide guid-
ance for future research in using course behavioral data to measure students’
autonomy, relatedness, and competence in online classes.
Second, prior research found that students with certain background characteris-
tics, such as lower levels of academic preparation, have more difficulty succeeding
in online courses than in face-to-face courses (Xu & Jaggars, 2014). Yet, it is unclear
what specific personal attributes make an individual particularly vulnerable in an
online learning environment. As a result, it is difficult to identify students who may
suffer in online courses and provide preventative interventions, early warnings, or
additional support to increase the success rate of these students. With the rich infor-
mation collected through the pre-course survey, we found that students who lacked
self-regulation skills were more likely to suffer and have lower performance in
online classes. These findings are in line with the online learning literature that
268 D. Glick et al.
s elf-regulation ability plays a particularly important role in the virtual learning envi-
ronment where students are required to take greater responsibility to control and
regulate their own learning (Duffy & Kirkley, 2003). To support students with low
self-regulation abilities in online courses, institutions could provide academic
recourses, such as workshops on online learning or self-regulation skills, to improve
students’ capacity to learn effectively online. In addition, institutions may also
require pre-course assessment to evaluate students’ readiness for online learning
before allowing them to take an online course, which could help students make bet-
ter informed decisions about which course delivery format to opt in.
Third, we found behavioral variables such as time spent in course and midterm
exam score were significant predictors of student success. Using these real-time
measures, institutions can develop an early warning system to identify struggling
students early in a course (Dominguez, Bernacki, & Uesbeck, 2016; Macfadyen &
Dawson, 2010). Instructors can then provide timely pedagogical interventions to
at-risk students and help them to succeed (Jaggers & Xu, 2016). For instance, our
study suggests that students who spend more time in a course might experience
higher levels of difficulties understanding the course content or completing the
assignments. Instructors can help those students through consultancy support or by
digging deep into the behavioral data to identify the specific problem that the stu-
dents had encountered and provide learning materials that better support students.
There are a few caveats to bear in mind when interpreting the results from the
current study. First, while midterm score is highly predicative of students’ final
course grade and can therefore be used for early identification of at-risk students,
the extent to which this strategy could indeed help students largely depends on the
quality of follow-up support the instructor is able to provide to the student. Thus, the
early warning system needs to be accompanied with comprehensive and effective
supports and resources for struggling students. In addition, although midterm score
proves to be a strong predictor of students’ final grade, it is collected at a point when
students have already finished a significant amount of the coursework. Therefore,
future research is needed to explore factors that can help identifying at-risk learners
as early as possible. The pre-course survey measures that are found to be predictive
of persistence and achievement in this study, such as learning beliefs and percep-
tions on peer learning, provide guidance on future investigations along this direc-
tion. Another potential candidate unexplored in this study is user behavior that is
indicative of high risk of failing the course. The online learning environment pro-
vides educators and researchers with an unprecedented opportunity to achieve a
timely understanding of student course experiences by automatically recording a
large swath of behavioral information from each learner. Future research needs to be
conducted to correlate such behavioral information with course outcomes. Finally,
several achievement predictors identified in this study are measured based on stu-
dent self-report before the course, which is the most commonly used method for
measuring psychological constructs, such as self-regulation skills and motivational
beliefs (Winne & Perry, 2000). However, the usefulness of self-reported informa-
tion is limited when the response rate is low. Future research may wish to take
advantage of the nuanced behavioral data available in the online learning
e nvironments and explore possible ways to measure these psychological constructs

using the behavior data.
Students’ competence, autonomy, and relatedness are important predictors of
student persistence, achievement, and motivation in online learning. In this study,
we developed measures of students’ competence, autonomy, and relatedness, and
among these measures, we identified strong predictors of student success in online
language courses. Results from this study could therefore assist institutions and
instructors in identifying students who are at risk of failure in online learning and
providing effective interventions early on during a course to improve students’
learning outcomes.
Acknowledgments We would like to thank Betty Luz Zegarra Angulo of the Universidad Señor
de Sipán for helping make available the data for this study as well as providing detailed informa-
tion on the study context.
References
Ai, J., & Laffey, J. (2007). Web mining as a tool for understanding online learning. MERLOT
Journal of Online Learning and Teaching, 3(2), 160–169.
Allen, D. F., & Bir, B. (2012). Academic confidence and summer bridge learning communities:
Path analytic linkages to student persistence. Journal of College Student Retention: Research,
Theory & Practice, 13(4), 519–548.
Andrade, M. S., & Bunker, E. L. (2009). A model for self-regulated distance language learning.
Distance Education, 30(1), 47–61.
Angelino, L. M., Williams, F. K., & Natvig, D. (2007). Strategies to engage online students and
reduce attrition rates. Journal of Educators Online, 4(2), 1–14.
Appana, S. (2008). A review of benefits and limitations of online learning in the context of the stu-
dent, the instructor, and the tenured faculty. International Journal on E-Learning, 7(1), 5–22.
Baard, P. P. (2002). Intrinsic need satisfaction in organizations: A motivational basis of success in
for-profit and not-for-profit settings. Handbook of Self-Determination Research, 2, 255–275.
Bai, Y., Mo, D., Zhang, L., Boswell, M., & Rozelle, S. (2016). The impact of integrating ICT with
teaching: Evidence from a randomized controlled trial in rural schools in China. Computers &
Baker, R. S., & Inventado, P. S. (2014). Educational data mining and learning analytics. In
Learning analytics (pp. 61–75). New York, NY: Springer.
Baker, R. S., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future
visions. Journal of Educational Data Mining, 1(1), 3–17.
Bakia, M., Shear, L., Toyama, Y., & Lasseter, A. (2012). Understanding the implications of online
learning for educational productivity. Washington, DC: Office of Educational Technology, US
Department of Education.
Bañados, E. (2006). A blended-learning pedagogical model for teaching and learning EFL success-
fully through an online interactive multimedia environment. Calico Journal, 23(3), 533–550.
Banditvilai, C. (2016). Enhancing student’s language skills through blended learning. The
Electronic Journal of E-Learning, 14(3), 220–229. Available from www.ejel.org
Bandura, A., Barbaranelli, C., Caprara, G. V., & Pastorelli, C. (2001). Selfefficacy beliefs as
shapers of children’s aspirations and career trajectories. Child Development, 72, 187–206.
Barani, G. (2011). The relationship between computer assisted language learning (CALL) and lis-
tening skill of Iranian EFL learners. Procedia-Social and Behavioral Sciences, 15, 4059–4063.
270 D. Glick et al.
Black, A. E., & Deci, E. L. (2000). The effects of instructors’ autonomy support and students’
autonomous motivation on learning organic chemistry: A self-determination theory perspec-
tive. Science Education, 84, 740–756.
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a
new era. Educause Review, 42(4), 42–57.
Cheng, J., Kulkarni, C., & Klemmer, S. (2013). Tools for predicting drop-off in large online
classes. In Proceedings of the 2013 conference on computer supported cooperative work com-
panion (pp. 121–124). New York, NY: ACM.
Clay, M., Rowland, S., & Packard, A. (2009). Improving undergraduate online retention through
gated advisement and redundant communication. Journal of College Student Retention, 10(1),
93–102.
Cohen, A., & Nachmias, R. (2006). A quantitative cost effectiveness model for web-supported
academic instruction. The Internet and Higher Education, 9(2), 81–90.
Cohen, A., & Nachmias, R. (2012). The implementation of a cost effectiveness analyzer for
web-supported academic instruction: An example from life science. International Journal on
E-Learning, 11(2), 5–22.
Cohen, A. (2017). Analysis of student activity in web-supported courses as a tool for predicting
dropout. Educational Technology Research and Development, 65(5), 1285–1304.
Cronquist, K., & Fiszbein, A. (2017). English Language Learning in Latin America. Inter-American
Dialogue. Available from: www.dropbox.com/s/zabf293t0a12ten/English-Language-Learning-
in-Latin-America-Final.pdf?dl=0
Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S. (2014). Current state and future trends: A
citation network analysis of the learning analytics field. In Proceedings of the fourth interna-
tional conference on learning analytics and knowledge (pp. 231–240). New York, NY: ACM.
https://doi.org/10.1145/2567574.2567585
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior.
New York, NY: Plenum.
Dickinson, L. (1995). Autonomy and motivation a literature review. System, 23(2), 165–174.
Dietz-Uhler, B., & Hurn, J. E. (2013). Using learning analytics to predict (and improve) student
success: A faculty perspective. Journal of Interactive Online Learning, 12(1), 17–26.
Dominguez, M., Bernacki, M. L., & Uesbeck, P. M. (2016). Predicting STEM achievement with
learning management system data: Prediction modeling and a test of an early warning system.
In EDM (pp. 589–590).
Driscoll, A., Jicha, K., Hunt, A. N., Tichavsky, L., & Thompson, G. (2012). Can online courses
deliver in-class results? A comparison of student performance and satisfaction in an online ver-
sus a face-to-face introductory sociology course. Teaching Sociology, 40(4), 312–331. https://
doi.org/10.1177/0092055X12446624
Duffy, T. M., & Kirkley, J. R. (2003). Learner-centered theory and practice in distance education:
Cases from higher education. New York, NY: Routledge.
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of
Psychology, 53(1), 109–132.
EF English Proficiency Index. (2016). EF education first. Available from https://www.theewf.org/
uploads/pdf/ef-epi-2016-english.pdf
Erdem, M., & Kibar, P. N. (2014). Students’ opinions on facebook supported blended learning
environment. TOJET: The Turkish Online. Journal of Educational Technology, 13(1), 199–206.
Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote
one size fits all: The effects of instructional conditions in predicting academic success. The
Internet and Higher Education, 28, 68–84.
Giesbers, B., Rienties, B., Tempelaar, D., & Gijselaers, W. (2013). Investigating the relations
between motivation, tool use, participation, and performance in an e-learning course using
web-videoconferencing. Computers in Human Behavior, 29(1), 285–292.
Glick, D., Xu, D., Warschauer, M., Rodriguez, F., Li, Q., & Cung, B. (2016). Maximizing learn-
ing outcomes through blended learning: What research shows. Paper presented at the 2016
Association of Binational Centers of Latin America Conference, Houston, TX.
Goldstein, P. J., & Katz, R. N. (2005). Academic analytics: The uses of management informa-
tion and technology in higher education. Washington, DC: EDUCAUSE Center for Applied
Research.
Hartnett, M., George, A. S., & Dron, J. (2011). Examining motivation in online distance learning
environments: Complex, multifaceted and situation-dependent. The International Review of
Research in Open and Distributed Learning, 12(6), 20–38.
Ho, J., & Crookall, D. (1995). Breaking with Chinese cultural traditions: Learner autonomy in
English language teaching. System, 23(2), 235–243.
Holec, H. (1981). Autonomy and foreign language learning. Oxford, England/New York, NY:
Pergamon Press (First Published 1979, Council of Europe).
Jaggers, S. S., & Xu, D. (2016). How do online course design features influence student perfor-
mance? Computers & Education, 95, 270–284.
Järvelä, S., Volet, S., & Järvenoja, H. (2010). Research on motivation in collaborative learning:
Moving beyond the cognitive–situative divide and combining. Educational Psychologist, 45,
15–27.
Johnson, L., Becker, S., Estrada, V., & Freeman, A. (2015). The NMC Horizon Report: 2015
Higher Education Edition. Austin, TX: New Media Consortium.
Komarraju, M., & Nadler, D. (2013). Self-efficacy and academic achievement: Why do implicit
beliefs, goals, and effort regulation matter? Learning and Individual Differences, 25, 67–72.
Lee, L. (2016). Autonomous learning through task-based instruction in fully online language
courses. Language Learning & Technology, 20(2), 81–97.
Levi-Gamlieli, H., Cohen, A., & Nachmias, R. (2015). Detection of overly intensive learning by
using weblog of course website. Technology, Instruction, Cognition and Learning (TICL),
10(2), 151–171.
Levy, Y. (2007). Comparing dropouts and persistence in e-learning courses. Computers &
Lim, J. M. (2016). Predicting successful completion using student delay indicators in undergradu-
ate self-paced online courses. Distance Education, 37(3), 317–332. https://doi.org/10.1080/01
587919.2016.1233050
Lu, J., Yu, C. S., & Liu, C. (2003). Learning style, learning patterns, and learning performance in
a WebCT-based MIS course. Information & Management, 40(6), 497–507. https://doi.org/10.
1016/S0378-7206(02)00064-2
Lykourentzou, I., Giannoukos, I., Nikolopoulos, V., Mpardis, G., & Loumos, V. (2009). Dropout
prediction in e-learning courses through the combination of machine learning techniques.
Computers & Education, 53, 950–965. https://doi.org/10.1016/j.compedu.2009.05.010
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop “early warning system” for
educators: A proof of concept. Computers & Education, 54, 588–599. https://doi.org/10.1016/j.
compedu.2009.09.008
Marzban, A. (2011). Improvement of reading comprehension through computer-assisted language
learning in Iranian intermediate EFL students. Procedia Computer Science, 3, 3–10.
Maki, R., & Maki, W. (2003). Prediction of learning and satisfaction in Webbased and lecture
courses. Journal of Educational Computing Research, 28(3), 197–219.
Mandernach, B. J. (2009). Effect of instructor-personalized multimedia in the online classroom.
International Review of Research in Open and Distance Learning, 10(3), 19.
Massengale, L. R., & Vasquez, E. (2016). Assessing accessibility: Are online courses better than
face-to-face instruction at providing access to course content for students with disabilities?
Journal of the Scholarship of Teaching and Learning, 16(1), 69–79. https://doi.org/10.14434/
josotl.v16i1.19101
Miltiadou, M., & Savenye, W. C. (2003). Applying social cognitive constructs of motivation to
enhance student success in online distance education. AACE Journal, 11(1), 78–95.
272 D. Glick et al.
Ministério de Educación Perú. (2016a). Currículo nacional de la educación básica. Retrieved

from http://www.minedu.gob.pe/curriculo/pdf/curriculo-nacional-2016-2.pdf.
Ministério de Educación Perú. (2016b). Plan de implementación al 2021 de la política nacional de
enseñanza, aprendizaje y uso de idioma Inglés – Política “Inglés, puertas al mundo. Retrieved
from http://www.minedu.gob.pe/ingles-puertas-al-mundo.
Ministério de Educación Perú. (2017). Jornada escolar completa: Secundaria. Retrieved from
http://www.minedu.gob.pe/jec/escuela-jec.php.
Moore, M. G., & Kearsley, G. (2011). Distance education: A systems view of online learning.
Belmont, CA: Cengage Learning.
Rios, S. M., & Cabrera, A. F. (2008). La efectividad de un model de aprendizaje combinado
para la enseñanza del inglés como lengua extranjera: Estudio empírico. RLA, Revista de
Linguistica Teórica y Aplicada, 46(2), 95–118. Retrieved from http://www.scielo.cl/scielo.
php?script=sci_arttext&pid=S0718-48832008000200006
Muilenburg, L. Y., & Berge, Z. L. (2005). Student barriers to online learning: A factor analytic
study. Distance Education, 26(1), 29–48. https://doi.org/10.1080/01587910500081269
Nistor, N., & Neubauer, K. (2010). From participation to dropout: Quantitative participation patterns
in online university courses. Computers & Education, 55, 663–672. https://doi.org/10.1016/
j.compedu.2010.02.026
OECD. (2015). Education at a glance 2015: OECD indicators. Paris, France: OECD Publishing.
https://doi.org/10.1787/eag-2015-en
Otter, R. R., Seipel, S., Graeff, T., Alexander, B., Boraiko, C., Gray, J., … Sadler, K. (2013).
Comparing student and faculty perceptions of online and traditional courses. The Internet and
Higher Education, 19, 27–35. https://doi.org/10.1016/j.iheduc.2013.08.001
Palmer, S., & Holt, D. (2010). Students’ perceptions of the value of the elements of an online
learning environment: Looking back in moving forward. Interactive Learning Environments,
18(2), 135–151.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the
motivated strategies questionnaire (MSLQ). Ann Arbor, MI: University of Michigan, National
Center for Research to Improve Postsecondary Teaching and Learning.
in online learning. Educational Technology & Society, 12(4), 207–217.
Rienties, B., Tempelaar, D., Van den Bossche, P., Gijselaers, W., & Segers, M. (2009). The role
of academic motivation in computer-supported collaborative learning. Computers in Human
Behavior, 25(6), 1195–1206.
Roby, T., Ashe, S., Singh, N., & Clark, C. (2013). Shaping the online experience: How administra-
tors can influence student and instructor perceptions through policy and practice. The Internet
and Higher Education, 17, 29–37.
Rodriguez, M. C., Rooms, A., & Montañez, M. (2008). Students’ perceptions of online-learning
quality given comfort, motivation, satisfaction, and experience. Journal of Interactive Online
Learning, 7(2), 105–125.
Romero, C., López, M. I., Luna, J. M., & Ventura, S. (2013). Predicting students’ final perfor-
mance from participation in on-line discussion forums. Computers & Education, 68, 458–472.
https://doi.org/10.1016/j.compedu.2013.06.009
Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert
Systems with Applications, 33(1), 135–146. https://doi.org/10.1016/j.eswa.2006.04.005
Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new
directions. Contemporary Educational Psychology, 25(1), 54–67.
Santana, M. A., Costa, E. B., Neto, B. F. D. S., Silva, I. C. L., & Rego, J. B. (2015). A predictive
model for identifying students with dropout profiles in online courses. In Proceeding of the 8th
international conference on educational data mining, EDM Workshops.
Schunk, D. H. (2012). Learning theories, an educational perspective (6th ed.). Boston, MA:
Pearson Education Inc.
Siemens, G., & Gašević, D. (2012). Special issue on learning and knowledge analytics. Educational
Technology & Society, 15(3), 1–163.
Sife, A., Lwoga, E., & Sanga, C. (2007). New technologies for teaching and learning: Challenges
for higher learning institutions in developing countries. International Journal of Education and
Development Using ICT, 3(2), 57–67.
Vahdat, S., & Eidipour, M. (2016). Adopting CALL to improve listening comprehension of iranian
junior high school students. Theory and Practice in Language Studies, 6(8), 1609–1617.
Vansteenkiste, M., Lens, W., & Deci, E. L. (2006). Intrinsic versus extrinsic goal contents in
self-determination theory: Another look at the quality of academic motivation. Educational
Psychologist, 41(1), 19–31.
Willging, P. A., & Johnson, S. D. (2009). Factors that influence students’ decision to dropout of
online courses. Journal of Asynchronous Learning Networks, 13(3), 115–127.
Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In Handbook of self-
regulation (pp. 531–566).
Wladis, C., & Samuels, J. (2016). Do online readiness surveys do what they claim? Validity, reli-
ability, and subsequent student enrollment decisions. Computers & Education, 98, 39–56.
https://doi.org/10.1016/j.compedu.2016.03.001
Xie, K. U. I., Debacker, T. K., & Ferguson, C. (2006). Extending the traditional classroom through
online discussion: The role of student motivation. Journal of Educational Computing Research,
34(1), 67–89.
Xu, D., & Jaggars, S. S. (2014). Performance gaps between online and face-to-face courses:
Differences across types of students and academic subject areas. The Journal of Higher
Education, 85(5), 633–659.
You, J. W. (2016). Identifying significant indicators using LMS data to predict course achievement
in online learning. The Internet and Higher Education, 29, 23–30. https://doi.org/10.1016/j.
iheduc.2015.11.003
Yuan, J., & Kim, C. (2014). Guidelines for facilitating the development of learning communities in
online courses. Journal of Computer Assisted Learning, 30(3), 220–232.
Zakrzewska, D. (2009). Cluster analysis in personalized e-learning systems. In N. T. Nguyen &
E. Szczerbicki (Eds.), Intelligent systems for knowledge management (pp. 229–250). Berlin,
Germany: Springer.
Chapter 15
Using Learning Analytics to Examine
Relationships Between Learners’ Usage
Data with Their Profiles and Perceptions:
A Case Study of a MOOC Designed
for Working Professionals
Min Liu, Wenting Zou, Chenglu Li, Yi Shi, Zilong Pan, and Xin Pan
1 Introduction
MOOCs, as an emerging online instruction format, have attracted thousands of peo-

ple around the world to learn topics of their interests without time and geographical
constraints. MOOC participants interact with course content, instructors, and their
peers through a MOOC platform which leaves a large amount of data on how stu-
dents access various features of a MOOC. Such analytics can provide useful insights
into students’ learning behaviors and the design of MOOC instruction. The purpose
of this study was to investigate how participants in a MOOC designed for working
professionals interacted with various key components of the MOOC (e.g., discus-
sion forums, readings, videos, quizzes, optional resources). Examining and under-
standing such usage patterns can help MOOC instructors and researchers gain
insights into students’ knowledge building and MOOC course designs.
1.1 Use of Learning Analytics in MOOCs
In recent years, the use of learning analytics is being investigated in MOOC research
focusing on different aspects. For example, studies on MOOCs have been conducted
about learner behavioral patterns (Khalil & Ebner, 2016, 2017; Kizilcec, Pérez-
Sanagustín, & Maldonado, 2017; Tseng, Tsao, Yu, Chan, & Lai, 2016), students’
engagement and motivation (Lu, Huang, Huang, & Yang, 2017; Xing, Chen, Stein,
M. Liu (*) · W. Zou · C. Li · Y. Shi · Z. Pan · X. Pan

The University of Texas at Austin, Austin, TX, USA

https://doi.org/10.1007/978-3-319-64792-0_15
276 M. Liu et al.
& Marcinkowski, 2016), and predicting learning performances and outcomes

(de Barba, Kennedy, & Ainley, 2016; Elbadrawy et al., 2016; Formanek, Wenger,
Buxner, Impey, & Sonam, 2017; Tseng et al., 2016).
1.2 Learner Behavioral Patterns in MOOCs
To understand learners’ behavioral patterns, Tseng et al. (2016) used students’
behavioral data from three courses from a university in Taiwan to identify learners’
engagement and learning patterns in MOOCs. The results of cluster analyses found
that only 1% of students were classified as active learners, while 90% of students
were bystanders, and 9% of students were passive learners. Another interesting find-
ing was that students were relatively more active during the first 2 weeks; thus, it
was a crucial point of time to keep students motivated and engaged in MOOCs.
Similarly, Kizilcec et al. (2017) conducted a study to examine various types of self-
regulated learning (SRL) strategies when learners were interacting with MOOCs.
They collected data on learner events within a certain period of time (defined as
“sessions”) to look at their sequential behaviors and the frequency of transitions
between different sessions. Additionally, survey data were collected to identify
learner characteristics. Results revealed that students with high levels of self-
reported SRL were more likely to go back to course materials, especially course
assessments, to repeatedly test themselves. In another study, Khalil and Ebner
(2016) presented the implementation of a learning analytics prototype in an Austrian
MOOC platform to analyze learners’ behavioral patterns. Their learning analytics
tool kept track of various learner interactions, such as the logging frequency, the
total of document downloads, forum posts from each learner, video statistics, and
total of quiz attempts. The researchers applied this tool to examine the logs of two
different MOOCs. Results showed that students were much more active (as demon-
strated in video views, forum posts, forum reads) during the first 2 weeks of the
courses. They also analyzed course scores with student activities in the forum and
found that students who read more forum posts may not necessarily achieve higher
scores, as their performances were also influenced by other factors, such as what
specific content they read in the discussion forum.
Several studies have been done to classify learners based on the level and content
of their interaction with other learners (Gillani & Eynon, 2014; Goggins, Galyen,
Petakovic, & Laffey, 2016; Khalil & Ebner, 2017; Yang, Wen, Howley, Kraut, &
Rose, 2015). Khalil and Ebner (2017), for example, used clustering methods to clas-
sify students into appropriate categories based on their level of engagement. Their
algorithm fetched user behavioral data such as viewed videos, downloaded files,
reading in forums, posting in forums, quiz results, and logins. Three groups were
identified with distinctive access patterns: “gaming the system group,” “perfect stu-
dent group,” and “dropout group.” Their study implied that besides motivating
learners extrinsically like providing certificates or badges, course designers should
also pay attention to improving learners’ intrinsic motivation, such as refining the
instructional design and course content itself, in order to make students progress
positively toward the “perfect student group.”
15 Using Learning Analytics to Examine Relationships Between Learners’ Usage… 277
1.3 Students’ Engagement and Motivation in MOOCs
Implementing analytics into MOOCs can contribute to improving students’ engage-

ment and enhance learning outcomes. Researchers had investigated ways to increase
engagement and diagnose issues through the algorithms of learning analytics (Lu
et al., 2017; Xing et al., 2016). For example, Lu et al. (2017) investigated the effec-
tiveness to increase students’ engagement of MOOCs by involving analytics to a
programming course. They adopted a parallelized action-based engagement mea-
surement algorithm (PAbA) which measured students’ levels of engagement through
calculating their video views and forum posts. The instructor would receive report
and provide intervention when a student’s level of engagement was lower than a
certain threshold. Their results showed that the monthly reports sent to instructors
increased their timely interventions, which ultimately led to higher levels of behav-
ioral, cognitive, and emotional engagement in students. This study indicated that it
was crucial for instructors to have easy access to students’ online behavioral data
and gave nudges to those who were falling behind the schedule.
Likewise, Xing et al. (2016) presented a temporal modeling approach for the
early and accurate identification of students at risk of dropping out. Students were
labeled based on their level of activeness for each week of the course, which were
associated with the weekly course feature vectors: the number of discussion post,
number of forum views, number of quiz views, number of module views, number of
active days, and frequency of interactions in social network. The data for each indi-
vidual were then input into the machine learning algorithms to predict the dropout
label for a specific week for that student. The study proved how prediction models
can be adopted effectively to improve the quality of intervention and increase stu-
dent engagement.
1.4 Predicting Learning Performances and Outcomes
A main reason for applying learning analytics in MOOCs is to predict students’

learning performances and outcomes through the data generated through students’
learning progress and learning styles (Nunn, Avella, Kanai, & Kebritchi, 2016;
U.S. Department of Education, Office of Educational Technology, 2012).
Researchers had implemented series of learning analytics models and techniques to
gain more insights on students’ success in MOOCs (de Barba et al., 2016; Elbadrawy
et al., 2016; Formanek et al., 2017; Pursel, Zhang, Jablokow, Choi, & Velegol, 2016;
Qu & Chen, 2015; Tseng et al., 2016). For instance, Pursel et al. (2016) conducted
a study on MOOC to find out what variables were indicative of student completion.
They used regression analysis to examine the correlations among completion rate,
video views per week, posts per week, and comments per week in the discussion
forums. Results showed that prior online learning experience in MOOCs had no
impact on student completion. Video views were found to be a strong predictor of
MOOC completion. Although the forums were not graded, forum posts and com-
ments were significant predictors of completion.
278 M. Liu et al.
Tseng et al. (2016) performed an agglomerative hierarchical clustering analysis

and combined the results from chi-square and one-way ANOVA tests to measure
learning performances among different types of learners. The research found posi-
tive correlation between students’ video and assignments completing frequencies
and their passing rate. Similarly, the study of Qu and Chen (2015) showed that stu-
dents with various types of learning behaviors in MOOCs revealed different levels of
learning outcome. Active learners who completed homework assignments and
watched videos frequently showed significantly higher rates of passing the courses
than the others. It indicated the connection between students’ learning motivation
and learning outcomes, which corresponded with a research by de Barba et al.
(2016). In their study, they measured students’ video hits and quiz attempts during
learning as an indicator of students’ participation and then used Spearman correla-
tions to find out the correlations between the factors that indicated types of motiva-
tion and the students’ behaviors. The results showed that students’ intrinsic motivation
and participation were the significant predictors of final learning performances, and
both of them relate to students’ MOOC learning outcomes directly and indirectly.
Moreover, several researches had focused specifically on learning analytics
model itself as to develop its ability in identifying key performance indicators
(Maté, De Gregorio, Cámara, Trujillo, & Luján-Mora, 2016; Peral, Maté, & Marco,
2017). Peral et al. (2017) proposed a five-step data processing model which involved
support vector machines (SVM) and decision trees (DT) to refine existing key per-
formance indicator model so as to better predict students’ learning performances
and outcomes.
2 Purpose of the Study and Research Questions
Given the potential benefits of using analytics, more research is needed to investi-
gate the relationships between behavioral patterns and learner engagement and to
understand how learners’ usage connected to their profiles and perceptions. Little
research is available on this topic for MOOCs designed for working professionals.
In this study, we asked the following research questions:
1. How do MOOC participants access various key MOOC components (e.g., dis-
cussion forums, readings, videos, quizzes, optional resources)?
2. Is there a connection between participants’ usage patterns and learner profiles
(i.e., gender and prior MOOC experience) and their perceptions?
3 Method
3.1 Participants and Research Context
Participants were the students enrolled in a 6-week MOOC offered through the
Knight Center for Specialized Journalism in the Americas at the University of Texas
at Austin in spring 2017. This MOOC was designed primarily for journalism
professionals but also were open to everyone interested and had a total of 6675
individuals registered, representing 148 countries, and 2 instructors taught this
MOOC.
3.2 Data Sources and Analysis
Two types of data sources were included in this study: (a) the survey responses
regarding participants’ perceptions of the usefulness of various course components
and (b) the usage data.
Survey Data A post-survey was administered by the end of the course. The survey
was designed to help capture participants’ demographics and their perceptions
related to the research questions of this study. Eight questions were asked, and the
responses were categorized into two or more groups for each question:
1 . What is your gender (male or female)?

2. Have you taken MOOC courses before (first time or non-first time)?
3. Of the various reasons you mentioned, which was the most important reason you
enrolled?
4. Approximately, how many hours per week did you spend on this MOOC (1 h
less or more than 1 h)?
5. How many course-related exercises and assignments have you completed (some/
all or none)?
6. Indicate your MOOC completion status and whether you plan to apply for cer-
tificate (have applied/will apply or will not apply).
7. Compared to face-to-face instruction, is this MOOC instruction better, worse, or
about the same?
8. To what extent did you find discussion forums of this MOOC course to be help-
ful (helpful, neutral, or not helpful)?
Usage Data Usage data came from user-generated logs from the MOOC course
platform—Moodle. A challenge we had when collecting the log data was Moodle’s
incapability to handle large-quantity data download requests. There were 615,328
rows of log data for this course, and a crash would happen if we used the administer
tool provided by Moodle to download the data. Alternatively, we implemented a
web scraping script with a Python package, called Scrapy, to collect the data. During
the data collecting process, we were cautious of data loss caused by network request
failures. Failed network requests would be retried for ten times. If network requests
still failed after ten times, an error would return to prompt us, so we could decide
what to do next. Fortunately, the scraping process was performed smoothly, and we
were able to get all the available data.
Data Cleaning and Transformation First, irrelevant raw log data to our research
questions were removed. The final dataset consisted of 301,010 rows. However, the
raw log data did not offer us the granularity we needed for this study such as module
information on various key course components (discussion forums, quizzes, videos,
280 M. Liu et al.
readings, and optional resources); or the format was in such a way it was hard to tell
which log was about videos, readings, or optional resources. For example, all log
events were categorized with a general label of “url component.” After browsing the
MOOC course web page, we found that each page has a unique ID associated with
each page. Using the unique IDs, we manually collected these page IDs and con-
structed a dictionary with page ID as the key and module number as the value using
a Python script. This dictionary allowed us to trace the module information that was
obviously displayed from the raw data. We constructed another dictionary for vari-
ous course components (e.g., videos, readings, and optional resources) that helped
transform log data with the details we needed for this study.
Matching User’s Log Data with Their Survey Data A total of 578 participants com-
pleted the survey. The survey was anonymous. We were interested in understanding
the connection between participants’ survey responses and their usage data (RQ2).
Using IP addresses, we matched respondents’ survey data with their data logs. As a
result, a total of 200 unique IP addresses were found during the 6-week course with
a total of 26,327 rows of data for further analysis. A row of log data consisted of the
participant’s IP address, the date and time, and a series of course events.
Given our research questions, nine events related to the course components were
extracted and analyzed (see Table 15.1 for each event name and its definition).
These events included can be categorized into active and passive learning behaviors.
Active learning behaviors indicate students’ active participation in the course, such
as discussion created and quiz started. Passive learning behaviors, on the other
hand, represent students’ passive receiving of course materials, such as video
viewed and reading materials viewed.
Data Analysis We first plotted graphs to examine the data descriptively. There were
eight survey questions and five course components (discussion forums, quizzes,
videos, readings, and optional resources) in the log data. Each component contained
Table 15.1 Definitions of events in user log file

Event name Definition
Discussion created The number of times students created a new thread under a discussion
question
Discussion The number of times students created an email subscription to a
subscription created discussion thread
Discussion viewed The number of times students accessed a discussion thread
Discussion replied The number of times students replied to other students’ post
Reading materials The number of times students accessed reading materials provided by
viewed the instructor
Video viewed The number of times students accessed instructional videos in the
course
Optional resources The number of times students accessed optional materials provided by
viewed the instructor, including videos and readings
Quiz started The number of times students started a quiz
Quiz submitted The number of times students finished and submitted a quiz
different events. For example, discussion forums included events such as discussion
viewed and discussion replied. Drawing graphs for each of these components were
labor intensive. To make the process more efficient, a Python script was written
using packages of Seaborn and Matplotlib to automatically plot the graphs. Seaborn
and Matplotlib are graphing packages in Python, which allow users to plug in in
variables to plot different graphs easily. In our case, we wrote a script with Seaborn
and Matplotlib to generate bar charts of events’ frequency in each component. We
first tested several auto-generated graphs and checked if they matched the manually
plotted graphs and found the automatically drawn graphs matched manually drawn
ones 100%. The research team examined each graph for its accuracy before includ-
ing them in the analysis. When missing data or data ranges were incorrectly dis-
played, we reexamined the input data for the graphs and redrew the graphs. These
graphs along with their data points were then used to analyze the overall usage
patterns, and descriptive statistics were used. We also used the graphs to help us
decide which grouping variable(s) that might show differences to further investigate
using statistics.
Because the data did not meet all assumptions of parametric hypothesis testing,
such as normality, the Mann-Whitney U and Kruskal-Wallis H tests were used to
compare differences between the groups of similar sizes. In two cases, the sizes of
the groups were very different from each other, and descriptive statistics were used
instead. For all cases, data visualizations, using Tableau, were used to create graphs
to illustrate the findings, which allowed us to plot multiple data points in one graph
(see Fig. 15.3 below).
4 Findings
4.1 Overall Usage Patterns

4.1.1 Passive Learning Behaviors
Figure 15.1 showed the overall video viewed, reading materials viewed, and optional
resources viewed across 6 weeks from all participants (n = 6675). In general, the
frequency of views of these three course components was gradually declining.
Particularly, for optional resources viewed and reading materials viewed, the most
precipitous drop occurred in Week 2 (a drop around 6800 from Week 1 for each). By
contrast, for video viewed, there was a slight increase from Week 1 to Week 2, fol-
lowed by a sharp drop of 6248 in Week 3. Since Week 3, video viewed decreased
steadily (a weekly drop around 3000) until it went down to 4074 by Week 6. For
optional resources viewed, the number continued to drop until Week 5 (a slight
increase of 522 from Week 4). By Week 6, the number went up slightly to 1313. For
reading materials viewed, despite of its continuous declining trend, the decrease
was much less dramatic in Week 5 (a mild drop of 363).
282 M. Liu et al.
Fig. 15.1 Overall videos, readings, and optional resources viewed across 6 weeks
4.1.2 Active Learning Behaviors
Similarly, the events in quizzes experienced the same downward trend (Fig. 15.2).
In particular, the trend of quiz submitted closely followed that of quiz started. It is
worth noting that the number of quiz submitted was consistently fewer than quiz
started since participants may not finish every quiz they had started. Again, Fig. 15.2
showed that the most significant decrease for both events happened in Week 2 (an
average drop around 650 from Week 1), followed by more steady drops in Week 3
and Week 4. There was a noticeable uplift in Week 5 (an average increase around 90
for both events). Nevertheless, they quickly resumed their declining trend by Week 6.
For events in discussion forums (Fig. 15.3), the decline of discussion viewed
(from 10,313 in Week 1 to 1337 in Week 6) was much more conspicuous than the
rest of the three events: discussion created, discussion subscription created, and
discussion replied. The most dramatic drop for discussion viewed occurred in Week
2 and Week 3, with a plummet of 4867 and 2200, respectively, from their previous
weeks, followed by a slower decreasing trend for the rest of the course. For discus-
sion created, discussion subscription created, and discussion replied, the most
noticeable drop happened in Week 3 (discussion created, −197; discussion sub-
scription created, −359; discussion replied, −202). During the following weeks, the
frequency for these three events continued to fall but in a much lower speed.
Could the decline of usage be due to variations of the course materials provided?
Table 15.2 presented the number of course materials provided in the MOOC for
each week. For videos, the number of videos provided almost doubled after Week 1,
yet the frequency of video viewed steadily declined from Week 2, as shown in
Fig. 15.1. Similarly, reading materials viewed were observed in a downward trend
although a similar number of reading materials were provided. For optional
resources, the drop the frequency of views may partly be due to that fact that less
optional resources were provided after Week 1. However, the optional resources in
Fig. 15.2 Overall usage of quizzes across 6 weeks
Fig. 15.3 Overall usage of discussion forums across 6 weeks
Table 15.2 Number of course materials provided in the MOOC for each week
Videos provided Reading materials provided Optional resources provided
Week 1 5 5 8
Week 2 9 8 4
Week 3 11 4 3
Week 4 10 4 2
Week 5 10 6 3
Week 6 7 4 6
Week 2 was half of that in Week 1 in quantity, for example, yet the number of views
in Week 2 shrunk to only 30% of the views observed in Week 1. That is, that the
declines of video viewed, reading materials viewed, and optional resources viewed
appeared to be attributed to reduced usage rather than less course content provided
in subsequent weeks.
284 M. Liu et al.
4.2 C
onnection Between Usage Patterns and Participants’
Survey Responses
Given participants’ responses for each question, we categorized them into different
groups to find out how participants from different groups accessed the MOOC. The
means and standard deviations of frequency of events for each group were presented
in Table 15.3.
4.2.1 Gender and Prior MOOC Experience
In order to examine whether there was a difference in MOOC usage between gen-
ders, Mann-Whitney U tests were conducted between male (n = 77) and female
(n = 108) groups. The analysis revealed significant differences in quiz started
(z = −2.361, p < 0.05) and quiz submitted (z = −2.128, p < 0.05). There were no
other significant differences for other events (Fig. 15.3).
Regarding participants’ familiarity with the form of MOOC learning, we catego-
rized them into two groups based on their responses: (a) first-time MOOC users
(n = 81) and (b) non-first-time MOOC users (n = 104) who have taken other MOOC
courses before. Mann-Whitney U tests were conducted to determine whether there
were significant differences in the frequency of events between these two groups.
Results of analysis found significant difference only in discussion viewed
(z = −2.005, p < 0.05).
4.2.2 Time Spent and Assignment/Exercise Completion
In looking at usage patterns of the participants who spent more time or completed
more exercises/assignments, we examined descriptively the mean frequency of the
nine events performed by each group of participants, given the quite uneven sample
size between groups. The analysis showed more usage of different course compo-
nents by those who spent more time or completed more exercises/assignments than
those who contributed less efforts (see Table 15.3).
4.2.3 Most Important Reason to Enroll and Certificate Applying Plan
We asked the participants to indicate the most important reason for them to enroll in
this MOOC. The results showed varied responses regarding their use of discussion
forums (see Table 15.4). Participants who were motivated by earning a certificate/
statement of accomplishment were most active across four different types of events
in discussion forums compared to those who chose other reasons to enroll. Those
who chose general interest in the topic and relevant to academic research exhibited
low participation in discussion forums. Additionally, we further categorized the par-
ticipants into two groups based on the reasons they indicated: (a) intrinsically moti-
vated group (n = 82) that contained participants who chose general interest in topic,
Table 15.3 Usage patterns for each group
Optional
Discussion Quizzes resources Readings Videos
Discussion Optional Reading
Discussion subscription Discussion Discussion Quiz Quiz resources materials Video
Groupings created created viewed replied started submitted viewed viewed viewed
Mean Mean Mean
Categories Mean (SD) Mean (SD) Mean (SD) Mean (SD) (SD) (SD) Mean (SD) Mean (SD) (SD)
Gender Male n = 77 1.77 (3.09) 3.16 (5.28) 16.87 1.96 (3.37) 5.04a 4.81a 5.47 (6.17) 13.88 34.87
(24.73) (3.9) (3.94) (12.37) (25.92)
Female n = 108 1.54 (2.71) 2.39 (4.15) 16.83 1.82 (4.99) 3.88a 3.77a 5.43 (7.22) 12.50 30.30
(29.01) (4.97) (4.98) (13.17) (30.41)
Previous MOOC First time n = 81 1.72 (3.02) 3.10 (4.54) 19.17a 1.94 (3.79) 4.94 4.81 5.09 (6.72) 14.74 37.79
experiences (27.04) (5.42) (5.44) (14.12) (32.48)
Non-first time 1.59 (2.77) 2.43 (4.76) 15.20a 1.85 (4.83) 3.95 3.76 5.76 (6.88) 11.89 28.13
n = 104 (27.5) (3.77) (3.77) (11.65) (24.58)
Most important Intrinsically 1.00a (2.32) 1.59a (2.85) 11.62a 0.93 (1.56) 3.74 3.55 8.39 14.34 33.89
reason to enroll in motivated n = 82 (17.12) (3.73) (3.80) (10.24) (13.78) (31.97)
MOOC Extrinsically 1.92a (2.95) 3.35a (5.28) 22.99a 2.20 (4.25) 4.94 4.87 8.71 17.55 40.65
motivated n = 99 (35.08) (5.21) (5.23) (10.57) (16.42) (34.20)
Hours per week 1 h or less n = 35 0.14 (0.55) 0.54 (1.75) 2.06 (5.46) 0.49 (1.56) 1.20 1.11 2.77 (4.95) 5.06 (6.62) 14.94
spent on MOOC (2.11) (2.07) (21.59)
More than 1 h 1.99 (3.08) 3.23 (4.98) 20.41 2.21 (4.77) 5.13 4.95 6.09 (7.03) 15.03 36.43
n = 150 (29.14) (4.69) (4.72) (13.21) (28.6)
Exercise and None n = 13 0.08 (0.28) 0.15 (0.38) 1.31 (2.46) 0.08 (0.28) 0.54 0.46 2.46 (4.45) 5.77 (7.97) 13.62
assignment (1.45) (1.39) (14.64)
completion status Some or all n = 172 1.76 (2.95) 2.92 (4.78) 18.12 2.02 (4.53) 4.67 4.51 5.69 (6.9) 13.7 33.78
15 Using Learning Analytics to Examine Relationships Between Learners’ Usage…
(27.95) (4.61) (4.63) (12.97) (28.96)

(continued)
285
Table 15.3 (continued)
286
Optional
Discussion Quizzes resources Readings Videos
Discussion Optional Reading
Discussion subscription Discussion Discussion Quiz Quiz resources materials Video
Groupings created created viewed replied started submitted viewed viewed viewed
Mean Mean Mean
Categories Mean (SD) Mean (SD) Mean (SD) Mean (SD) (SD) (SD) Mean (SD) Mean (SD) (SD)
Certificate Completed MOOC, 2.82a (3.64) 4.71a (5.56) 24.74a 2.65a (3.65) 5.53 5.43 5.86 (6.99) 15.72 38.93
application plan going to apply or (28.29) (4.05) (4.04) (13.64) (28.49)
received the
certificate n = 72
Completed MOOC 1.28a (2.25) 2.18a (4.44) 16.21a 2.21a (6.37) 5.46 5.33 5.93 (6.41) 15.02 39.89
but do not plan to (29.78) (5.62) (5.62) (13.59) (31.14)
apply for the
certificate n = 57
Perception toward MOOC is much 1.78 (3.69) 3.02 (5.74) 18.42 1.64 (3.11) 3.82 3.78 6.40 (7.66) 13.96 29.71
MOOC compared better or somewhat (29.59) (4.29) (4.26) (13.81) (26.9)
to face-to-face better n = 45
instruction
Perception toward MOOC is about the 2.07 (2.95) 3.38 (4.96) 17.39 2.14 (3.89) 5.77 5.66 5.09 (6.04) 13.57 36.73
MOOC compared same n = 56 (25.29) (5.61) (5.59) (12.91) (31.5)
to face-to-face MOOC is 1.66 (2.73) 2.34 (4.13) 16.85 2.37 (6.73) 3.59 3.27 4.83 (6.19) 11.66 31.41
instruction somewhat worse or (31.5) (3.54) (3.54) (11.39) (28.49)
much worse n = 41
Perception toward Helpful n = 57 1.65 (3.69) 3.47 (7.34) 14.06 1.88 (3.85) 4.12 4.18 4.00 (4.46) 11.24 30.29
discussion forums (31.85) (3.9) (3.89) (12.44) (22.61)
Neutral n = 52 1.78 (2.88) 2.83 (4.16) 16.47 1.88 (2.5) 4.52 4.45 5.03 (6.06) 13.65 34.47
(20.81) (3.96) (3.91) (12.79) (26.24)
Not helpful n = 17 2.11 (3.25) 3.44 (5.08) 22.14 2.60 (6.29) 4.78 4.43 5.86 (7.3) 14.16 33.67
(33.48) (5.66) (5.74) (13.41) (32.95)
M. Liu et al.
a
Significantly different from the other(s), p < 0.05
Table 15.4 Events in discussion forums in relation to participants’ reasons to enroll in the MOOC
Intrinsic motivation Extrinsic motivation
For
General personal For Relevant to
interest in the growth and For fun and Earn a career Relevant to school or degree
topic enrichment challenge certificate change academic research Relevant to job program
Avg. discussion created 0 1 1 5 1 1 2 2
Avg. discussion 1 2 2 4 3 0 4 3
subscription created
Avg. discussion viewed 8 14 16 38 28 6 23 17
Avg. discussion replied 0 1 1 5 2 0 3 1
15 Using Learning Analytics to Examine Relationships Between Learners’ Usage…
287
288 M. Liu et al.
for personal growth and enrichment or for fun and challenge. We considered this
group to enroll in MOOC purely out of their own will and interest without expected
external rewards; and (b) extrinsically motivated group (n = 99) that included par-
ticipants who stated one of the following reasons: relevant to school or degree pro-
gram, for career change, relevant to job, relevant to academic research, or to earn a
certificate/statement of accomplishment. We considered this group to enroll in
MOOC due to some external factors. Mann-Whitney U tests revealed significant
differences between the intrinsic motivation and extrinsic motivation groups for
three types of events: discussion created (z = −2.300, p < 0.05), discussion subscrip-
tion created (z = −2.373, p < 0.05), and discussion viewed (z = −1.962, p < 0.05).
However, there was no significant difference in discussion replied between these
two groups or other events.
Mann-Whitney U tests were conducted to determine whether there were signifi-
cant differences in the usage of discussion forums between the group who had com-
pleted MOOC and planned to apply for certificate (n = 72) and the group who
completed the MOOC but did not plan to apply for certificate (n = 57). Results of
analysis found significant differences in all four events of discussion forums: dis-
cussion created (z = −3.014, p < 0.05), discussion subscription created (z = −4.008,
p < 0.05), discussion viewed (z = −2.520, p < 0.05), and discussion replied
(z = −2.677, p < 0.05).
4.2.4 Perceptions
To examine how participants’ perceptions toward MOOC affected their usage of the
MOOC, a Kruskal-Wallis H test was performed to see whether there were differ-
ences among three groups: (a) the group that thought MOOC was much better or
somewhat better than face-to-face instruction (n = 45), (b) the group that thought
MOOC was about the same with face-to-face instruction (n = 56), and (c) the group
that thought MOOC was somewhat worse or much worse (n = 41). The results
showed that there was no statistically significant difference in any events among
those three groups.
The participants were also asked about their perceptions of helpfulness of the
discussion forums for learning. Their responses were categorized into three groups:
(a) the group who found discussion forums to be helpful (n = 57), (b) the group
whose perception was neutral (n = 52), and (c) the group who found discussion
forums to be unhelpful (n = 17). A Kruskal-Wallis H test was conducted in an effort
to find statistically significant differences among these three groups in the events of
discussion forums. No significant difference was found.
Apart from the overall usage patterns as presented above, we then further exam-
ined the daily usage of discussion forums by the three groups. The participants’
daily usage data showed that among the four events, postings to the discussion
forums had a lower frequency compared to participants’ viewing and subscription
of the discussion forums. Overall, the participants who rated discussion forums as
helpful showed more usage than those in the other two groups (see Fig. 15.4). This
Fig. 15.4 Events in discussion forums in relation to participants’ perceptions toward discussion
forums
290 M. Liu et al.
active usage seemed to continue throughout the course. The total number of partici-
pants in the group that found forums unhelpful was much smaller. It is worth notic-
ing that although the number of people in group that considered discussion forums
neutral was slightly more than three times of that of the group finding forums
unhelpful, and the usage pattern of discussion forums of these two groups was simi-
lar. However, during the first 2 weeks the group considering discussion forums neu-
tral appeared more active than the group finding forum unhelpful across all four
events in discussion forums (see Fig. 15.4). But again, this could be caused by the
difference on the number of people between the two groups.
5 Discussion and Conclusion
In this study, we examined how MOOC participants used various course compo-
nents and whether and how their usage patterns connected to their perceptions. The
course components we were interested in were the use of discussion forums, read-
ings, videos, quizzes, and optional resources, which were the key components this
MOOC platform offered, and the eight grouping categories are (a) gender, (b) previ-
ous MOOC experience, (c) most important reason to enroll in MOOC, (d) hours per
week spent on MOOC, (e) exercise and assignment completion status, (f) certificate
application plan (whether they plan to apply for certificate), (g) perceptions toward
MOOC compared to face-to-face instruction, and (h) perceptions toward discussion
forums (whether they consider it useful).
5.1 Overall Usage Patterns
The results of the overall behavioral patterns showed there was an overall decline
from the first week to the last week in accessing the various course components, and
the biggest drop occurred in Week 2. This declining trend was consistent with other
research (Khalil & Ebner, 2016; Qu & Chen, 2015; Tseng et al., 2016) showing
students were more active during the first 2 weeks, an important point for MOOC
instructors to keep in mind. This result suggests the critical point of time to retain
students in MOOCs is during the first 2 weeks. This finding is especially important
in light of the high dropout rates found in the MOOC literature. Of all course com-
ponents provided, it is understandable that optional resources were used the least.
Interestingly, the overall usage of videos provided was much higher than the use of
readings provided. This could mean that of the three course components (readings,
videos, and optional resources), the participants found the videos more useful or
engaging, which is probably because of the presence of instructors in the videos to
present teaching in a more interactive way.
5.2 Usage Patterns by Grouping Categories
In examining the usage patterns by eight grouping categories based upon partici-
pants’ survey responses, the results revealed some interesting findings. Although
there was an overall decline trend from week to week, male participants consistently
accessed significantly more course components than female participants.
Specifically, male participants accessed quizzes significantly more than female par-
ticipants, in both events quiz started and quiz submitted. First-time MOOC users
had significantly higher frequency of discussion viewed. It is worth noting that there
were no significant differences in discussion created and discussion replied, two
more active actions between first-time MOOC users and non-first-time MOOC
users. This finding corresponds with a previous study (Pursel et al., 2016) in which
users’ prior MOOC experience had no significant impact on posts and comments
created in discussion forums.
Looking at the usage patterns by the time the participants spent per week, the
findings showed those who spent more time per week accessed more course compo-
nents than those who spent only 1 h or less. Those who completed some or all exer-
cises and assignments accessed more course components than those who did not
complete exercises and assignments. Such findings are expected, as if one spent
more time in the MOOC, it was most likely they were using the MOOC components
as indicated by the analytics. These findings are aligned with previous research,
which found self-regulation ability to be closely related to students’ level of engage-
ment in MOOCs (Lu et al., 2017).
We examined the most important reason to enroll in the MOOC by the partici-
pants. The findings showed that students who wanted to earn a certificate were more
active in the events in discussion forums, especially discussion created and discus-
sion replied, which make sense since certificate requirement of the course included
participation in discussion forums. Students who took the course because it was
relevant to their current jobs or degree programs or for career change also showed
more frequent usage of the discussion forums. We further investigated the usage
patterns by participants in two categories: those who enrolled for reasons showing
intrinsically motivated (e.g., general interest in topic, for personal growth and
enrichment, and for fun and challenge) and those who enrolled for reasons related
to external factors or extrinsically motivated (e.g., relevant to school or degree pro-
gram, for career change, relevant to job, relevant to academic research, and earning
a certificate/statement of accomplishment). The results showed that while there
were no significant findings in usage patterns for most of the course components,
there was one noticeable and significant finding between those enrolled in MOOC
for intrinsic reasons and those enrolled for extrinsic reasons: the use of the discus-
sion forums. The participants extrinsically motivated had significantly higher fre-
quency in discussion created, discussion subscription created, and discussion
viewed. The discussion forums is the most interactive component of this MOOC; a
higher frequency is an indication of higher engagement. This finding supports other
research that showed motivation were significant predictors of student engagement
in the course (Xiong et al., 2015). However, there was no significant difference
292 M. Liu et al.
between the two groups in discussion replied. This finding is further supported by
the analysis of usage patterns on MOOC certificate status: those who indicated they
completed MOOC and were going to apply or already received the certificate used
discussion forums significantly more than those who completed the MOOC but did
not plan to apply for the certificate. This suggests providing more extrinsically moti-
vating factors may help some participants complete a MOOC and use course com-
ponents more often (Xiong et al., 2015). By contrast, previous research showed that
providing a free completion certificate had no impact on students’ completion rate
(Impey, Wenger, & Austin, 2015).
While there were no significant differences in usage patterns between the groups
considering discussion forums helpful, neutral, or not helpful in the survey, the
“helpful” group showed a higher frequency of overall usage in discussion forums,
although this difference was not statistically significant. A surprising finding was
there were no significant differences in usage patterns between the groups consider-
ing if the MOOC was better, same, or worse when compared to face-to-face instruc-
tion. Those who considered that the MOOC was about the same as face-to-face
instruction consistently accessed more course components than the other two
groups. This seems to suggest the delivery platform, face to face or online, did not
affect how participants accessed the course. More research is needed.
This MOOC was designed for working professionals, and assessments such as
exercises, assignments, and quizzes were offered to help participants learn and
practice what they learned, not to give grades. The analysis of participants’ behav-
iors offered a snapshot of how these working professionals accessed the course
components in this setting and what components were important to these partici-
pants given their motivation to enroll in the MOOC. Research has indicated partici-
pation in a MOOC could be a strong predictor of performance (de Barba et al., 2016;
Xiong et al., 2015). Findings of this study revealed factors connected or not con-
nected to their participation as reflected in the analytics. De Barba et al. (2016) also
indicated that students’ intrinsic motivation was a significant predictor of final learn-
ing performances. The finding of this study suggests that students’ extrinsic motiva-
tion is also very important for MOOCs such as this one where participants enrolled
to gain new knowledge and skills related to their jobs or careers. Some significant
differences were found between several group categories as discussed. However,
these differences occurred were in passive learning behaviors (e.g., discussion
viewed), not active learning behaviors (e.g., discussion created). The frequencies of
passive actions were higher in most cases. This finding is in line with research show-
ing that 90% of students were passive learners and bystanders (Qu & Chen, 2015).
6 Limitations of the Study
This study is limited in that the survey was given out anonymously, and due to vol-
untary nature of the survey, not all participants took the survey. Therefore, the data
for this study was limited to those who completed the survey. Additionally, given
the anonymous nature of the surveys, in order to match participants’ usage data and
their survey responses, we used participants’ IP addresses which were traced to a
computer but not a person. This reflects the challenges of analyzing log file data. On
the one hand, log data provides a more truthful picture of how users access various
components of a computer system. At the same time, matching user characteristic
data with their behavioral patterns anonymously remains to be a challenge for
researchers who are interested in making sense of log data. Finally, the findings of
this study are confined to the MOOC under study. Readers should not assume gen-
eralizability to other MOOCs.
7 Conclusion
There have been increasing interests in recent years by researchers and practitioners
in examining and using analytics to support teaching and learning. In this study, we
investigated the analytics of the participants in a MOOC designed for working pro-
fessionals. While the results of this study provided additional evidence to support
previous research, a few findings are shown to merit additional attention. Most
importantly, extrinsic motivation factors are important for MOOC participants,
especially working professionals, who seek to learn new knowledge and skills from
MOOCs on contemporary topics. Given high dropout rate typically happens during
the first 2 weeks, MOOC instructors should consider providing some interventions
or incentives to motivate participants to continue. Future research should further
examine the usage patterns by those extrinsically and intrinsically motivated. We
hope this study offers useful insights for MOOC instructors and designers as they
create and offer MOOC courses.
References
de Barba, P. G., Kennedy, G. G., & Ainley, M. D. (2016). The role of students’ motivation and
participation in predicting performance in a MOOC. Journal of Computer Assisted Learning,
32(3), 218–231. https://doi.org/10.1111/jcal.12130
Elbadrawy, A., Polyzou, A., Ren, Z., Sweeney, M., Karypis, G., & Rangwala, H. (2016). Predicting
student performance using personalized analytics. Computer, 49(4), 61–69. https://doi.
org/10.1109/MC.2016.119
Formanek, M., Wenger, M. C., Buxner, S. R., Impey, C. D., & Sonam, T. (2017). Insights about
large-scale online peer assessment from an analysis of an astronomy MOOC. Computers &
Gillani, N., & Eynon, R. (2014). Communication patterns in massively open online courses. The
Internet and Higher Education, 23, 18–26. https://doi.org/10.1016/j.iheduc.2014.05.004
Goggins, S. P., Galyen, K. D., Petakovic, E., & Laffey, J. M. (2016). Connecting performance to
social structure and pedagogy as a pathway to scaling learning analytics in MOOCs: An explor-
atory study. Journal of Computer Assisted Learning, 32(3), 244–266. https://doi.org/10.1111/
jcal.12129
294 M. Liu et al.
Impey, C. D., Wenger, M. C., & Austin, C. L. (2015). Astronomy for astronomical numbers:
A worldwide massive open online class. International Review of Research in Open and
Distributed Learning, 16(1), 57–79. Retrieved from https://eric.ed.gov/?id=EJ1061069
Khalil, M., & Ebner, M. (2016). What massive open online course (MOOC) stakehold-
ers can learn from learning analytics? ArXiv:1606.02911 [Cs], 1–30. https://doi.
org/10.1007/978-3-319-17727-4_3-1
Khalil, M., & Ebner, M. (2017). Clustering patterns of engagement in massive open online courses
(MOOCs): The use of learning analytics to reveal student categories. Journal of Computing in
Higher Education, 29(1), 114–132. https://doi.org/10.1007/s12528-016-9126-9
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strate-
gies predict learner behavior and goal attainment in massive open online courses. Computers &
Lu, O. H. T., Huang, J. C. H., Huang, A. Y. Q., & Yang, S. J. H. (2017). Applying learning analytics
for improving students’ engagement and learning outcomes in an MOOCs enabled collaborative
programming course. Interactive Learning Environments, 25(2), 220–234. https://doi.org/10.1
080/10494820.2016.1278391
Maté, A., De Gregorio, E., Cámara, J., Trujillo, J., & Luján-Mora, S. (2016). The improvement of
analytics in massive open online courses by applying data mining techniques. Expert Systems,
33(4), 374–382.
Nunn, S., Avella, J. T., Kanai, T., & Kebritchi, M. (2016). Learning analytics methods, benefits,
and challenges in higher education: A systematic literature review. Online Learning, 20(2), 13.
https://doi.org/10.24059/olj.v20i2.790
Peral, J., Maté, A., & Marco, M. (2017). Application of data mining techniques to identify relevant
key performance indicators. Computer Standards & Interfaces, 50, 55–64.
Pursel, B. K., Zhang, L., Jablokow, K. W., Choi, G. W., & Velegol, D. (2016). Understanding
MOOC students: Motivations and behaviours indicative of MOOC completion. Journal of
Computer Assisted Learning, 32(3), 202–217.
Qu, H., & Chen, Q. (2015). Visual analytics for MOOC data. IEEE Computer Graphics and
Tseng, S. F., Tsao, Y. W., Yu, L. C., Chan, C. L., & Lai, K. R. (2016). Who will pass? Analyzing
learner behaviors in MOOCs. Research and Practice in Technology Enhanced Learning, 11(1),
8. https://doi.org/10.1186/s41039-016-0033-5
U.S. Department of Education, Office of Educational Technology. (2012). Enhancing teaching and
learning through educational data mining and learning analytics: An issue brief. Retrieved
from https://tech.ed.gov/wp-content/uploads/2014/03/edm-la-brief.pdf
Xing, W., Chen, X., Stein, J., & Marcinkowski, M. (2016). Temporal predication of dropouts
in MOOCs: Reaching the low hanging fruit through stacking generalization. Computers in
Human Behavior, 58, 119–129. https://doi.org/10.1016/j.chb.2015.12.007
Xiong, Y., Li, H., Kornhaber, M. L., Suen, H. K., Pursel, B., & Goins, D. D. (2015). Examining
the relations among student motivation, engagement, and retention in a MOOC: A structural
equation modeling approach. Global Education Review, 2(3), 23–33. Retrieved from http://ger.
mercy.edu/index.php/ger/article/view/124
Yang, D., Wen, M., Howley, I., Kraut, R., & Rose, C. (2015, March). Exploring the effect of
confusion in discussion forums of massive open online courses. In Proceedings of the
Second (2015) ACM Conference on Learning@ Scale (pp. 121–130). ACM. https://doi.
org/10.1145/2724660.2724677
Chapter 16
Learning Analytics Leading to Remote
Invigilation for eTests: A Case Study
Michael Baird, Lesley Sefcik, Steve Steyn, and Connie Price
1 Introduction
Online education is growing in popularity; platforms such as EdX are testament to

this (Vioreanu, 2018; Woldeab, Lindsay, & Brothen, 2017; Yu & Hu, 2016). Students
often perceive that the workload of an online course will be less rigorous than in
a classroom setting, a perception that is not supported in the literature (Brocato,
Bonanno, & Ulbig, 2015). The perception in academia is that traditional face-to-
face courses can be put online with little to no modification, but this is also incorrect
(Harris & Parrish, 2006; Porter, Pitterle, & Hayney, 2014). Courses must be altered
to accommodate the affordances and limitations of the online environment, espe-
cially in regard to assessment. In the hurried transition to the online environment,
only certain assessment types or selected aspects of assessment have migrated,
while others have been largely ignored. Consequently, eAssessment is dominated
by the multiple-choice question (MCQ) format which can be ‘marked’ automati-
cally within the learning management system (LMS), while open-ended response
questions are generally unused. Some may consider electronic submission of text-
based assessments (e.g. via text-matching software) to be a form of eAssessment.
However, both of these approaches to assessment have inherent risk to academic
integrity or jeopardise the assurance of student learning. In the rush to capitalise on
flexibility for learners and thus to attract new students, higher education providers
have willingly accepted compromises. Increasingly, these compromises have begun
to erode the credentialing of student learning, which arguably is the key function of
the modern university in the information age where knowledge is readily accessible.
M. Baird (*) · L. Sefcik · C. Price

S. Steyn
Engineering Institute of Technology, Perth, WA, Australia

https://doi.org/10.1007/978-3-319-64792-0_16
296 M. Baird et al.
eAssessment, in its current dominant form of largely MCQs presented via a LMS
test/quiz tool, certainly has its place within the learning landscape. When used
appropriately, the tool, which is easily understood by the majority of students and
academics, offers good opportunity for students to milestone their learning prog-
ress. Thus it is most suited to formative or low stakes summative assessments.
However, when the function of the assessment is medium to high stakes, the inabil-
ity to control the test taking environment and the student’s behaviour during a non-
invigilated online test creates unacceptable risk to the integrity of the assessment
and assurance of diligent study and genuine study success.
During the early 2000s, the webcam has been commandeered as a window into
the test taker’s environment. This combined with the ability of computer systems to
record different input sources, such as what is displayed on screen, ambient sound
via built-in microphones, and keyboard or mouse interactions, a number of ‘surveil-
lance’ type approaches have emerged. These approaches, however, have significant
technical and practical limitations (James, 2016). A key limitation is cost. Some
approaches have attempted to overcome technical limitations of the built-in web
cameras by requiring students to use specialised camera devices with 360° pan-
oramic views; however, access to and cost of these devices is prohibitive in most
circumstances. The cost to institutions to license, integrate, and support remote
invigilation software is significant and must increasingly be evaluated in terms of
the cost/benefit ratio. This cost has largely been passed on to the student in North
America, whereas within Australia institutions are responsible for this cost. A hid-
den institutional cost that must also be considered is that of staff time to monitor the
recordings and data collected and to investigate and process suspected breaches of
academic integrity. Given the limitations imposed by the acceptance of built-in
cameras to reduce overall user costs, the ratio of false-positive identifications of
suspect behaviour is high and may result in significant staff workload.
One of the biggest barriers to this technology has been staff and student resis-
tance. Student resistance may be something of a white elephant, as the use of this
technology can be written into the course requirements such that non-invigilated
assessments will not be accepted. However, this requires institutions to ensure that
students have the facilities and resources available to use the technology. Staff resis-
tance is another matter, as in a university environment where the pressure on aca-
demics is constantly increasing (Papadopoulos, 2017), adding the review of videos
to ensure academic integrity for online students is difficult (Garrison, Anderson, &
Archer, 1999; Gregory & Lodge, 2015).
Remote invigilation is becoming an increasingly popular educational approach.
One company, Software Secure Inc., has licence agreements with over 200 institu-
tions (Davis, Rand, & Seay, 2016). The authors of this study however found third-
party services to be cost prohibitive and therefore decided to develop a solution
in-house. Compared to other identity-verifying alternatives, e.g. optic retinal scan,
fingerprint scans and keystroke pattern analysis (Jortberg, 2010), webcams coupled
with a browser-based plug-in are less costly and complex. Moreover, unlike other
techniques, webcams can place a virtual teacher in the presence of the student, cre-
ating a greater sense of being part of a learning community.
16 Learning Analytics Leading to Remote Invigilation for eTests… 297
1.1 Case Structure
Research has shown that ensuring online students are adequately invigilated when
completing eTest in their own time is problematic (Karim, Kaminsky, & Behrend,
2014). It is also difficult to ensure academic integrity for student cohorts located in
different regions and time zones without some form of remote invigilation. Recent
developments in remote invigilation technology may help to better ensure assess-
ment integrity for online students, which has not previously been available (Davis
et al., 2016; Karim et al., 2014; Phillips & Lowe, 2003; Schaffhauser, 2017). Hence,
this research provides an account of a proof-of-concept study examining the use of
a browser-based plug-in to remotely invigilate students’ online tests.
This case begins with an overview of the course, Business Capstone, and first use
of eTest learning analytics amongst an internal study mode cohort (Table 16.2). This
prompted the examination of internal compared to external results (Table 16.3).
Learning analytics data gathered successively was used to build a body of evidence
suggesting that online students were potentially engaging in academic misconduct.
This data included average weekly eTest scores, overall average eTest scores, a
benchmark assessment score and study mode comparisons. The case shows how
learning analytics data evidence raised suspicions to the point where action had to
be taken and a remote eTest invigilation trial commenced, wherein year on year
eTest scores (with/without remote invigilation) are compared to the internal on-
campus cohort.
Detail is provided on the problem of academic misconduct and the use of learn-
ing analytics to help resolve the issue. The case study highlights some of the chal-
lenges faced, explains the technological development process and shares details of
the trial-and-error process. Furthermore, it examines the staff perspectives in terms
of time, effort and organisational requirements and provides information about the
student perspectives in terms of their usage and comfort with the technology, with
the aim of highlighting the staff and student perspectives for using this sort of tech-
nology for online assessments, now and into the future.
2 The Vehicle
Business Capstone is Curtin University’s Bachelor of Commerce capstone unit. The

unit is owned and run by the School of Marketing and is a common core unit, mean-
ing all commerce students must complete the unit, regardless of major (Accounting,
Business Law, Economics and Finance, Information Systems, Management,
Marketing). The unit is in all commerce major study plans in the final study period
(semester/trimester); however, it can be taken in the second last study period. The
unit has two prerequisites: (1) completed 400 credit points (equivalent to 2 full years
of study) and (2) completed Communication in Business, a first year common core
unit. The unit has three study modes: internal on-campus (Perth, Australia; Singapore;
Miri, Malaysia; Mauritius), fully online, and Open Universities Australia (OUA).
298 M. Baird et al.
Table 16.1 Business Seminar Content

Capstone course overview
1 Introduction and overview
2 Personal and team dynamics
3 Governance and strategy
4 Accounting, finance and economics
5 Marketing, production, Research and Development
6 Human resource management
7 Labour negotiations and ethics
8 Management
9 Business information systems, supply chain and
logistics
10 International business
11 Business sustainability and resilience
12 Company presentations and conclusion
The unit runs in the flipped teaching mode; students have a 2-h weekly seminar
(instead of 3 h for non-flipped units) which they must attend regardless of study
mode. Students are expected to come to class fully prepared to participate in the
unit, having viewed/read the iLecture, pre-readings and associated videos for that
week.
The aim of Business Capstone is to give all commerce students an understanding
of the different disciplines of business and of how they interact. With this, the lec-
ture content is broken into different business disciplines each seminar, with the
exceptions of seminars 1 and 12, across the 12 seminar teaching period. Table 16.1
shows the breakdown of Business Capstone’s content coverage.
2.1 eTests
The eTest assessments are a series of ten weekly individual multiple-choice elec-
tronic tests. Business Capstone uses a ‘choose your own eTest’ system where, for
each week’s eTest, students have the option to complete one out of a choice of two
eTests, assessing slightly different material from the course. Each eTest consists of
ten randomly selected questions (from a bank of 50+ questions per eTest) and is
worth 2% of the unit (ten eTests × 2% = 20% total). The eTests are an effective tool
to ensure students learn the required pre-seminar material as required by the flipped
teaching mode, such as reading of journal articles and watching of specific videos.
Students have 10 min allocated for each eTest (ten questions = 10 min). Students
are permitted one attempt for each eTest, and each eTest must be completed in one
session. Students are permitted to refer to paper copies of specific material for each
eTest and handwritten notes.
2.2 History of the eTest Assessment
The eTests were developed to ensure students attended class fully prepared. Prior to
the implementation of the eTests, when the flipped teaching mode was first initiated,
students would not do any preparation work (pre-reading/watching) before coming
to class, and since the in-class time was reduced from 3 contact h to 2, students
would understandably struggle.
This prompted the initial application of a series of ten formative weekly eQuiz-
zes, which were not assessed, and a final summative eTest at the end of the study
period. The summative eTest (worth 20% of the unit) would assess the entire study
periods material, with questions being drawn from the formative eQuiz data banks.
Students were given 30 min to complete 20 questions in the final, summative eTest.
Unfortunately, it was found that since the eQuizzes were formative, the majority
of students would not complete them and feedback showed that students did not like
having a 30-min assessment with material covering the entire study period.
Therefore, a proposal was submitted to change this to be a series of ten weekly
eTests, worth 2% each. The university has a policy of every unit containing no more
than four assessment items. As a single eTest worth 20% (one assessment) was
being transformed into ten eTests worth 2% each (ten assessments), special permis-
sion had to be granted for this to occur.
The first study period of weekly eTests was closed book, 15 min eTests with no
option to choose. They were also done outside of class. Over subsequent study peri-
ods, due to student feedback and review to improve the assessment, the time was
reduced to 10 min, eTests were done in class time, they became partial open book,
and the ‘choose your own eTest’ system was implemented.
Another assessment within the course is the written case study report. The case
changes every study period; however, the assessment criteria remain the same, so
the results are stable. Unlike the eTests, students know the report is submitted to
text-matching software. From the students’ perspective, to engage in academic mis-
conduct while writing the report poses a higher risk comparted to the eTests.
Therefore, the report average is included as a benchmark of how the students genu-
inely perform. See Table 16.2 which highlights this across a 20-month period in a
single campus location.
Of the assessment iterations, the change from the eTests being done outside of
class to in class time in 2015 was the most pressing, because it was found that stu-
dents were getting abnormally high results on eTests yet performing noticeably
weaker on all other assessments in the unit, particularly the written report
(Table 16.2).
Prior to trimester 3A, 2015 for all study modes, no cases of alleged academic
misconduct were reported for the eTest assessment within the Business Capstone
unit, as the eTests were done without invigilation, and this was recognised as a seri-
ous risk to the integrity of the assessment. From trimester 3A, 2015, to trimester 3A,
2016, all internal study modes completed the eTest in class, and a total of 18 cases
of alleged academic misconduct (cheating during a test or exam) were reported. In
all of these cases, students brought into class cheat notes; printouts of questions and
300 M. Baird et al.
Table 16.2 Internal study mode eTest with benchmark comparison

Report average eTest average
Study period eTest done Invigilation (out of 20)a (out of 20) Difference
Trimester Before No 10.65 13.32 13.4% better on
1A, 2015 class eTests than report
Trimester Before No 11.61 14.25 13.2% better on
2A, 2015 class eTests than report
Trimester In class Yes—In class 11.46 10.16 6.5% worse on
3A, 2015 eTests than report
1A, 2016b eTests than report
2A, 2016 eTests than report
a
The report average does not include late submissions
b
A proportion of eTest material (all Video Business Interviews) were revised prior to this study
period, with old videos and questions being deleted in place of a new series of videos and associ-
ated question banks
answers of former or current questions of eTests that students who did the eTest
without invigilation (prior to trimester 3A, 2015) had taken screenshots and posted
the material up on websites, such as www.coursehero.com.
The learning analytics collected to date had proved very useful. Monitoring eTest
results by location with/without in-class invigilation and benchmarking against
another assessment (where academic misconduct is arguably less likely) led to a
suspected academic integrity issue amongst internal students in trimester 1A and
trimester 2A, 2015. Since recognising that this was almost certainly occurring at
regular intervals (every 4–6 months) the Unit Coordinator of the Business Capstone
unit performed online searches for these question and answer pages of the eTest
material and requests the website(s) to remove them (if they are US based and abide
by the Digital Millennium Copyright Act [DMCA]). To date, this has resulted in six
takedown requests with between 30 and 78 (per request) different web addresses of
content (eTest questions and answers, Unit Outlines, course notes) owned by the
Business Capstone unit. In each of these requests, the website has complied and
removed the offending material within 24 h.
When given the opportunity (trimester 1A and trimester 2A, 2015), internal stu-
dents cheated. That much is clear. The solution was to hold all eTests during class
time with an invigilator. But what of online students? With no invigilator, were they
behaving like their internal counterparts? The authors’ attention now turned to the
online study mode.
For the online study modes (both fully online [hereto referred to as online] and
OUA), Business Capstone requires attendance at a 2-h synchronous online seminar
every week of the study period. Thus, completing the unit in the online mode had
parity with completing it internally, with the exception of attending a virtual class-
room on a computer as opposed to a physical classroom. However, one major differ-
Table 16.3 Comparisons for internal verses onlinea

Study Study Report average eTest average
period modeb Invigilation (out of 20)c (out of 20) Difference
Semester 1, Internal Yes—In 11.91 11.48 Online performed 3.5%
2016 class better on eTests than
Online No 11.64 12.18 internal
Semester 2, Internal Yes—In 12.48 11.65 Online performed 0.8%
2016 class worse on eTests than
Online No 12.11 11.49 internal
a
Tables 16.2 and 16.3 pertain to two different student cohorts in different locations within Australia
b
All comparisons between study modes have been analysed via t-test and are significantly different
at 0.000
c
ence was that online students completed their eTests before the online seminar and
were not invigilated since the instructor did not have the technology available to
invigilate the assessment virtually for all online students synchronously. Therefore,
prior to the remote invigilation trial taking place (semester 1, 2017), online study
modes were at greater risk of assessment integrity breaches compared to face-to-
face modes of study due to a lack of invigilation. This discrepancy and risk to the
integrity of the online assessment caused concern, since a student that is actively
looking to cheat on the eTest assessment can do so without fear of retribution in the
online study mode.
In semester 1, 2016, online students performed 3.5% better on eTests compared
to their internal study mode counterparts (as seen in Table 16.3). This result was
counter to the report average, and indeed the norm, as it has been found that online
study mode students generally perform worse on all assessments for a number or
different reasons such as parenting duties and full-time work schedules (many stu-
dents choose online study due to their full-time job or parental responsibilities) and
the lack of physical interaction. The analysis of this data led to the theory that online
study mode students scored better on their eTests through cheating due to a lack of
assessment invigilation (Carstairs & Myors, 2009). The trend was investigated
further in semester 2, 2016 but was not shown to continue. Still, given the trimester
results (Table 16.2), the possibility of academic misconduct amongst online stu-
dents was still very strong.
In early 2017, an opportunity was presented to trial a remote invigilation system,
which had the possibility of creating equity between online and internal study
modes. The use of the remote invigilation system for the eTest assessment allows
invigilation for online students, regardless of their location. The online students of
Business Capstone already required a webcam and microphone for the unit to effec-
tively complete the other assessments, so there was little that had to change initially
to include invigilation in the unit. The fact that the unit already had ten separate,
weekly eTests of just 10 min duration also made it a perfect choice for testing and
improving the invigilation system.
302 M. Baird et al.
3 The Technology
A chance meeting at a conference by some of the co-authors led to Curtin University

and the Engineering Institute of Technology (EIT) collaborating on a remote invigi-
lation system proof-of-concept study. EIT had been developing and using the soft-
ware within their courses for a few years and were eager to share and collaborate.
Curtin University had never used the software before and, with assistance from EIT,
are implementing it in a large-scale environment, bringing with it rigorous market
testing and potential commercialisation possibilities.
The Business Capstone unit was chosen to test the system implementation. The
team was assembled for the first time in February 2017 where the requirements
were discussed. The first challenge to overcome was integration into the LMS: EIT
uses the Moodle platform, whereas Curtin uses Blackboard. EIT proposed that the
software remain browser-based, as the Google Chrome internet browser has fea-
tures to facilitate this, making the software lightweight, and easy to activate on
restricted corporate networks (Engineering Institute of Technology, 2017). This is
unlike most competitor offerings whereby a software package must be downloaded
and installed by the student onto their personal machine. Browser-based extensions
require a small plug-in file to be downloaded into the browser itself.
3.1 Functionality Overview
The remote invigilation software was designed to monitor the audio, video and
screen of students’ work environment during online tests and automatically flag
behaviour that may show academic dishonesty. The application needed to be rela-
tively nonintrusive, easy to use for students and easy for academic staff to assess and
manage afterwards.
EIT upgraded their existing invigilation module (Engineering Institute of
Technology, 2016) to an Angular 2 (at the time of writing) single-page application
that is incorporated into Google Chrome as a stand-alone plug-in/extension. This
stand-alone plug-in is able to record for several hours and, upon completion of the
test, upload the recorded data to the server.
The plug-in’s user interface includes four steps asking students for their names,
IDs, microphone and webcam access. Furthermore, a screenshot of student IDs are
taken, and the student’s computer screen is captured. These fields are mandatory to
progress to the test.
The stand-alone remote invigilation module integrates into LMS platforms by
means of Uniform Resource Locator (URL) detection. When the predetermined
assessment is in the browser URL, the extension pops up and starts the invigilation
process. The final submission of the test prompts the extension to stop recording and
upload the recorded files. This same methodology is used for Blackboard LMS and
can be used for other LMS integrations in the future.
Facial presence recognition was implemented to search for instances when a face
is not identifiable on the student facing camera. Initially, facial recognition was
analysed after the recorded file was uploaded to the server, at which time the system
flagged suspicious sections of the recording. Recently, a client side detection library
was introduced to indicate to the student their facial recognition status in real time,
which is analysed on their own machine rather than on the server.
Finally, the recorded sessions are accessed via a cloud-based password-protected
administration and lecturer platform, with review features such as recording and
flagged frame summary playback, filtering and session search.
3.2 Implementation
The first major task was to ensure that the remote invigilation system extension was
supported in the Blackboard LMS used by Curtin. By using a Chrome extension, as
opposed to an embedded Blackboard or Moodle plug-in, the Chrome extension
remains LMS agnostic and was found to work successfully in both institutions’
LMS.
Fortunately, there was little that had to change within the course itself, as the
eTests were already set up for the students, and they required little modification. The
main element that required implementation was the instructions for students, as it
had to be made clear that this was a requirement of students taking the course and
that the Google Chrome browser must be used, and the Curtin University plug-in
downloaded into the browser. With this, information was placed in the Unit Outline
(the course’s legally binding contract) and in the Blackboard LMS.
Perhaps the biggest alteration to the remote invigilation process between the dif-
ferent cohorts of students came in the form of passwords. In the study periods in the
first half of 2017 there were no passwords for the eTests for the online and OUA
students. This was standard, as mentioned previously, since the students could
complete the eTest in their own time before attending their virtual classroom; hence,
there was no need to restrict any usage of the eTests.
Consequently, there were no measures in place to force students to use the remote
invigilation plug-in in the Google Chrome browser. This offered the opportunity for
students to exploit the system. If a student were to use Internet Explorer or Firefox,
they could still complete the eTest without remote invigilation. If a student used
Chrome and did not install the plug-in, they also could still complete the eTest. This
resulted in no recordings being present for some students. While follow-up could be
and often was made with students, ultimately this increased the academics workload
as the responsibility was currently upon the student themselves to ensure they had
the invigilation operational.
This issue was resolved by implementing Blackboard passwords for each eTest
in the study periods in the second half of 2017. The remote invigilation software
was updated so it could automatically populate the password field in the eTest, after
the invigilation started, thereby eliminating the need for staff or students to exchange
304 M. Baird et al.
passwords and ensuring the remote invigilation software is active and in use. Thus,
without activating and running the invigilation software, a student would be unable
to complete the eTest each week. Each password was a random generation of letters
and numbers of varying length, so although students could not see the characters,
they could witness the different numbers of characters each week.
This had increased the number of eTest remote invigilation recordings captured
but also uncovered another issue that needed to be resolved—students shutting
down the software before the recording could finalise and upload to the server.
Depending on internet speed (bandwidth), the upload process would take up to 30 s
after completion of the eTest. If students closed the invigilation window before the
file had completely uploaded, the file would be deleted and would not be recover-
able. This meant that while students were being remotely invigilated, staff could not
check the recording as the system had nothing uploaded. The solution to this issue
was twofold: first to further clarify the instructions, headings and prompts within
the software were implemented to make students aware that they must wait for the
recording upload to finish (at which time the invigilation window would automati-
cally close) and second to save recording files to a student’s computer and allow the
software to recommence the upload upon opening the software for the next eTest,
thereby saving the recording regardless of when the software is closed. If the
recording of the final eTest was not available in the server, the student would be
advised to reopen the software and press a button that uploads any stored recordings
to the server.
3.3 Outcomes
The technology proved very effective for the course in the proof-of-concept that
took place. Table 16.4 shows eTest comparisons for both online and OUA students
across a 1-year period without remote invigilation and a 1-year period with remote
invigilation. While it is difficult to say definitively the result remote invigilation had
on the eTest average, it can be concluded that the numbers are very consistent
throughout the periods analysed (with the possible exception of the Semester 1,
2016 standard deviation, but this had the lowest student number cohort).
From this, it could be surmised that the use of remote invigilation, at least for a
unit with medium to large student numbers per study period, has no beneficial nor
detrimental effect on student grades yet helps ensure academic integrity. This is in
agreement with the findings of Karim et al. (2014) and Amanullah, Zaman, Patel,
and Mohanna (2013). In Table 16.4, it is of interest to note that the highest eTest
average was achieved when they were completed in class time, the ‘choose your
own eTest’ option had no noticeable effect on the eTest average (or if it did it was
moderated by the remote invigilation), and the ‘choose your own eTest’ option
resulted in students completing an average of 12.4 eTests per semester, compared to
an average of 9.7 eTests per semester without it. The difference between the eTest
average and the Report average was never more than one grade point, or 5%.
Table 16.4 eTest comparisons
Type of invigilation Number of eTests eTest average eTest standard Report average
Study period eTest done used studentsa completed (out of 20) deviation (out of 20)b
Semester 1, 2016 In class In class 75 740 12.26 0.112 11.55
Semester 2, 2016 Before class None 125 1184 11.70 0.085 11.82
Semester 1, 2017c Before class Remote 94 1186 11.67 0.088 12.67
Semester 2, 2017 Before class Remote 98 1192 12.19 0.084 12.94
a
The number of students includes online (semester) and OUA (study period)
b
c
This study period saw the introduction of the Choose your own eTest option, allowing students to choose the material they are assessed upon, as well as allow
them the option to complete both eTests in which case the higher score will be recorded
16 Learning Analytics Leading to Remote Invigilation for eTests…
305
306 M. Baird et al.
The use of the remote invigilation software did result in the capture of the first
case of academic misconduct in the units’ online teaching formats. This occurred in
semester 2, 2017 where a student was attempting both of the seminars eTests in a
single sitting. The student would (1) complete eTest A, (2) screenshot their eTest
page, (3) go and check the answers and mark a tick or a cross upon the screenshot
image and (4) complete eTest B with these images as unauthorised help. In this
case, the screenshots did not help the student perform any better on their second
eTest, as questions were randomly selected from a bank of 50+ questions per eTest,
meaning the student did not see any of the same questions twice. Nonetheless, as
this was discovered, it was put through the university’s process for academic mis-
conduct. Needless to say, this was an excellent result in less than 1 year of usage and
shows the importance of the technology for ensuring academic integrity going
forward.
The analysis of learning analytics has been crucial in getting the project to its
current state and will be necessary going forward. Learning analytics provide
important data points for making the technology effective, efficient and user-
friendly, as well as providing important justification for the project (Amigud,
Arnedo-Moreno, Daradoumis, & Guerrero-Roldan, 2017). Without the initial learn-
ing analytics data, it would have been more difficult to justify the project, as it was
through the analysis of this data that it was found there was a problem. The analysis
of the proof-of-concepts’ first year of data also helped highlight issues, such as the
lower than expected recording uploads, which has resulted in solutions developed to
fix these issues for the future.
The learning analytics of the technology will need to be closely monitored as the
project continues and expands, especially as the proof-of-concept is turned into a
trial phase whereby more units are approached to use the remote invigilation. With
more units comes additional assessment types, thereby allowing for a greater scope
of data which could be used for analysis. The learning analytics within remote
invigilation has huge potential to uncover more than just academic integrity issues.
4 Student Perspective
With the introduction of anything new, there will always be some form of student
resistance until the practice becomes commonplace. In regard to the eTests, it was
also important to ensure students felt no more or less stressed while completing the
assessment. So it did surprise the authors that throughout 2017, there were no issues
or complaints coming from students about the use of remote invigilation at the start
of the study periods; students simply accepted the fact (n = 192). However, feedback
was received at the end of the course. It came through two channels: (1) a research-
based questionnaire created specifically for this purpose and (2) the university’s
general unit feedback questionnaire. Both channels were anonymous and optional
for students to complete but had chances to win prizes as motivation.
The research-based questionnaire included the following questions:
• Before you participated in the project, please describe how you felt about the
idea of being remotely invigilated.
• After you participated in the project, please describe your level of comfort in
relation to being remotely invigilated.
• How has your attitude towards remote invigilation changed over the course of the
unit?
• How effective do you think remote invigilation is for discouraging cheating?
While the general unit feedback questionnaire included:
• The learning experiences in this unit help me to achieve the learning outcomes.
• The assessment tasks in this unit evaluate my achievement of the learning
outcomes.
• I make best use of the learning experiences in this unit.
• Overall, I am satisfied with this unit.
Some issues arose at the end of the course when students were given the oppor-
tunity to provide feedback. The instructions for students were one of the biggest
causes for concern; feedback after the first cohort of students that used the remote
invigilation system (semester 1, 2017) indicated that the instructions document was
not clear enough. While the instructions were clear on getting the remote invigila-
tion operational, they were not clear on issues such as who would be viewing the
recordings, where would they be kept and for how long will they be kept. This is in
line with the findings of Lilley, Meere, and Barker (2016). In the age of information
technology we live in, this detail is expected by students to ensure that their confi-
dential information is not being compromised. This was revised and improved by
adding additional information into the instructions document for the second study
periods in 2017. A set of frequently asked questions (FAQs) were also created read-
dressing some of the information in the instructions document (in case it was
missed) and adding additional questions and/or concerns that were raised.
Surprisingly, further feedback from the second cohort of students showed they still
wanted improvement in this area, although more focused on information security.
This was developed and clarified for the next study period in 2018 after seeking
clarity around the university’s IT and data retention policies.
Students generally understood the purpose and need for remote invigilation, as
was addressed by some comments from students. This is a good sign, as it shows
that students recognise the importance of this technology to ensure that everyone is
on the same playing field. However, the fact that they had come through at least
2 years of their degree and then in a final year unit to be told they must be remote
invigilated was concerning for some students, but this is something that, for the
most part, is unavoidable. As the proof-of-concept expands to a trial, and then onto
a university-wide rollout, it is anticipated that these concerns will dissipate.
Students were generally comfortable using the remote invigilation. Students
were being recorded in their workplace, kitchen, lounge room, lecture theatres, in
bed and even in a car. It became clear that during a 10 min eTest, the students focus
needed to be maintained on the test, not anything else around the student. As men-
308 M. Baird et al.
tioned previously, the concerns were primarily around the security and privacy of
the data, as opposed to concerns getting the plug-in working or why it was being
used.
5 Staff Perspective
The staff perspective may well be the most important consideration in using this
technology, as without staff believing in and supporting the process it will never
work as it is designed. The pressure and workload on academic staff is constantly
increasing (Papadopoulos, 2017), so the addition of another step in the assessment
grading process may be met with some resistance. There will always be some staff
members that are very eager to use this technology to ensure the academic integrity
of their assessments within their unit for online students. However there will also be
staff at the other end of the spectrum, who are hesitant and resistant to change and
ignore any potential breaches of academic misconduct rather than using a potential
solution that may increase their workload. In this way, institutional readiness plays
an important role (West, Luzeckyj, Searle, Toohey, & Price, 2018).
The good news is that the technology is easy to use, and once adequate processes
are in place, the workload becomes quite minimal. The initial setup of integrating
the remote invigilation into a course will take an hour or two; this includes putting
links to the remote invigilation plug-in into the LMS, linking/creating instruction
sheets on downloading and/or using the plug-in, and creating a demonstration eTest
(if not already available) so students can verify the plug-in works on their computer
as expected before any real eTest takes place. The setup of integrating the eTest back
into the remote invigilation plug-in is very quick, as all that is required is an eTest
name, web address of the eTest from the LMS, time of the eTest and LMS password
of the eTest.
The review time required for staff will vary according to the time of each eTest
and the number of eTests in a given study period. To focus on the example within
the case study given, 10 min eTests were used; each eTest required the staff approxi-
mately 1 min per student to review their submission. If it was found that students
were not following the specified rules, for example, had headphones on or the cam-
era was not adequately showing their face, additional time was required for follow
up with the student.
The time required for student queries or concerns was lower than anticipated in
our trial. As mentioned earlier, students accepted the fact that they had to use the
remote invigilation as it had been written into the course documentation. In semes-
ter 2, 2017, when the plug-in was refined and did not require updates during the
study period, there were only a handful of emails relating to the remote invigilation,
and these primarily occurred due to students not attending the online class or read-
ing the documents. The emails that were received could be answered with a stan-
dardised email, asking questions such as ‘are you using the Google Chrome
browser?’, ‘have you downloaded the remote invigilation plugin?’, and ‘have you
completed the demonstration eTest?’ The time required for responses to student
queries would be less than 1 h per study period.
Overall the effort required for a staff member is fairly marginal, as once the
documentation has been created, the rest of the setup is very minimal. The review of
the recording for each eTest actually allows a wonderful insight into students’
behaviour during eTests and can allow staff to improve their teaching and/or assess-
ments to help ensure study success. For example, the remote invigilation actually
allows a window into the students’ life when conducting an eTest; students quite
quickly get used to being recorded, and assessing students’ reactions to certain
questions within an eTest was insightful. A quick meta-analysis of this data could
prove to be very interesting and would draw back to the advantages of using learn-
ing analytics to support study success. These types of learning analytics provided by
remote invigilation allow a shift from broad, numbers based, quantitative type data
analysis to a more real-life, personal, qualitative type analysis to be done.
6 Discussion
Through the data gathered and analysed over numerous study periods and campus
locations, it highlights that learning analytics has played an important role in the
assessment and review process. Remote invigilation as a field of research is still in
its infancy, so there is much more work to be done to gain a complete understanding
of the benefits and drawbacks of the technology. However, it is something that will
be gaining momentum and popularity as institutions look for ways to better assure
academic integrity during this transition to the digital educational economy.
From a student perspective, online education allows them to study at their conve-
nience, fitting around their work and other obligations which may not be otherwise
possible (Woldeab et al., 2017). Jefferies et al. (2017) state that online invigilation
‘can be no more stressful for assessments when taken in their chosen personal envi-
ronment’ (p. 221). That is not to say any form of test or exam is not stressful. Rather
the point is that remote invigilation should not make them any more stressed than in
a physical environment (classroom or exam hall). It should be noted that the major-
ity of concerns students have had with the software in this case have been addressed
to date, and continual monitoring of learning analytics will ensure this going for-
ward. The aim here is to improve genuine study success specifically for students
who choose to study online.
From a staff perspective, there are two schools of thought regarding the uptake of
this technology: (1) implement it now while there is time to modify assessments and
analyse the learning analytics or (2) wait until it becomes an institution requirement
(which in some cases may never occur). Jefferies et al. (2017) note the practical
issues with the software technology are still substantial and ‘must receive ample
attention because the procedures and technologies currently result in too many flaws
and failures and backup procedures are not clarified’ (p. 227). The authors believe
their remote invigilation software to be very effective in the context of this case but
310 M. Baird et al.
recognise that further testing and analysis of data is required before their institu-
tions, let alone any others, have a mass rollout of the technology.
It has to be acknowledged that other forms of remote invigilation exist in the
market. One such form different from the method in this case study is the use of live
remote invigilators. Other studies have examined these in detail (e.g. Jefferies et al.,
2017; Woldeab et al., 2017) and found live invigilators (in an online context) to be
distracting to students. The authors’ remote proctoring software seems much more
capable and user-friendly for the purpose it was used for in this case study than
some other providers in the market, such as that researched by Woldeab et al. (2017).
6.1 Limitations and Future Research
It is worth noting some limitations within this case study. This case has only dis-
cussed one assessment type (MCQ eTests) in one course type (business unit); hence,
the authors do not consider the results generalisable. It is hoped, however, that this
example of using a variety of learning analytics to uncover a problem will stimulate
others to do the same, in any context. The result in this example, being remote
invigilation, is no silver bullet; ongoing analysis of the learning analytics must be
done to provide a stronger case for genuine study success.
A further limitation is that the unit of analysis is the student, specifically how
they behave in a test environment. The remote invigilation solution to the problem
does not provide the ability to know what the student is actually learning or what
they are doing while they learn. The style of the study is behaviourism, (in this case
the very end of the learning process), which clearly has its place, but is not without
its critics (e.g. Siemens, 2005).
The possibilities for future research in this field are enormous, as the authors are
only just scratching the surface of what remote invigilation can do. Further develop-
ment of countermeasures, intelligent flagging and alternative test types (non-MCQ)
will be investigated as the proof-of-concept progresses to pilot and commercial
implementation.
In regard to learning analytics, student behaviour during an online test can be
examined (calm-anxious, relaxed-confused), as well as time taken for online tests
(and how much this varies depending upon test type) or even time to complete spe-
cific questions. The location of where students choose to complete the online tests
would make an interesting analysis, as well as the time of day/night.
A trial could also compare results with an expensive third-party invigilation ser-
vice on the same eTest assessment to see what happens to the eTest results. That
would permit three data points to compare; no invigilation, invigilation via the Chrome
plug-in and invigilation via more complex methods (e.g. biometrics, eye/fingerprint
scans, key stoke analysis). Analysis of this dataset could analyse if students are more
or less content with the third-party service. Further answers could be gained to the
questions of whether third-party services create even greater internal/online parity,
and whether third-party service catch more cases of academic misconduct.
7 Conclusion
The use of learning analytics helped justify the creation of a browser-based plug-in
to remotely invigilate online students while completing an eTest. The analysis of the
learning analytics data acted as a warning system which leads to some real concerns
that would not be unique to this University. Hence the case study explaining the year
of rapid development between EIT and Curtin provides a simple, cost-effective and
reliable tool. This tool helps ensure academic integrity by remotely invigilating stu-
dents while they take an eTest thereby providing equity between the internal and
online study modes of the unit and promoting study success.
Work on the project will continue into the future, with a list of further improve-
ments for students and for staff constantly being developed. The proof-of-concept
will be expanded into a trial in the second half of 2018, with the potential for com-
mercialisation after this. One way or another, universities are going to have to
address the issue of increasing online student numbers/assessments and the inherent
difficulties they bring with regard to academic integrity and the greater potential for
academic misconduct.
References
Amanullah, M., Zaman, G. S., Patel, A. A., & Mohanna, K. (2013). A comparative study of open
book-open web (OBOW) exams and invigilated closed book-pen and paper (ICBPP) exams.
Merit Research Journal of Education and Review, 1(4), 97–106.
Amigud, A., Arnedo-Moreno, J., Daradoumis, T., & Guerrero-Roldan, A.-E. (2017). Using learn-
ing analytics for preserving academic integrity. International Review of Research in Open and
Distributed Learning, 18(5), 192–210.
Brocato, B. R., Bonanno, A., & Ulbig, S. (2015). Student perceptions and instructional evalua-
tions: A multivariate analysis of online and face-to-face classroom settings. Education and
Information Technologies, 20(1), 37–55.
Carstairs, J., & Myors, B. (2009). Internet testing: A natural experiment reveals test score inflation
on a high-stakes, unproctored cognitive test. Computers in Human Behavior, 25(3), 738–742.
Davis, A. B., Rand, R., & Seay, R. (2016). Remote proctoring: The effect of proctoring on grades.
In Advances in accounting education (Vol. 18). Bingley, England: Emerald Group Publishing.
Engineering Institute of Technology. (2016). EIT internal R&D activity report - June 2015 to July
2016. Perth, Australia.
Engineering Institute of Technology. (2017). EIT internal R&D activity report - June 2016 to July
2017. Perth, Australia.
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment:
Computer conferencing in higher education. The Internet and Higher Education, 2(2-3),
87–105.
Gregory, M. S.-J., & Lodge, J. M. (2015). Academic workload: The silent barrier to the implemen-
tation of technology-enhanced learning strategies in higher education. Distance Education,
36(2), 210–230.
Harris, D. M., & Parrish, D. E. (2006). The art of online teaching: Online instruction versus in-
class instruction. Journal of Technology in Human Services, 24(2-3), 105–117.
James, R. (2016). Tertiary student attitudes to invigilated, online summative examinations.
International Journal of Educational Technology in Higher Education, 13, 19.
312 M. Baird et al.
Jefferies, A., Barton, K., Meere, J., Peramungama, S., Pyper, A., & Yip, A. (2017). Trialling online
proctoring for e-assessments: Early outcomes from the Erasmus + OP4RE project. Paper pre-
sented at the European Conference on e-Learning, Porto, Portugal.
Jortberg, M. (2010). Experiences verifying the identity of distance learning students. In Third
annual report on identity in distance learning. Little Rock, AK: Acxiom.
Karim, M. N., Kaminsky, S. E., & Behrend, T. S. (2014). Cheating, reactions, and performance
in remotely proctored testing: An exploratory experimental study. Journal of Business and
Psychology, 29(4), 555–572.
Lilley, M., Meere, J., & Barker, T. (2016). Remote live invigilation: A pilot study. Journal of
Interactive Media in Education, 1(6), 1–5.
Papadopoulos, A. (2017). The mismeasure of academic labour. Higher Education Research and
Development, 36(3), 511–525.
Phillips, R., & Lowe, K. (2003). Issues associated with the equivalence of traditional and online
assessment. Paper presented at the 20th Annual Conference of the Australasian Society for
Computers in Learning in Tertiary Education (ASCILITE), Adelaide, Australia.
Porter, A. L., Pitterle, M. E., & Hayney, M. S. (2014). Comparison of online versus classroom
delivery of an immunization elective course. American Journal of Pharmaceutical Education,
78(5), 96–104.
Schaffhauser, D. (2017). Nobody’s watching: Proctoring in online learning. Retrieved from
https://campustechnology.com/articles/2017/07/26/nobodys-watching-proctoring-in-online-
learning.aspx
Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of
Instructional Technology and Distance Learning, 2(1), 3–10.
Vioreanu, D. (2018). Distance learning course options get more popular every year. Retrieved
from https://www.distancelearningportal.com/articles/393/distance-learning-course-options-
get-more-popular-every-year.html
West, D., Luzeckyj, A., Searle, B., Toohey, D., & Price, R. (2018). The use of learning analyt-
ics to support improvements in teaching practice. Melbourne, Australia: Innovative Research
Universities.
Woldeab, D., Lindsay, T., & Brothen, T. (2017). Under the watchful eye of online proctoring. In
I. D. Alexander & R. K. Poch (Eds.), Innovative learning and teaching: Experiments across the
disciplines. Minneapolis, MN: University of Minnesota Libraries Publishing.
Yu, J., & Hu, Z. (2016). Is online learning the future of education? Retrieved from https://www.
weforum.org/agenda/2016/09/is-online-learning-the-future-of-education/
Chapter 17
Epilogue: Future Directions on Learning
Analytics to Enhance Study Success
Dana-Kristin Mah, Jane Yin-Kim Yau, and Dirk Ifenthaler
1 Introduction
Enhancing student success in higher education has been a crucial issue for many
years. Learning analytics as an emerging educational field show promise to improve
learning and teaching in higher education and thus increase student retention.
Students’ data is captured when they interact with digital learning environments,
such as learning management systems, mobile devices, and social media.
Accordingly, the expectations towards learning analytics to predict student success,
identify students at risk, and provide personalised feedback and academic support
services are very high (Ifenthaler, 2015; Long & Siemens, 2011).
This edited volume presents a broad collection of work and findings on how
educational data contribute towards successful learning and teaching scenarios.
Overall, this edited volume features chapters focusing on theoretical foundations,
technological frameworks and innovations, issues and challenges for implementing
learning analytics systems, as well as case studies, empirical research findings, and
examples of higher education institutions, which adopted learning analytics.
This epilogue provides an analysis of the previous chapters with major themes
that have emerged. Moreover, this chapter presents ideas for future directions on
learning analytics and shall serve as a platform for further discussion and dialogues
about enhancing student retention in higher education.
D.-K. Mah · J. Y.-K. Yau

D. Ifenthaler (*)

https://doi.org/10.1007/978-3-319-64792-0_17
314 D.-K. Mah et al.
Firstly, we provide an overview of learning analytics in higher education. Then,

we discuss four key themes: the first theme is acceptance and competence for the
implementation of learning analytics, the second theme is personalised learning and
early interventions, the third theme is data privacy and ethics, and the fourth and last
theme is technical considerations. We conclude with a discussion of implications
and future directions on using learning analytics for supporting study success.
2 Learning Analytics to Enhance Study Success
One objective of learning analytics is to construct predictive and prescriptive mod-

els in order to identify students at risk of failing a course and who are thus more
likely to leave the institution prior to degree completion (Ifenthaler, 2015). In this
regard, learning analytics are used as an early warning system, which provides stu-
dents with personalised feedback and information about academic support services
to help them to improve their skills and therefore enhance student success (Mah &
Ifenthaler, 2017). Prominent examples for learning analytics systems include course
signals by Purdue University in the USA, predictive analytics at Nottingham Trent
University in the UK, or personalised adaptive study success (PASS) by Open
Universities Australia (Ifenthaler & Schumacher, 2016). Providing students with
meaningful real-time feedback and academic support is crucial for successful stud-
ies (Tinto, 2005). This may be particularly relevant for the first year in higher educa-
tion, as students often decide to leave the institution within this period (Mah &
Ifenthaler, 2018). Thus, many studies and conferences such as the ‘European First
Year Experience Conference’ address this relevant period, showing its importance
for higher education institutions.
Study success includes the successful completion of a first degree in higher educa-
tion to the largest extent and the successful completion of individual learning tasks to
the smallest extent (Sarrico, 2018). The essence here is to capture any positive learn-
ing satisfaction, improvement, or experience during learning. As some of the more
common and broader definitions of study success include terms such as retention,
persistence, and graduation rate, the opposing terms include withdrawal, dropout,
non-completion, attrition, and failure (Mah, 2016). Learning analytics show promise
to enhance study success in higher education (Pistilli & Arnold, 2010). For example,
students often enter higher education academically unprepared and with unrealistic
perceptions and expectations of academic competencies for their studies. Both the
inability to cope with academic requirements and unrealistic perceptions and expec-
tations of university life, in particular with regard to academic competencies, are
important factors for leaving the institution prior to degree completion (Mah, 2016).
However, rigorous empirical research focusing on the impact of learning analyt-
ics on study success and student retention in higher education is still scarce
(Ferguson & Clow, 2017). Sclater, Peasgood, and Mullan (2016) provide a review
of learning analytics in higher education which identifies promising case studies
and their impact on student success. Further, projects like this edited volume or the
17 Epilogue: Future Directions on Learning Analytics to Enhance Study Success 315
project ‘STELA – Utilising Learning Analytics for Study Success’, supported by

the German Ministry of Education and Research, aim to capture empirical evidence
regarding learning analytics’ contribution to student retention and study success in
higher education (Yau, Mah, & Ifenthaler, 2018). The broad collection of chapters
presented in this edited volume provides valuable insights into theoretical consider-
ations and practical experience on learning analytics.
3 Key Themes Emerging from Current Research
Various perspectives on learning analytics have emerged from the chapters of this
book. We identified four major issues in using learning analytics to enhance study
success which shall be highlighted and further analysed: (1) acceptance and compe-
tence for the implementation of learning analytics, (2) personalised learning and
early interventions, (3) data privacy and ethics, and (4) technical considerations.
3.1 Stakeholders’ Acceptance and Competence

for the Implementation of Learning Analytics
Stakeholders’ acceptance and competence regarding learning analytics are crucial

prerequisites for utilising this emerging research field in higher education (Daniel,
2015). Overall, numerous learning analytics frameworks have been proposed con-
sidering important aspects for developing learning analytic projects and for success-
ful implementation in institutions such as by Scheffel (2017), Ifenthaler and
Widanapathirana (2014), or Greller and Drachsler (2012). For example, Leitner
et al. (Chap. 6) describe learning analytics challenges to overcome in higher educa-
tion institutions. With this regard, they present seven main categories for imple-
menting learning analytics initiatives, which include representation and actions,
data, IT infrastructure, development and operation, privacy, ethics, and purpose and
gain. The latter addresses the stakeholders’ perspective on learning analytics and the
relevance and the suggestion to provide the highest possible transparency.
Focusing on students, research such as the LAPS project, described by
Hinkelmann and Jordine (Chap. 7), shows that many students appreciate the feed-
back concerning learning process, risk analysis, and additional support offerings.
Furthermore, research shows that transparent communication and aspects such as
data privacy are important issues for students’ acceptance and a successful imple-
mentation of learning analytics in higher education institutions.
Issues while implementing learning analytics into existing digital platforms and
database systems add another obstacle. Klasen and Ifenthaler (Chap. 4) describe
how a prompting application has been implemented into an existing university envi-
ronment by adding a plugin to the local digital learning platform which injects
user-centric prompts to specific objects within their digital learning environment.
For educators, the competence for interpreting the provided data correctly is nec-
essary. They need to know about the underlying algorithms, which usually include
variables such as socio-demographics, grades, and activities in the learning environ-
ment, as well as about the many reasons that may influence students’ progression
such as illness or general risk factors for student dropout. Thus, advisors should
participate in mandatory consultation workshops to improve this competency
(Hinkelmann and Jordine, Chap. 7).
Overall, multiple benefits can derive from using learning analytics in higher
education institutions; however, Ifenthaler (2017) reveals that there is a lack of
specialised staff with a strong background in learning and teaching as well as in data
science for learning analytic projects.
3.2 P
ersonalised Learning and Early Interventions to Enhance
Student Success
Personalised feedback provides students with information about their learning per-
formance, their likelihood of being successful in a course, and guidance for support
services, helping them to improve their skills. On the basis of the currently available
data, students can receive real-time information about their learning, for instance, as
just-in-time feedback after taking a test or summative in order to understand learn-
ing habits, analyse learning outcomes, or track their progress towards goals
(Ifenthaler, 2015). Personalised feedback on their learning status and their risk sta-
tus may be particularly valuable for first-year students, who are often unsure of what
is expected of them in academic terms. Besides, the emphasis on personalised learn-
ing is essential due to increasing student diversity. For instance, Chernobilsky and
Hayes (Chap. 12) compared nontraditional to traditional student performance in
various online course formats. The results indicate that on average, nontraditional
students are not succeeding in online courses at the same rate as traditional under-
graduate students. Thus, personalised feedback, personalised support recommenda-
tions, and early interventions may be very helpful for students, especially in their
first year of higher education, to keep on track and thus be successful in higher
education (Mah & Ifenthaler, 2017). In the face of growing student numbers, tuition
fee costs, and diversification, Arthars and colleagues (Chap. 13) present the a plat-
form (the Student Relationship Engagement System (SRES)), which directly helps
teachers to act on data to provide at-scale personalised support for study success.
Hawlitschek, Krenz, and Zug (Chap. 5) emphasise the need of adaption in order
to facilitate individualised learning environments and thus to support efficient and
effective learning and reduce high dropout rates. In their study, they used learning
analytics to identify learner characteristics, which are relevant for dropout rates in
computer science courses. Based on their findings, they can automatically detect
learners that got stuck in their learning path and apply interventions suited for the
different needs of these learners. Similarly, Derr and colleagues (Chap. 8) highlight
heterogeneous student groups with different educational backgrounds, knowledge
levels, and needs. Their research provides insight into how web-based preparatory
courses can support the highly heterogeneous student body in the transition from
school education to higher education studies.
3.3 Data Privacy and Ethics
Due to the essential capture and storage of personal data, which is required to
facilitate the implementation of learning analytics models and systems, a number of
data privacy and ethical concerns have been raised. Initially, these concerns pre-
sented huge obstacles and barriers to possible implementations especially due to the
new European General Data Protection Regulation (GDPR) that came into effect on
25 May 2018. The GDPR protects persons with regard to the processing of personal
data and on the free movement of such data. Under the new law, prior to their per-
sonal data being obtained, individuals must give their consent. Institutions utilizing
personal data must make it transparent what data is being stored, the purpose of
the data being stored, and the duration of storage. These obstacles and barriers to
implementations can be overcome with some knowledge and skills required to build
successful and transparent learning analytics systems in higher education institu-
tions as seen in various chapters of this book. It is important to note that some
countries (such as the USA and Australia) have less strict regulations concerning
personal data and theoretically it is simpler to implement learning analytics systems
without considerations to what may breach the law.
Examples of data transparency in European countries include Hinkelmann and
Jordine (Chap. 7). Their concept includes voluntariness, self-determination, and
self-responsibility, respecting individuality, confidentiality, and anonymity. Students
are made known that when they deregister from the course for any reason, their
personal data will no longer be visible to any user of the system. It is possible to
view their personal data and risk analysis only with explicit consent from enrolled
students. Students may decide or change their decision at any time regarding their
participation. Students receive information via email to inform them how their data
is used, the system is presented to students, as well as an information booth (Q & A)
once per semester is available to students. The system adheres fully to the GDPR.
In the work of Derr and colleagues (Chap. 8), the university’s data privacy official
gave ethical approval and that participants of their study were informed on the
purpose of the study and gave consent to their data being collected, anonymised,
and analysed. Interview participants took part voluntarily and were informed that
their data would be kept secured and analysed confidentially.
Another example is described by Klasen and Ifenthaler (Chap. 4) where the
students are tracked via a pseudonymous hash. This enables a collection of students’
data throughout various systems without the necessity to collect further personal
data. It further enables to merge this data with other university-known data like
demographic data and grades at the end of the semester into a complete, anonymous
dataset for further investigation.
3.4 Technical Considerations
The implementations of learning analytics models and systems in higher education

institutions are still relatively new and rare. Technical issues that should be consid-
ered include the availability of technical staff with the necessary skills and knowl-
edge for system maintenance and the corresponding technological resources
(Ifenthaler, 2017).
As Arthars and colleagues (Chap. 13) learnt two important lessons from their
study, technology itself plays a role in influencing and shaping teachers’ teaching
practices, and the way that teachers use the technology may be mismatched with
students’ needs or expectations. They highlighted three implications for learning
analytics practice—(1) address actual needs, (2) start small but provide for growth,
and (3) foster communities. (1) The needs of institutions vary amongst each other as
well as the needs of the teachers and students within the institutions. Therefore, it is
very important to support a bottom-up implementation/approach of learning analyt-
ics, which is to support teachers collect and use meaningful data relevant to their
pedagogical needs. The benefits of this can include greater workload efficiency in
data entry, analysis, and communication with students. (2) Some of their teachers
also acknowledged that they found simple functions (such as collecting attendance)
to be useful. Allowing simple as well as complex functions in the system is there-
fore advised to allow teachers to apply functions of their choice and that they are
comfortable with. (3) Having a learning and teaching support unit available could
make a large difference in supporting staff in using the system and providing
support to the relevant areas such as learning design, educational technology, and/or
software development expertise (see also Chap. 4 by Klasen and Ifenthaler).
4 Future Directions on Learning Analytics
On the basis of the previous chapters, many areas for future research and future
directions can be identified. Overall, more longitudinal research is needed to pro-
vide insight into how learning analytics impact learning and teaching in higher edu-
cation. With this regard, Wong and colleagues (Chap. 1) emphasise the importance
of taking learning theories into account when employing learning analytics in stud-
ies to support study success. They conducted a review on whether learning theories
were integrated in the utilisation of learning analytics. Their results showed that
self-regulated learning, motivation, and social constructivism theories were used in
studies utilising learning analytics. However, at present, these studies are mostly
correlational and thus lacking experimental and empirical data (see also Chap. 2 by
Ifenthaler, Yau, and Mah). Initial work has been conducted on how to facilitate
educational research employing learning theories to guide the data collection and
analyses of the learning analytics and forms the basis for future work.
Arthars and colleagues (Chap. 13) emphasised the importance of personalising
learning support for each individual student and developed a platform utilising
learning analytics to fulfil this aim. Precisely, the identified future works include the
evolving uses of students’ data, the factors that lead to the inclusion and analysis of
this data, and enabling teachers to personalise the learning experiences for students.
Similarly, in Hawlitschek, Krenz, and Zug (Chap. 5), the identified future research
includes improving the detection of error streaks, which are used to analyse the
specific attributes of at-risk students. Following this, employment of more sophisti-
cated and individually personalised support, guidance, and assistance can be more
successfully utilised, thus forming a good foundation for future directions.
The implementation of learning analytics systems into higher education institu-
tions is not a straightforward process in any domain or country (see Chap. 4 by
Klasen and Ifenthaler). Leitner and colleagues (Chap. 6) presented six challenges
and possible ways to overcome these forming further research directions in learning
analytics. The challenges are as follows: (1) shortage of leadership, (2) shortage
of equal engagement, (3) shortage of pedagogy-based approaches, (4) shortage of
sufficient training, (5) shortage of studies empirically validating the impact, and
(6) shortage of learning analytics-specific policies.
Automated analysis is another important issue for future directions. Due to growing
student numbers, personalised face-to-face feedback is challenging to provide.
Thus, automated feedback processes derived from learning analytics systems
may be helpful to deal with the limited time resources. Bektik (Chap. 9) describes
writing analytics that focuses on the measurement and analysis of written texts to
improve the teaching and learning of writing. The chapter highlights various chal-
lenges and ethical considerations when using automated text analysis based on
machines. Machines and human markers should complement each other, with the
aim of providing better feedback to students. This perspective is crucial when talk-
ing about algorithms, machine learning, and artificial intelligence in general. Online
courses become more and more established and thus more data will be available for
analyses. Based on the vast amount of student data, advanced learning analytics
may identify known but also new student patterns (e.g. through machine learning,
deep learning) regarding behaviour, preferences, and study success factors, for
instance. Likewise, Hinkelmann and Jordine (Chap. 7) described an automated
process (involving three phases – extraction, transformation, and load), which can
be utilised in the future to improve the upload of new student and examination data.
Additionally, these can be anonymised to protect the students’ privacy.
Another future research field may be the use of multimodal analysis (e.g. eye
tracking data). For instance, using eye tracking for digital material or online courses,
various information can be captured such as view time, preferences, and difficulties.
Together with learning analytics data, adaptive and personalised learning content
could be generated which may help to provide a personalised learning journey.
5 Conclusion
Learning analytics becomes more and more established in higher education institu-
tions. The expectations in this emerging research field are high, and current research
supports its promise to positively impact student retention. The chapters of this
edited volume provide in-depth insights and understanding of the current state of
utilising learning analytics to enhance study success in higher education. There
remain serious challenges and concerns, while more and more higher education
institutions embrace learning analytics (Ifenthaler, 2015):
• Not all educational data is relevant and equivalent.
• Learning analytics need to grow as an interdisciplinary field including (but not
limited to) learning science, educational psychology, data science, learning
design, and computer science.
• Ethical issues and data privacy need to be considered by all stakeholders when
building learning analytics strategies for higher education institutions.
• Limited access to educational data and analytics algorithms generates disadvan-
tages for involved stakeholders.
• Continuous professional learning is required for the preparation of stakeholders
involved in learning analytics.
• Information from distributed networks and unstructured data cannot be directly
linked to educational data; hence, data quality needs to be confirmed before
interventions can be provided to stakeholders.
• Technical frameworks and organisational change management need to be in
place before learning analytics can be implemented in higher education.
• Learning analytics need to move beyond the collection and analysis of numerical
data (e.g. click streams)—A qualitative analysis of semantic-rich data (e.g. con-
tent of discussion forums, responses to open-ended assessments) enables a better
understanding of learning processes and possible misconceptions.
In sum, learning analytics are moving towards a mature field of research and
development. A broader (and system-wide) adoption of learning analytics will pro-
vide new testbeds for empirical research. In addition, the growing field of learning
analytics also requires experimental and quasi-experimental investigations demon-
strating the validity of learning analytics to support learning and teaching as well as
study success.
References
Daniel, B. (2015). Big data and analytics in higher education: Opportunities and challenges.
British Journal of Educational Psychology, 46(5), 904–920. https://doi.org/10.1111/bjet.12230
Ferguson, R., & Clow, D. (2017). Where is the evidence? A call to action for learning analyt-
ics. In LAK ’17 Proceedings of the Seventh International Learning Analytics & Knowledge
Conference (pp. 55–65). New York, NY: ACM.
Ifenthaler, D. (2017). Are higher education institutions prepared for learning analytics? TechTrends,
61, 366.
Ifenthaler, D., & Schumacher, C. (2016). Learning Analytics im Hochschulkontext. WiSt-

Wirtschaftswissenschaftliches Studium, 45(4), 172–177.
Learning, 19(1-2), 221–240.
Long, P. D., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education.
Educause Review, 46(5), 31–40.
Mah, D.-K. (2016). Learning analytics and digital badges: Potential impact on student reten-
tion in higher education. Technology, Knowledge and Learning, 21(3), 285–305. https://doi.
org/10.1007/s10758-016-9286-8
Mah, D.-K., & Ifenthaler, D. (2017). Academic staff perspectives on first-year students’ academic
competencies. Journal of Applied Research in Higher Education, 9(4), 630–640. https://doi.
org/10.1108/JARHE-03-2017-0023
Mah, D.-K., & Ifenthaler, D. (2018). Students’ perceptions toward academic competencies: The
case of German first-year students. Issues in Educational Research, 28(1), 120–137.
Pistilli, M. D., & Arnold, K. E. (2010). Purdue Signals: Mining real-time academic data to enhance
student success. About campus: Enriching the student learning experience, 15(3), 22–24.
Sarrico, C. S. (2018). Completion and retention in higher education. In J. C. Shin & P. Teixeira
(Eds.), Encyclopedia of international higher education systems and institutions. Dordrecht,
The Netherlands: Springer.
Scheffel, M. (2017). The evaluation framework for learning analytics. Heerlen, The Netherlands:
Welten Institute at the Open Universiteit.
Sclater, N., Peasgood, A., & Mullan, J. (2016). Learning analytics in higher education. A review
of UK and international practice. Full report. Retrieved from https://www.jisc.ac.uk/sites/
default/files/learning-analytics-in-he-v3.pdf
Tinto, V. (2005). Reflections on student retention and persistence: Moving to a theory of institu-
tional action on behalf of student success. Studies in Learning, Evaluation, Innovation and
Developmental Psychology, 2(3), 89–97.
Yau, J. Y.-K., Mah, D.-K., & Ifenthaler, D. (2018, June 25). Utilizing learning analytics for
study success: A systematic review. Paper presented at the EdMedia Conference, Amsterdam,
Netherlands.
Index
A C
Aalen University of Applied Sciences Campus-Management-Systems (CMS), 107
study design Cognitive load theory (CLT), 77, 83
data base and protection, 44 Cross-institutional study, 230
effectiveness, supportive measures, 44
(see Exemplary data analyses)
research method, 45–47 D
research questions, 45 Data manipulation and analysis, 227, 229
Study Support Center (SSC), 42–43 Data privacy and ethics, 31, 314, 315, 317
Academic misconduct, 297, 299–301, 306, Data transparency, 317
308, 310 Dependent variables, 257
Academic writing, 143, 144, 148 Developing countries, 249, 250, 252, 266
Active learners, 278 Discussion forums, 284, 290, 291
Active learning behaviors, 282 Dropout
Adaptive learning models, 28, 33, 159 adaptive learning environment, 82
Automated analysis, 319 adaptivity, 74
Automated essay scoring (AES), 146 analyses of variance (ANOVA), 81
Automated text analysis, 145–146 cognitive load group comparisons, 82
cognitive variables, 82, 83
description, course, 79
B error streaks, 84
Behavioral variables, 260, 261 extraneous cognitive load, 84
Biology course, 239 face-to-face instruction and online study, 82
Blended learning field of computer science, 73
digital learning environments, 75 heterogeneity of students, 73
dropout intrinsic motivation/self-efficacy, 83
cognitive load, 77–78 knowledge group comparison, 81
motivation, 76 learning management system, 74
user behavior, 78–79 methods and instrumentation, 79–80
e-learning and face-to-face-learning, 75 motivation and interest, 73
internal factors, 75 motivation group comparison, 81
learning performance, 75 principal component analysis, 82
STEM subjects, 75 process of programming, 83
Business Capstone, 297, 298, 301, 302 sociodemographic factors, 73

https://doi.org/10.1007/978-3-319-64792-0
324 Index
E eTests, 298, 299, 304, 306

Earning beliefs, 262 Ethics, 31, 32
eAssessment, 296 European Learning Analytics Exchange
Educational data mining (EDM), 28, 39, 161 (LACE) project, 102
Elational pedagogy, 224 Evaluation, 39, 43, 44, 47
e-Learning Exemplary data analyses
A-level mathematics classes, 121 educational biographies, 47, 49
attitude, mathematics, 129 entry-level competences, 47
behavior, 119 influence of, 47
cognitive predictors, 121 learning outcomes, 52
data, 119 pre-course of students, 51
data collection, 124, 135–136 semester-accompanying tutorials, 48
economics, 121 SSC DPS, 50
effects, pre-course participation, 126–129 Extraneous cognitive load (ECL), 77
ethical considerations, 125
evaluation of, 120
face-to-face/e-tutoring support, 131–132 F
identification of variables, 133–135 Facebook, 259
learning environments, 122 Facebook graph API, 259
liminal phase, 119 Factor analysis, 260, 262, 264, 266
limitations, 125 Faculty of Education and Languages
mathematics and study success, (FELS), 148
engineering, 126 Feedback questionnaire, 307
measured learner behavior, 120 First-year students, 42, 44, 47, 52
metacognitive strategies, 121 Frequently asked questions (FAQs), 307
pre-course design, 122–124
pre-course learning/academic
achievement, 120 G
pre-course outcomes, 120 General Data Protection Regulation (GDPR),
qualitative data, 125 63, 91, 101, 317
quantitative data, 124 German Federal Government, 37
self-evaluation and self-reflection, 133 Google Chrome browser, 303
self-regulated learning, 120, 121 Grade point average (GPA), 121, 126
social interaction, help-seeking and peer Graph comparison methods
learning, 132 argumentation, 194
STEM subjects, 119 concept map, 176
task strategies, 131 didactic and lecture-based methods, 175
technical faculties, 119 educational research, 177
time management and organizational field of education, 177, 193
strategies, 130 GED, 181, 183
time on task, 130 graph embeddings, 182, 186
transition phase, 136–137 graph kernels, 182, 184
web-based learning environments, 119 implementations
Engineering Institute of Technology (EIT), 302 influence of, 187–188
English First English Proficiency Index (EF methods selection, 188–190
EPI), 251 verification, 190–191
English language education, 251 inquiry-based learning, 192
English language learning (ELL), 249, 251 instructional strategies, 175
learning environment, 252 knowledge representation, 192
online courses, 252 limitation, 194
programs, 251 and machine learning, 181
English proficiency levels, 251 multiple-choice approach, 176
eQuizzes, 299 pattern recognition, 177
Error streaks, 84–86 problem-centered instruction, 177
Index 325
problem-solving approach, 175 quality assurance, 114

problem-solving contexts, 175 risk dimensions, 109
reference expert map, 177 risk of termination, 110
referent-free, 177 rules, 109
student’s representation of knowledge, 176 student information, 113–114
Graph edit distance (GED), 181–184 student response rate, 116
Graph embeddings, 186 students’ gender information, 115
Graph kernels, 184–186 study progression analysis approaches, 106
training phase, 108
university’s student and learning support
H center, 105
Higher education institutes (HEIs), 37 Learning analytics (LA), 4, 225, 253
absence, learning theories, 18
advantage, 243
I analysis, 306
Ill-structured problems (ISP) Australasian, 225
assessment, 179 benefits and issues, 39
developing knowledge structures, 178 challenges and dimensions, 93
Incremental Thesaurus for Assessing Causal characterized, 225
Maps (ITACM) software, 187 code of conduct, 94
Independent variables, 255–257 data and computer science, 61
Information technology (IT), 99 data collection, 226
Instructional design, 276 data literacy and evaluations, 95
Intelligent tutoring systems (ITSs), 3, 159 data privacy, 63–64, 69, 70
Interactive learning environments, 3 data representation, 91
Internal vs. online, 301 digital learning management system, 71
Intrinsic cognitive load (ICL), 77 digital learning material, 38
digital learning platform, 66–67
digital platforms and database systems, 315
J educational research and practice, 6
Journal Citation Reports (JCR), 7 ethical guidelines, 92
ethics and privacy, 92
feedback, 39, 41
K higher education institutions, 316, 317
Kruskal-Wallis H test, 288 implementations, 318
data, 98
development and operation, 100
L ethics, 102
LAPS project framework, 96
Apriori algorithm, 109 IT infrastructure, 99
consultation of students, 116 privacy, 100
data analysis, 116 purpose and gain, 96–97
data basis, 107–108 representation and actions, 97
data collection and big data analysis, 106 insufficient evidence, 31–32
data model, 115 and intervention measures, 32
development, 108 JavaScript libraries, 71
feasibility study, 111 LAPS project, 315
German academic system, 106 LeAP application, 62, 63
learning analytics process, 106 LeAP core component, 64–65
machine learning methods, 108 learning progress, 70
machine learning techniques, 105 MCQs, 296
predictor models, 106 motivation, 14
principle, machine learning, 108 personalised support are prompts, 62
privacy and ethics, 111–113 policies, 28
326 Index
Learning analytics (LA) (cont.) learning materials, 4

policy- and decision-making, 91 levels of education, 3
positive evidence, 29–31 online learning environments, 3, 19
practical tool, 92 study success, 4, 6, 19, 21
practice, 28, 242 trace-based methodology, 20
privacy and data protection, 94 Zimmerman’s model, 20
prompts, 67–69
remote invigilation, 296
research focus, 68 M
research methodology, 28–29 Mann-Whitney U tests, 288
review methodology, 6–8 Marketing course, 240
scientific community, 94 Massive Open Online Course (MOOC)
self-regulated learning, 61 ANOVA tests, 278
social constructivism, 15 components, 278, 291
specific learning concepts course materials, 283
assessment and learner effort, 16 data sources, 279
deep learning, 17 enroll, 288, 292
educational researchers, 16 exercises/assignments, 284
implicit science knowledge, 17 genders, 284
learning-related concepts, 18 goal setting and strategic planning
learning theories, 10–12 activities, 13
post-passing behaviours, 16 instructor, 277
psychological and behavioural learners’ behavioral patterns, 276
characteristics, 17 learning analytics, 275, 277
role of feedback, 17 literature, 290
team-based discussion, 18 overall behavioral patterns, 290
three-dimensional analysis, 17 perceptions, 288
SRL, 9 platform, 275
stakeholders, 40–42, 315 SRL survey data, 9
strategic planning and policy, 94 usage patterns, 291
student behaviour, 28, 310 Mathematics
supervising institutions, 94 attitude, 129–130
teachers and students, 225, 226 data, 119
technology, 306 measured learner behavior, 120
tools, 226, 241 regression analysis, 128
tracking data, 68 and study success, 126
workflow, 226 technical faculties, 119
Learning Analytics and Knowledge conference web-based pre-course, 122
2018 (LAK18), 102 Ministry of Education, 251
Learning dashboard for Insights and Support Mixed-methods methodology, 231
during Study Advice (LISSA), 41 Mobile learning analytics
Learning management systems (LMS), 3, 9, applications, 166
123, 217, 250, 295 blended learning, 157
Learning strategy, 120, 121, 130, 135, 136 challenges, stakeholders, 167–168
Learning theories definition and features, 160–161
dashboards and educational technologies, 22 digital media and learning, 158
educational research, 4 higher education institutions, 158
employment, 19 interim conclusion, 163
evolution of, 5, 6 learning analytics, 161–162
holistic and social process, 20 mobile devices and Internet access, 157
implicit knowledge, 21 MyLA app, 163–164
instructional design, 4 MyLA dashboard, 164–166
iterative loop, 21 opportunities for stakeholders, 166–167
learning analytics, 4 privacy and ethical standards, 158
Index 327
qualitative testing, 166 Personalised learning, 314

ROMA, 170 Personalised learning environments, 33
smart environments, 159–160 Personalization, 224, 227, 241
technology-enhanced learning (TEL) Personalized learning, 227, 315
complex, 169, 170 Peruvian university students, 255
Motivated Strategies for Learning Predictors, 120, 121, 134, 135
Questionnaire (MSLQ), 130
R
N Randomized controlled trial (RCT), 193
Natural language processing (NLP), 144, 145 Rapid Outcome Mapping Approach (ROMA),
94, 170–171
Readiness survey, 257
O Readiness variables, 261, 262
Online courses, 252, 319 Regression model, 266
Online language learning, 250, 266 Relational pedagogy, 225, 242
Online learning, 251 Remote invigilation, 296, 297, 301–303
adult students, 206, 211 Remote invigilation software, 302, 303, 306
andragogy approaches, 204 Research-based questionnaire, 306
course content type, 214 Research variables
data, 201 independent variables, 255
distance learning, 204 readiness survey, 257
elements, 204 student readiness variables, 256
flexibilities, 205 Review of Educational Research (RER), 5
formats, 213
groups, 212
higher education, 201 S
implications, 216 Scimago Journal and Country Rank (SJR), 7
learning analytics and data mining, 202–204 Self-determination theory (SDT), 250
limitations, 217 autonomy, 254
massive enterprise-level engagements, 201 competence, 254
methodology motivation, 254
data preparation, 210 relatedness, 254
data sources, 208–210 traditional learning environments, 254
setting, 207 Self-regulated learning (SRL), 9
university, 208 Smart learning environments (SLE), 159
outcomes, 205 Spring-Boot applications, 64
quasi-randomized study, 205 SSC data privacy statement (SSC DPS), 51
researchers, 211, 212 Staff perspective, 308, 309
retention issues, 206–207 Stakeholders
social and economic factors, 202 Aalen UAS, 52
student type and course format, 211 feedback, 53, 55
variable of interest, 213 macro-level, 54
Online learning environments, 3 mega-level, 55
Online study modes, 300 meso-level analytics, 53
Open education research (OER), 188 microlevel analytics, 53
Open Universities Australia (OUA), 297 Statistical analyses, 259
Optional resources, 282 Student engagement, 223
Student information system (SIS), 44
Student learning success, 253
P Student-Life-Cycle, 107
Passive learning behaviors, 280, 281 Student persistence, 250, 257, 264, 265, 269
Personalised adaptive study success (PASS), 314 Student readiness survey, 258
Personalised feedback, 316 Student readiness variables, 256
328 Index
Student relationship engagement system (SRES) Transparency, 111

adoption, 243 Trends in International Mathematics and
assessment feedback, 234 Science Study (TIMSS), 129
communications, 231
data collection, 228
data curation, 229 U
design and development, 236 Uniform Resource Locator (URL), 302
empowered, 242 Universities of Applied Sciences (UAS), 37
integration and adoption, 240 University entrance qualifications (UEQ), 37
LA data, 228 University of Melbourne, 232, 238
online participation, 234 University of New South Wales (UNSW)
participants, 236 Sydney, 227
teacher intelligence, 229 University of Sydney, 236, 245
tutors, 235
UNSW Sydney, 238
use, 238 V
Student Relationship Engagement System Virtual learning environment (VLE), 32
(SRES), 41
Student success, 223, 313, 314
Study entry phase, 37, 44, 45, 54–56 W
Study success, 314 Writing analytics
digital learning environments, 34 argumentation, 145
digital learning platforms, 33 educational contexts, 143
higher education institutions, 34 effective written communication, 143
limitations, 34 ethical concerns, 146
personalised prevention and intervention focus group
strategies, 33 belief, 150
Study Support Center (SSC), 42 FELS, 148
Supporting Higher Education to Integrate non-verbal actions, 150
Learning Analytics (SHEILA) power and politics, 151
project, 94 problems, 152
Support vector machines (SVM), 278 qualitative thematic analysis, 150
Systems thinking, 185, 193 XIP analysis, 148
higher education, 144
labour-intensive process, 143
T one-to-one interviews, 147, 149
Teacher adoption, 226
Technology Acceptance Model (TAM), 76
Technology-Enhanced Learning (TEL), 91 X
Time spent learning, 262 Xerox Incremental Parser (XIP), 144, 145

2019 Book UtilizingLearningAnalyticsToSu

Uploaded by

Copyright:

Available Formats

2019 Book UtilizingLearningAnalyticsToSu

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2019 Book UtilizingLearningAnalyticsToSu

Uploaded by

Copyright:

Available Formats

Dirk Ifenthaler · Dana-Kristin Mah

Utilizing Learning Analytics

Jane Yin-Kim Yau

ISBN 978-3-319-64791-3 ISBN 978-3-319-64792-0 (eBook)

Library of Congress Control Number: 2018968406

© Springer Nature Switzerland AG 2019

Advances in educational technology have enabled opportunities to provide insight

undergraduate course registrants through mining a dataset to determine trends and

Mannheim, BW, Germany Dirk Ifenthaler

Part I Theoretical and Technological Perspectives

Part II Issues and Challenges for Implementing Learning Analytics

7 The LAPS Project: Using Machine Learning

Part III Learning Analytics Case Studies: Practices and Evidence

Dirk Ifenthaler is Professor and Chair of Learning, Design, and Technology at the

Dana-Kristin Mah is a researcher and consultant in the field of educational tech-

Jane Yin-Kim Yau is a researcher in Learning Analytics and Mobile Learning at

Mohammed Zaki Ahmed is an Associate Professor (Senior Lecturer) in

Natasha Arthars is a current PhD candidate at the Centre for Research on Learning

Martine Baars is an Assistant Professor of Educational Psychology at the Erasmus

Michael Baird is a lecturer in the School of Marketing at Curtin University,

Duygu Bektik is a mixed methods researcher and currently works at the Institute

academia and empowering women in academia/STEM. She has worked on several

Chenglu Li is a doctoral student in Learning Technologies Programme and has a

Ellina Chernobilsky is a Professor in the School of Education at Caldwell

Anat Cohen is a tenured-track senior academic staff member at Tel Aviv University’s

Dan Davis is a PhD candidate at Delft University of Technology. He develops

Björn B. de Koning is an Assistant Professor of Educational Psychology in the

understanding the influence of collaboration on learning processes and perfor-

Mollie Dollinger is an associate lecturer at La Trobe University where she

Martin Ebner is currently Head of the Department Educational Technology at

Armin Egetenmeier studied business mathematics at the University of Ulm. He

Eitan Festinger is a second-year MA student at the Tel Aviv University’s School

Philippe J. Giabbanelli, PhD is an Assistant Professor of Computer Science at

Danny Glick is an educational researcher, learning technologist, and TMLL

Vishrant K. Gupta is a graduate student supervised by Dr. Giabbanelli. His thesis

Anja Hawlitschek is currently working as research associate at Magdeburg-­

Susan Hayes is the Director of Institutional Research and Assessment at Caldwell

Mathias Hinkelmann received a diploma (Dipl-Ing) in mechanical engineering in

Miriam Hommel received her diploma in geodesy and geoinformatics from

Andrea Honal is a Business Professor of Management, Marketing, and Media at

Geert-Jan Houben is Full Professor of Web Information Systems at the Software

Reinhold Hübl, PhD (1987), is a Professor at Baden-Wuerttemberg Cooperative

Dirk Ifenthaler is a Professor and Chair of Learning, Design and Technology at

Mohammad Khalil is a postdoctoral candidate at Delft University of Technology,

Deborah M. King is an Associate Dean (undergraduate programmes) in the

tertiary learning and teaching, particularly through the professional development of

Elsuida Kondo is a developer of learning and teaching for the Faculty of Science

Matthias Kuhnel studied information design and media management at the

Philipp Leitner is currently working for the Department of Educational Technology

Qiujie Li is a PhD candidate in the Department of Education at the University of

Danny Y.-T. Liu is a molecular biologist by training, programmer by night,

Min Liu is a Professor of Learning Technologies at the University of Texas at

Dana-Kristin Mah is a researcher and consultant in the field of educational tech-

Fred Paas is a Professor of Educational Psychology at Erasmus University

Connie Price is a senior curriculum consultant specializing in assessment design.

Luisa Seiler studied pedagogy and business education at the University of

Lesley Sefcik is a lecturer and academic integrity advisor at Curtin University. She

Current projects include the development and implementation of academic integrity

Steve Steyn is a professional engineer and certified project manager with a decade

Mannheim, BW, Germany Dirk Ifenthaler

Part I Theoretical and Technological Perspectives

Part II Issues and Challenges for Implementing Learning Analytics

7 The LAPS Project: Using Machine Learning

Part III Learning Analytics Case Studies: Practices and Evidence

Anja Hawlitschek is currently working as research associate at Magdeburg-

2.1 Evolution of Learning Theories

3 Role of Educational Theories in Learning Analytics

3.1 Research Methodology

3.2 Results and Discussion

3.2.1 Learning Theories and Learning Analytics Applications

Learning theories Learning LA technique/ Performance-related

Learning theories Learning LA technique/ Performance-related

3.2.2 Absence of Learning Theories

4 Conclusion and Suggestions for Future Research

2.1 Research Methodology

2.2 Results of the Critical Reflection

2.2.2 Insufficient Evidence on the Use of LA to Support Study Success

1.1 Learning Analytics

1.2 Benefits and Issues of Learning Analytics

1.3 Stakeholders of Learning Analytics

2.1 Study Support Center

2.2 Study Design

2.2.1 Database and Data Protection

2.2.2 Research Questions

2.2.3 Research Method

2.2.4 Exemplary Analyses