The Impact of Learning Analytics on the Dutch Education System

Hendrik Drachsler
Welten Institute, Open University of the Netherlands [email protected]

Slavi Stoyanov
Welten Institute, Open University of the Netherlands [email protected]

Marcus Specht
Welten Institute, Open University of the Netherlands [email protected]

ABSTRACT
The article reports the findings of a Group Concept Mapping study that was conducted within the framework of the Learning Analytics Summer Institute (LASI) in the Netherlands. Learning Analytics are expected to be beneficial for students and teacher empowerment, personalization, research on learning design, and feedback for performance. The study depicted some management and economics issues and identified some possible treats. No differences were found between novices and experts on how important and feasible are changes in education triggered by Learning Analytics.

build community around the Learning Analytics (LA) and Educational Data Mining (EDM) by involving various stakeholders from educational science, data & computer science, policy makers, and learning technology start-up companies. To facilitate accomplishment of the LASI objectives a Group Concept Mapping study was conducted. The intention was to go beyond the-state-of-the-art of LA [1,2] and EDM [3,4], to identify the potentials and risks of LA in the Netherlands and to project the expectations of the LA stakeholders on the impact of LA on the Dutch education system. Most of the state-of-the-art publications indicate that LA & EDM are a mean to get more advanced research methods for education. The underlying assumption is that educators will be able to better assess their pedagogical practices to improve education. The main expectations towards LA could roughly be stated as follows: Awareness and reflection support [4] Improve educational decision-making and clearer goal setting [5] More timely and frequent feedback for students and teachers [6] Self-regulated and personalized learning [4,7] Creation of richer learning (behavior) data to facilitate educational research [8]

Categories and Subject Descriptors

J.1 [Administrative Data Processing] Education; K.3.1 [Computer Uses in Education] Collaborative learning, Computer-assisted instruction (CAI), Computer-managed instruction (CMI), Distance learning, A.1; [Introductory and survey]; H.1.1 [Information Systems] Models and principles, Systems and Information Theory; J.4 [Social and behavioral sciences].

General Terms
Measurement, Documentation, Design, Human Factors, Theory.

Keywords
Learning analytics, group concept mapping, focus group, community building.

1. INTRODUCTION
The study was conducted within the framework of the Learning Analytics Summer Institute (LASI) in Amsterdam, a strategic two-days event (July 4 -5, 2013) co-organized by SoLAR and the Dutch SURF Special Interest Group (SIG) on Learning Analytics. The LASI event Amsterdam was part of a global network of 10 LASI events in various countries worldwide. The objectives of the Dutch LASI was to stimulate cross-disciplinary interactions, identify research and teaching needs, share best practices and
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. LAK14, March 2428, 2014, Indianapolis, Indiana, USA. Copyright 2014 ACM 978-1-4503-2664-3/14/03$15.00. http://dx.doi.org/10.1145/2567574.2567617

In addition, concerns have been expressed in the literature that many recent LA technologies are detached from pedagogical experiences and practices [4]. Various authors emphasize that LA is more than a series of clicks and page visits, and that it needs to support learners to gain better insights into their learning progress [9]. With this background information in mind we had certain expectations in regard to the outcomes of the Group Concept Mapping (GCM) study: 1. The clusters depicted by the GCM study will be comparable to the thematic areas described in current state-of-the-art articles about LA. There will be differences in the expectations of the novices and the experts in the LASI event. The novice participants would be more skeptical than the experts about the benefits of LA and more concerned with the potential risks caused by LA.

2.

2. METHOD 2.1 Group Concept Mapping

Group Concept Mapping (GCM) is a research methodology that facilitates a group of people to arrive at a shared vision about a particular issue (e.g. the impact of LA on education). GCM differs from other methods for collecting and analyzing opinions (interviews, questionnaires, affinity diagram, delphi, collective classical concept mapping) in three substantial ways. Firstly, it is the participants, not the researchers who generate and structure the ideas. Secondly, the methodology implements some advanced multivariate statistics mainly Multidimensional scaling and Hierarchical Cluster Analysis that objectively identity emerging patterns in the data. Thirdly, the methodology presents the results in visual formats (conceptual maps, pattern matches and gozones) that are easy to grasp for further interpretation and implementation into practice [9].

2.2 Procedure
The participants in the LASI event were introduced to GCM and the kinds of outcomes they might expect. They were then invited to visit an online environment based on Concept System Global platform1 to brainstorm ideas on the impact of Learning Analytics on the Dutch education completing the following focus prompt: One specific change that Learning Analytics will trigger in Dutch education is... Each statement should be a short phrase expressing one idea only. In the next step of the procedure, the participants were asked to sort the statements into groups of similarity in meaning and label them. Next to sorting, the participants rated on 1-to-5 scale the statements on two values: importance (5 = very important; 1 = not important at all) and feasibility (5 = very feasible; 1 = not feasible at all). Sorting and rating were also facilitated through the online environment. Figure 1: Point map The two-dimensional graphical configuration represents the ideas on the impact of LA on Dutch education (as points on the map) and shows how they are related. The closer the points are to each other, the closer in meaning they are. This is a result of more people grouping them together during the sorting. The extent to which this mathematical model represents the original judgment of the participants can be determined by stress value (index), a routine statistics produced by MDS. For GCM studies it should be in the range between .05 and .35. The stress index of this study is .24, which indicates that the map represents very well the original sorting of the participants. In addition, MDS assigns each statement a bridging value, which is between 0 and 1. A low bridging value means that a statement has been grouped together with statements around it. A higher bridging value means that the statement has been grouped together with some statements further apart from the either side. Some groups of ideas about changes LA will bring to the Dutch education system can already be detected by a simple visual inspection, but to make the process more efficient, we applied Hierarchical Cluster Analysis (HCA). To determine the number of clusters that best reflects the data we checked different solutions provided by the HCA, starting from 16-cluster solutions until we arrive at 5-cluster solutions. This is inline with the GCM methodology, as most GCM studies report between 5 and 16 cluster solutions [10]. We prepared further a checklist with the suggestions made by the HCA for merging clusters to determine the best fitting solution. At each step of the merging two researchers indicated whether they agreed, undecided, or disagreed with the suggestion. After completing the assignment, the analysis team looked at the worksheets to determine where the agreed proposals moved to disagreed ones and selected the 7-clusters solution as the best fitting solution as shown in Figure 2.

2.3 Participants
31 participants took part in the idea generation phase. 63 were invited to participate in the second phase of the study, which required sorting and rating the statements. Of them, 39 completed the sorting, 36 the rating on importance and 34 the rating on feasibility.

3. RESULTS 3.1 Idea generation

31 participants of the Dutch LASI event generated 141 ideas. After idea cleaning and pruning (splitting statements containing more than one idea; removing identical and vague statements) the number of statements were reduced to 108. The GCM methodology suggests that the final list of statements for sorting and rating should not exceed 125.

3.2 Analysis of Sorting Data

Figure 1. presents the first result from the Multidimensional scaling (MDS) of the sorting data.

http://www.conceptsystemsglobal.com/learninganalytics/brainstorm

context will be better adapted to the needs and challenges of each individual student, and More differentiation in learning paths / individualization. The statements in the cluster Research & Learning Design emphasize the need for evidence-based solutions to learning design and teaching. Representative statements in this cluster are: New innovations will be more tested and evidence-based, Indicate whether certain educational activities are effective for the learning process of the students, and Improve educational research by providing a method for investigating effects of instructions on learning outcomes and transfer. Teacher Empowerment highlights the positive effect of LA on teaching. Some of the statements in this clusters are: Teachers get more insight into the effectiveness of learning content and assessments, Empowers teachers to get a dynamic view over time of student activity and if needed adapt the educational design accordingly and Empowers teachers to identify problem areas within their course. Feedback & Performance reflects the expectation of the participants for improvements in providing feedback, assessment and reflection. LA would shift the focus on passing the test to mastering the content, performance and participation. Some statements that compose this cluster are: More sophisticated assessments, Improves learning outcomes by providing insight into learners' misconceptions, and Measure effects of learning by facilitating accurate and direct feedback. The cluster Risks is about negative effects and drawbacks of LA in terms of data ownership, privacy, dependency on IT systems, and cheating. Statements representative for this cluster are: Will impose data dictatorship, Possible wrong priorities caused by misconceptions of data analyses (relationship is not cause) and Privacy issues in education will come more to the fore. Management & Economics is about monitoring and control functions of LA. The effects of LA on the costs of education are also discussed. Some representative statements are: Policy makers will use LA to cut costs, School leaders will be more transparent about their schools' performance to external stakeholders and Competitiveness between higher education organizations will increase because of learning analysis benchmarking. Very coherent clusters have a bridging value close to 0. This indicating that the participants tend to agreed on the content of the cluster. The most coherent cluster has been Students Empowerment (average bridging value of 0.19), followed by Personalization (0.29), Research & Learning Design (0.48), Teacher Empowerment (0.40), Feedback & Performance (0.51), Risks (0.61), and Management & Economics (0.65). The distances on the GCM map indicates that the relationships not only between individual statements but also between clusters. The closer the clusters are to each other the stronger the relationship between them is. The concept map in Figure 5 suggests two bigger areas of clusters. One (say east coast) consists of five clusters (Students Empowerment, Personalization, Research & Learning Design, Teacher Empowerment, Feedback & Performance and another (west coast) includes two (Risks, and Management & Economics). The clusters within each of these larger areas are related in some way but there seems to be no relationships between the clusters representing different areas (e.g. Personalization and Risks). In addition, Personalization

Figure 2: Seven clusters solution The next step in making sense of the data was to attach meaningful labels to the clusters. There are three methods available for entitling the clusters. The first method is to check what labels the GCM system proposes. The second method is to look at the bridging values of the statements included in the cluster. The statements with lower bridging values better represent a cluster. The third method is simply to read through all the statements in a cluster and to apprehend its overall theme. To define the cluster labels we combined all the three methods. The following seven clusters were identified: 1. Student Empowerment, 2. Personalization, 3. Research & Learning Design, 4. Teacher Empowerment, 5. Feedback & Performance, 6. Risks, and 7. Management & Economics (see Figure 3).

Figure 3: Seven clusters solution with labels Students Empowerment is about the benefits LA brings to students in terms of better insights into their weak and strong characteristics, self-regulated learning, tracking learning progress, and increase their responsibility and employability. Some representative statements in the cluster are: Can make students aware of their strengths and weaknesses and where they can improve, Can provide students with insights into effectiveness of self-regulated learning - show the relation between predictions of performance and actual learning outcomes and It empowers learners to steer their own learning process. Personalization is operationalized through key terms such as differentiation of learning paths, individualization based on capacity and level of knowledge, development of recommender systems and just-in-time feedback. Some statements included in this cluster are: Personalization of education by giving the student insight into her/his learning capabilities, Content and

apparently is more related to Students Empowerment rather than to Teacher Empowerment and Research & Learning Design, the last two have been seen more closely related to each other. Feedback & Performance plays a bridging role between the two pairs clusters in the east coast area. They are relevant for Student and Teacher empowerment.

3.3 Analysis of Rating Data

High rating is represented within GCM through multiple layers below a specific cluster. Figure 4. shows that the highest score (5 layers) goes to Teacher Empowerment, Personalization and Feedback & Performance. One layer less get the clusters Students Empowerment and Research & Learning Design. The clusters Management & Economics and Risks, score low (one layer only). Figure 6: Pattern match Importance vs. Feasibility With respect to our expectations for possible differences between novices experts we compared the novice group (n=20) with a group of expert users (advanced users n=17 + expert n=2). We tested both groups again on Importance (see Figure 7) and Feasibility (see Figure 8) against each other. As the figures show the novice participants highly agree with the experts on both the importance and feasibility of ideas in different clusters. The correlation between the two groups is very high for both values (r=0.99). Figure 4: Rating map on importance The most valued clusters on feasibility are Teaching Empowerment and Research & Learning Design, each with 5 layers. Four layers get the clusters Feedback & Performance and Personalization. Students Empowerment gets 3 layers, Risks and Management & Economics 1 (See Figure 5).

Figure 7: Pattern Match Novice vs. Expert on Importance

4. Discussion and Conclusions

The GCM study at the LASI event Amsterdam showed some interesting results for the impact of LA on the Dutch Education system. The studys outcomes confirm our expectation about a possible overlap between the issues identified in this study and themes discussed in the dedicated literature on Learning Analytics Figure 5: Rating map on feasibility The ladder graph in Figure 6, called also a pattern match, compares the clusters on their importance and feasibility ratings. There is a very strong relationship between the two set of data (r=0.94). Small differences can be detected in Personalization, Feedback & Performance, and Students Empowerment, which score relatively higher on importance than on feasibility, and Research & Learning Design, which in contrast score higher on feasibility. In contrast to what we expected, the GCM study demonstrated a very high agreement between the novice and the expert participants on how important and feasible are changes that LA brings to the Dutch education. It seems to be an indication for an already existing consensus between different stakeholders involved in the LA community in the Netherlands. The community highly agrees on required topics that are of importance to influence the educational system with LA.

2.

Ferguson, R. The State of Learning Analytics in 2012: A Review and Future Challenges a review and future challenges. The Open University UK, 2012. Baker, R. and Yacef, K. The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining 1, 1 (2009), 317. Romero, C. and Ventura, S. Educational Data Mining: A Review of the State-of-the-Art. IEEE Transactions on Systems 40, 6 (2010), 119. Calinski, T. and Harabasz, J. A Dendrite Method for Cluster Analysis. Communications in Statistics - Simulation and Computation 3, 1 (1974), 127. Drachsler, H. and Greller, W. The Pulse of Learning Analytics - Understandings and Expectations from the Stakeholders. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, ACM Press New York, NY, USA (2012), 120129. Govaerts, S., Verbert, K., Duval, E., and Pardo, A. The student activity meter for awareness and selfreflection. CHI, ACM Press (2012), 869884. McLaren, B.M., Loll, F., Pinkwart, N., and Scheuer, O. Computer-supported argumentation: A review of the state of the art. International Journal of Computer-Supported Collaborative Learning 5, 2010, 43102. Pardo, A. and Kloos, C.D. Stepping out of the box. Proceedings of the 1st International Conference on Learning Analytics and Knowledge - LAK 11, ACM Press (2011), 163. Manouselis, N., Drachsler, H., Verbert, K., Duval, E. Recommender Systems for Learning. Springer, (2012), p90. Verbert, K., Manouselis, N., Drachsler, H., and Duval, E. Dataset-Driven Research to Support Learning and Knowledge Analytics. Educational Technology & Society 15, 3 (2012), 133148. Trochim, W.M.K. An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning 12, 1 (1989), 116. Rosas, S.R. and Kane, M. Quality and rigor of the concept mapping methodology: A pooled study analysis. Evaluation and Program Planning 35, 2 (2012), 236245.

3.

4.

Figure 8: Pattern Match Novice vs. Experts on Feasibility Furthermore, they also agree on the feasibility of specific LA factors to impact Dutch education in a short timeframe. The results of the study indicated that novices and experts see LA as beneficial for empowering students and teachers. Both groups believe that LA would improve feedback, assessment and personalization for students and teachers. The study has identified also some possible risks of LA but neither the novices nor the experts see those as serious treats for the implementation of LA in the Dutch education system. Both groups value ideas related to students and teachers empowerment, personalization, and research & learning design more than management and economic aspects of LA. Personalization and self-regulated learning are the most salient benefits for students. Personalization has been identified as a cluster that scored the highest on both importance and feasibility. LA stimulates new forms of feedback and personalization, which is essential for improving both teaching (clusters teachers empowerment and research & learning design) and learning (clusters students empowerment and personalization). The study has some limitations with respect to the generalization of the findings. First of all, the findings are valid only for the Netherlands. It would be interesting to compare these results with those of other countries. We therefore, already team-up with partners in Australia that are interested to run a comparable GCM study. A meta-study including different GCM studies would be a unique research initiative that would provide a strong evidence for the rollout of LA methods and technologies in different educational systems. Second, although the optimal number of participants for the sorting have been reached, as indicated in a meta-analytical study on GCM [15], we could have tried to involve more experts for the rating phase.

2.

3.

4.

5.

6.

7.

8.

5. ACKNOWLEDGMENTS
We gratefully thank the members of the SURF SIG Learning Analytics for their support for the present study. Furthermore, we want to thank the participants of the GCM study for the time invested in the LASI event and the online meetings afterwards.

9.

6. REFERENCES
1. Greller, W. and Drachsler, H. Translating learning into numbers: A generic framework for learning analytics. International Journal of Educational Technology & Society. 15, 3 (2012), 4257.

10.