MASELTOV Deliverable Report 7.2: Feedback and Progress Indicators

Open Research Online The Open University’s repository of research publications and other research outputs MASELTOV Deliverable Report 7.2: Feedback and Progress Indicators Other How to cite: Gaved, Mark; Jones, Ann; Kukulska-Hulme, Agnes; Scanlon, Eileen; Jones, Jan and Lameras, Petros (2013). MASELTOV Deliverable Report 7.2: Feedback and Progress Indicators. MASELTOV Consortium, Graz, Austria. For guidance on citations see FAQs. c 2013 MASELTOV Version: Version of Record Link(s) to article on publisher’s website: http://www.maseltov.eu/publications/ Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyright owners. For more information on Open Research Online’s data policy on reuse of materials please consult the policies page. oro.open.ac.uk DELIVERABLE REPORT D7.2 “Feedback and Progress Indicators” collaborative project MASELTOV Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Grant Agreement No. 288587 / ICT for Inclusion project co-funded by the European Commission Information Society and Media Directorate-General Information and Communication Technologies Seventh Framework Programme (2007-2013) Due date of deliverable: Actual submission date: Start date of project: Duration: June 30, 2013 (month 18) June 30, 2013 (month 18) Jan 1, 2012 36 months Work package Task Lead contractor for this deliverable Editor Authors WP 7 – Persuasive Learning Services Task7.2 Feedback and Progress Indicators OU Quality reviewer Mark Gaved (OU), Ann Jones (OU), Agnes Kukulska-Hulme (OU), Eileen Scanlon (OU), Jan Jones (OU), Petros Lameras (COV) Lucas Paletta (JR) Project co-funded by the European Commission within the Seventh Framework Programme (2007–2013) Dissemination Level PU PP RE CO Public X Restricted to other programme participants (including the Commission Services) Restricted to a group specified by the consortium (including the Commission Services) Confidential, only for members of the consortium (including the Commission Services) MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 1 of 50 Contact for feedback on this report to the project coordinator: [email protected] Lucas Paletta DIGITAL – Institute for Information and Communication Technology JOANNEUM RESEARCH Forschungsgesellschaft mbH Steyrergasse 17 8010 Graz, Austria Contact for feedback on this report to the editor: [email protected] Mark Gaved Institute of Educational Technology (IET), The Open University, Walton Hall, Milton Keynes, MK7 6AA, United Kingdom MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 2 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services © MASELTOV - for details see MASELTOV Consortium Agreement partner Organisation 01 JOANNEUM RESEARCH FORSCHUNGSGESELLSCHAFT MBH AT 02 CURE – CENTER FOR USABILITY RESEARCH AND ENGINEERING AT 03 RESEARCH AND EDUCATION LABORATORY IN INFORMATION TECHNOLOGIES EL 04 UNDACIO PER A LA UNIVERSITAT OBERTA DE CATALUNYA ES 05 THE OPEN UNIVERSITY UK 06 COVENTRY UNIVERSITY UK 07 CESKE VYSOKE UCENI TECHNICKE V PRAZE CZ 08 FH JOANNEUM GESELLSCHAFT M.B.H. AT 09 TELECOM ITALIA S.p.A IT 10 FLUIDTIME DATA SERVICES GMBH AT 11 BUSUU ONLINE S.L ES 12 FUNDACION DESARROLLO SOSTENIDO ES 13 VEREIN DANAIDA AT 14 THE MIGRANTS' RESOURCE CENTRE UK MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 3 of 50 ctry Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services CONTENT 1. Executive summary.................................................................................................................. 6 2. Introduction ............................................................................................................................ 7 3. Definitions: key learning terminology....................................................................................... 7 4. Background to the deliverable: literature review ....................................................................... 9 4.1 Importance of feedback and progress indicators to a learner’s journey .......................... 9 4.2 Who gets the feedback? .............................................................................................. 13 4.3 Categories of feedback and progress indicators ........................................................... 13 4.3.1 Cognitive ............................................................................................................. 13 4.3.2 Affective .............................................................................................................. 14 4.3.3 Social ................................................................................................................... 15 4.3.4 Motivational ........................................................................................................ 16 4.4 FPIs for incidental learning ......................................................................................... 16 4.5 Learner analytics and FPI’s ......................................................................................... 17 4.5.1 Learner analytics .................................................................................................. 18 4.5.2 Social Learning analytics ...................................................................................... 18 5. Review of existing approaches ............................................................................................... 19 5.1 Introduction ............................................................................................................... 19 5.2 Types of feedback and progress indicators .................................................................. 19 5.2.1 Learning guides ................................................................................................... 20 5.2.2 Progress bars ....................................................................................................... 21 5.2.3 Levels of difficulty ............................................................................................... 22 5.2.4 Dashboards ......................................................................................................... 22 5.2.5 Assessment tools ................................................................................................. 24 5.2.6 Achievements ...................................................................................................... 26 5.2.7 Reputation and ratings ........................................................................................ 29 5.2.8 Virtual currencies................................................................................................. 30 5.2.9 In-game hints ...................................................................................................... 31 5.2.10 Prompts............................................................................................................... 32 5.3 Review of FPI’s used in language learning.................................................................. 32 6. Challenges.............................................................................................................................. 33 6.1 Assessment of skill levels ............................................................................................ 33 6.1.1 Objective test of skills.......................................................................................... 33 6.1.2 Self assessment .................................................................................................... 34 MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 4 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 6.1.3 Recommendation ................................................................................................ 35 6.2 Ethical issues .............................................................................................................. 35 6.3 Information prioritisation ........................................................................................... 35 6.4 FPI’s for mobile learning ............................................................................................ 36 7. Feedback and progress indicators: our recommendations ...................................................... 36 8. Summary/ conclusions .......................................................................................................... 37 9. References ............................................................................................................................. 39 Appendix A: Review of two web based learning environments: busuu.com and duolingo ......... 43 MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 5 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 1. EXECUTIVE SUMMARY This document explores the range of feedback and progress indicators (FPIs) that can be used to support incidental, mobile learning for the target MASELTOV audience, recent immigrants to the EU. We propose that feedback, and progress indicators (we differentiate between the two) should play an instrumental role in helping learners reflect upon individual, often isolated learning episodes mediated by single MASELTOV services, and enable them to reconceive them as constituting elements of a coherent, larger learning journey. The goal of feedback and progress indicators is to support the motivation for learning and from this the social inclusion of recent immigrants. Our underpinning assumption is that the MASELTOV software designers’ goal should be to encourage not just resolution of immediate challenges (e.g. finding a doctor, translating a sign) but a user’s reflection on their continuing progress towards integration into the host country, including improving their language skills. We define feedback as responses to a learner’s performance against criteria of quality and as a means of directing and encouraging the learner; and progress indicators as responses indicating the current position of a learner within a larger activity or journey (often related to time). Drawing partly from the worlds of web-based language learning and video games, we identify which feedback and progress indicators may best support incidental mobile learning, and the major challenges faced. For some MASELTOV services, feedback and progress indicators for large scale learning journeys are less apparent (e.g. TextLens, the MASELTOV tool that enables a user to take a photo of a sign and convert the image into text, potentially for future viewing or translation), while some services are explicitly educational (e.g. language lessons). However we see all of these as potentially part of an ecology of services that can support social inclusion, so all tools should include FPIs that encourage broader learning goals. In this document we draw on the Common European Framework of Reference for Languages as appropriate, and also reflect on learner perspectives (derived from WP2 and WP9 findings) to identify suitable FPIs, as well as being informed by academic literature. Furthermore, we recommend FPIs that would be suitable for the MASELTOV tools and services. The remainder of the deliverable handles the four identified key areas where mobile incidental learning particularly requires FPIs: 1. encouraging reflection 2. future goal setting 3. planning 4. social learning It should be noted that this document is a high level review, identifying significant literature and key examples of FPIs in practice. This document offers recommendations therefore in general terms. Decisions about specific FPIs to be implemented will be made in coordination with technical partners to identify which MASELTOV services and tools will support which specific feedback and progress indicators, and how they will be implemented within the system. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 6 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 2. INTRODUCTION This deliverable describes our review of Feedback and Progress Indicators, drawing on academic literature and examples of current practice, and we then recommend best practice to software and educational developers. The MASELTOV Description of Work outlines this Task as follows: “Cognitive, affective, social and motivational outcome measures will be considered, with due regard to the Common European Framework of Reference for Languages: Learning, Teaching and Assessment (CEFR) and learner perspectives on what matters most. Appropriate measures will take account of feedback information from the user in order to develop ‘progress indicators’ that will help determine the status of the user’s learning process.” The authors of this deliverable recognise that this prioritises outcome measures to the users, though we identify that this can be supported by providing feedback and progress indicators to other actors (mentors, software developers and maintainers, system administrators). We note that learning is emphasised, and languages are brought to the fore. We recognise that in MASELTOV, the tools and services fulfil a wider range of support than just offering language learning and that a long term learning trajectory may need to be encouraged for users to enable the goal of social inclusion. In this document, we offer definitions for key terms and review the academic literature (Section 3 and 4), identify challenges (Section 5), and review existing examples of practice (Section 6). We then offer our recommendations for FPIs (Section 7). Finally, we summarise our work and offer conclusions to inform other work packages (Section 8). 3. DEFINITIONS: KEY LEARNING TERMINOLOGY For the MASELTOV project, and its software tools and services, we define feedback as responses to a learner’s performance against criteria of quality and as a means of directing and encouraging the learner. Nicol and Macfarlane-Dick (2006) emphasise both encouragement and the learner focus, and also identify that timely feedback can enable progression in learning. Schön (1983) notes the value of reflection in learning, and we see feedback as a means of triggering this activity by learners. We define progress indicators as responses indicating the current position of a learner within a larger activity or journey (often related to time). While we distinguish these terms in specific circumstances, we will refer to them collectively in this document where appropriate using the acronym FPIs. The literature review (see Section 4) will describe cognitive, affective, social and motivational forms of feedback, and progress indicators. The Description of Work notes that we should pay due regard to the Common European Framework of Reference for Languages: Learning, Teaching and Assessment (CEFR). MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 7 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services This is a framework designed to provide “a comprehensive basis for the elaboration of language syllabuses [and] curriculum guidelines” (Council of Europe, 2001, p. 1). It aims to provide “a practical tool for setting clear standards to be attained at successive stages of learning and for evaluating outcomes in an internationally comparable manner” (English Australia, 2013). It is of particular interest to MASELTOV as it has been considered previously as a tool to enable the linguistic integration of adult migrants into Europe and hence may offer guidance to supporting linguistic integration of the MASELTOV target audience (Little, 2010). For this document, we note that the MASELTOV tools and services encompass a broader range of learning than just language learning, so the CEFR is only appropriate as a guide in some circumstances. However, its general approach (identifying levels of achievement, contextual learning, facilitating occupational mobility) is consistent with the wider MASELTOV learning approach and may inform the support of all tools and services. Within MASELTOV, there is a wide range of tools and services, and these may present feedback and progress indicators in very different ways: feedback from a language learning exercise will be very different from feedback within a serious game, or when using a very task-oriented tool like TextLens (photographing a sign for text translation). FPIs might be offered on a very task-specific level, often indicating instrumental aptitude (e.g. registering whether a button has been correctly pressed to trigger an action such as capturing an image, or indicating that a video game character has successfully collected a coin), however for this document we focus on feedback that helps the user of these various tools and services consider more strategic goals, such as language competency for particular situations or familiarity with cultural norms surrounding particular environments (e.g. accessing healthcare, negotiating local bureaucracy). The constituent tools and services in the overarching MASELTOV app can be used independently, without reference to each other. It is possible, therefore, that a user might just employ one or two of the tools in a very problemfocussed mode, responding to a specific situation in which they find themselves. For example, they may use the TexLens tool to capture a photo of a sign and translate it, if they suspect that it can help them understand the rules indicated. A little while later, they may use the MASELTOV information service to find out some other local information. The user might continue their day and not reflect on these events again. We would like to find opportunities to gather together such apparently isolated events and encourage the user to reconceive them as fragments that can be gathered together and reflected upon as elements of a larger journey towards cultural and social inclusion in their new host country. We believe that feedback and progress indicators could trigger such reflection. With a broad definition of learning in mind, we think of the MASELTOV user as undertaking a “learner’s journey” rather than a “learning journey”. The “journey” is a broader experience which has its widest concerns focussed around moving towards social inclusion in their new host country, rather than specifically focussed on learning in the formal educational sense of the word. The MASELTOV audience may not be focussed primarily on educational goals, and our feedback and progress indicators should reflect that by considering not just feedback in terms of formal educational modes of response, but in a more general sense to support this overall ambition (social inclusion). MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 8 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 4. BACKGROUND TO THE DELIVERABLE: LITERATURE REVIEW In this section we identify the key literature that informs the MASELTOV perspective on feedback and progress indicators, and the choice of suitable learner outcome measures. IMPORTANCE OF FEEDBACK AND PROGRESS INDICATORS TO A LEARNER’S JOURNEY 4.1 Feedback and progress indicators are part of a developing research agenda in which aspects of the formal learning process are re-examined and re-designed for effectiveness in a digital and mobile age, e.g. (Beetham & Sharpe, 2013). Educational research suggests that timely and appropriate feedback and indicators of progress can motivate learners (Nix & Wyllie, 2009); which may increase retention and contribute to the completion of programmes of study. As with our work on the incidental learning framework (see Deliverable D7.1.1 and D7.1.2), we draw on Kolb’s experiential learning model (Kolb, 1984) which is readily observable as integrating and including feedback as part of the experiential learning cycle (see Figure 1). Figure 1: Kolb’s learning cycle: image from (Davies & Lowe, n/d) Gibbs (1988) also explores reflection, noting that experiences need structured reflection and analysis, and that an action plan is help put into practice the learning and new understanding gained. Gibbs identifies seven stages to move from initial response to identifying a plan of action: 1. Description: What happened? No judgements or trying to draw conclusions; just a description. 2. Feelings: What were your reactions and feelings? Again don't move on to analysing these yet. 3. Evaluation: What was good or bad about the experience? Make value judgements. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 9 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 4. Analysis: What sense can you make of the situation? Bring in ideas from outside the experience to help you. What was really going on? Were different people's experiences similar or different in important ways? 5. Conclusions (general): What can be concluded, in a general sense, from these experiences and the analyses you have undertaken? 6. Conclusions (specific): What can be concluded about your own specific, unique, personal; situation? 7. Personal action plans: What are you going to do differently in this type of situation next time? What steps are you going to take on the basis of what you have learnt? Gibbs’ model of reflection (see Figure 2) indicates cognitive, affective and social aspects of the process, and we can see that these could form the basis for feedback prompts to the MASELTOV user to encourage them to both look back on what they have done, as well as forward to plan what they might do next. Figure 2: Gibbs’ model of reflection (1988) Well-presented feedback can “…enable individuals to reflect on their ‘learning self’ and to take responsibility for their own learning, while enabling teachers to assess the learning characteristics of groups and individuals” (Buckingham Shum & Crick, 2012). Feedback can support learner retention (Yorke, 2001). It is also recognized that learners can take little notice of feedback from their teachers and so rather than being mere recipients of performancerelated information, it is proposed that they should be actively involved in seeking, generating and using feedback (Boud & Molloy, 2012). Furthermore, feedback has to be managed sensitively: “…the effects of feedback on performance are highly variable; under some conditions, feedback may improve performance, and under other conditions, feedback may reduce performance (Kluger & DeNisi, 1996)” (Garris 2002). Within the context of incidental learning (unplanned or unintentional learning) which is the mode of learning in which we will support the MASELTOV target audience (see Deliverable D7.1.2 for further details), we see feedback, and progress indicators as fulfilling a number of key roles: MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 10 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services       enabling independent, isolated uses of MASELTOV tools and services to be reconceived as elements of a more coherent, longer term journey towards social and cultural inclusion in their new host country motivating continued learning reducing drop out (improving retention) capturing data on a learner’s activities to enable the software tools to provide recommendations (the user profile and recommendation system) capturing data on a learner’s activities to enable software developers to improve services providing information to mentors to enable better support of learners It should be noted that assessment in education is a vast and contested area; for example there is debate about the relative roles of formative and summative assessment, whether the focus should be on assessment for learning – or learning for assessment, and the role of the learner in evaluating their own learning (see e.g. Crick, Broadfoot and Claxton, (2004) or Swaffield, (2011)). However, there has been consistent evidence for some time that assessment is a big driver in formal learning, in affecting what learners do, how much they do and how they prioritise their time (see, e.g. Rowntree (1987)). Whilst some of this research in the area of formal learning is of limited relevance to informal and incidental learning, research in the areas of formative assessment, feedback and self regulation, with its emphasis on the learner’s role in the process is particularly pertinent. The section below draws on the work of Nicol and MacFarlarne-Dick (2006) which is particularly relevant to MASELTOV for four reasons: Firstly their work on feedback and formative assessment is framed within the concept of self regulated learning. This refers to the learner’s role in regulating their learning activities, for example in setting their own goals and monitoring their progress. They comment that: “In practice, self-regulation is manifested in the active monitoring and regulation of a number of different learning processes, e.g. the setting of, and orientation towards, learning goals; the strategies used to achieve goals; the management of resources; the effort exerted; reactions to external feedback; the products produced” (Nicol & Macfarlane-Dick, 2006, p. 199) Secondly, theoretically, Nicol and MacFarlarne-Dick’s socio-constructivist conceptualisation is consistent with the socio-constructive view of the learner taken in MASELTOV: “The student also actively constructs his or her own understanding of feedback messages derived from external sources …. This is consistent with the literature on student-centred and social constructivist conceptions of learning (Lea, Stephenson, & Troy, 2003; Palinscar, 1998)” (Nicol & Macfarlane-Dick, 2006, p. 201). Thirdly, in considering feedback they go beyond the cognitive realm (which tends to often be the focus of other research on feedback) arguing the need for: “repositioning formative assessment and feedback within a wider framework that encompasses self-regulation of motivation and behaviour as well as of cognition” . This aligns well with MASELTOV’s aim to consider cognitive, social, affective and motivational feedback, although we do not believe these sit neatly in separate categories and discuss this in Section 4.3, following. Fourthly they consider who provides the feedback in relation to how effective it is in MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 11 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services supporting the learning processes. They argue that feedback should not only be provided by the teacher but also by peers and by learners themselves, noting that: “Self-regulated learners also actively interpret external feedback, for example, from teachers and other students, in relation to their internal goals”(Nicol & Macfarlane-Dick, 2006, p. 200). We find this argument that feedback is provided by learners and their peers as well as by the teacher or mentor persuasive, as we believe that one role for feedback and progress indicators in MASELTOV is to support self-regulated, reflective learning (Gaved et al., 2013). The same argument for including learners’ own feedback is made by Buckingham-Shum (2012), described in more detail in Section 4.5 following. So, whilst MASELTOV learners do not have teachers, we expect them to be actively interpreting the feedback they get from the MASELTOV system, their peers, and more experienced community members. Nicol and Macfarlane-Dick note the considerable evidence that effective feedback works (i.e. it leads to learning benefits) and the role of both self and peer feedback in this process. They outline seven principles of good feedback practice, drawn from the literature. As the literature relates to formal learning, we have revised these principles to be applicable to MASELTOV: Original Good feedback practice: 1. helps clarify what good performance is (goals, criteria, expected standards); 2. facilitates the development of self-assessment (reflection) in learning; 3. delivers high quality information to students about their learning; 4. encourages teacher and peer dialogue around learning; 5. encourages positive motivational beliefs and self-esteem; 6. provides opportunities to close the gap between current and desired performance; 7. provides information to teachers that can be used to help shape how teaching is provided in the future. Feedback practice as used in MASELTOV: 1. Encourages goal setting and planning and measurement against peers in a community of informal learners 2. facilitates the development of self-assessment (reflection) in learning; i.e. helps to extend “incidents” into a learning journey 3. delivers high quality information to students about their learning; 4. encourages peer dialogue around learning; 5. encourages positive motivational beliefs and self-esteem: this may also involve peer feedback 6. provides opportunities to close the gap between current and desired performance; 7. provides information to the systems about the learners. As the MASELTOV service’s overarching goal is to enable the successful social inclusion of its target audience successfully into their new host country, we consider ‘scaffolding’ and ‘fading’ to be important concepts when considering feedback and progress indicators. These refer to the idea of providing the learner with additional support from other people, tools or resources temporarily to enable them to progress further than they could with their own resources (through “scaffolding”), drawing on Vygotsky’s (1978) notion of the Zone of Proximal Development (ZPD) so that the learner is able to attain outcomes, but aiming to gradually reduce this support (‘fading’) as the learner becomes more competent (Luckin, 2010). This concept is also found in gaming, where highly detailed and regular support is built MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 12 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services into initial game play at easier levels and then becomes sparser and less frequent as the game player achieves mastery and moves to more challenging levels. 4.2 WHO GETS THE FEEDBACK? Discussion of feedback and progress indicators is often centred on feedback to learners, the intended audience of a learning system. In MASELTOV, this is our primary focus when considering feedback and progress indicators. The provision of FPIs to the addressed audience aims to enable reflection and continued learning: as Crick et al. note, learning is a continuous process (Crick et al., 2004). However, we also recognise that feedback to other parties within the learning system can also enable better outcomes for the learner. Mentors (such as more experienced peers, other learners and volunteers), system developers and administrators benefit from understanding the nature and frequency of user engagement and hence can offer better support. More experienced learners “may share responsibility for welcoming newcomers, engaging all members and encouraging meaningful participation” (Ferguson & Buckingham Shum, 2012, p. 29). Peers at the same level or expertise can also enhance the learning process; from a social constructive perspective, learners can support each other’s learning through dialogue, asking questions and providing explanations (King, 1999). Learners’ activities captured by the system can enable automated processes to identify learners’ capacities, or “dispositions” (Buckingham Shum & Crick, 2012), and hence recommend resources or courses of action that will stretch the learner (Ferguson & Buckingham Shum, 2012). This automated capture of learner activity, coupled with the interactions instigated by the learners themselves, can provided a rich feedback mechanism to enable the users of the MASELTOV system to reflect on their activities and progress towards long term goals. Learner activity data can also be used by systems developers and administrators to identify if there are problems with the systems themselves (e.g. poor user interface design which means that users are confused about how to use the tools, or broken functionalities) and allow them to use this information to improve the tools and services to make them more effective. This is a common approach in the mobile phone app and gaming world, where close analysis of user data is used to inform rapid development iterations of services. 4.3 CATEGORIES OF FEEDBACK AND PROGRESS INDICATORS The MASELTOV Description of Work indicates four types of feedback and progress indicator and here we describe each, referring to the literature. We note that the division is not often clear, and that in reality feedback and progress indicators may encompass more than one category. 4.3.1 COGNITIVE Cognitive feedback focuses on the learner’s cognitive actions and output. In a real world language context, where a learner is communicating in the target language, they will get feedback from the person they are speaking to about whether or not they have been understood. In MASELTOV when users are learning their target language, feedback provided by the system would focus on the accuracy of the learner’s input: whether it is about the form of a verb or vocabulary. Language learning websites, such as busuu and duolingo, use a range MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 13 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services of these for assessing students’ competencies. Many computer-based assessment systems, especially when they are making use of multiple choice questions provide essentially cognitive feedback; see for example, Ross, Jordan and Butcher (2006). We describe these in more detail in Section 5.2.5 (“Assessment Tools”). Examples include:  Measuring retention of knowledge (e.g. vocabulary acquisition).  Multiple choice items  Tests (formative assessment) to “ identify knowledge gaps” (Kraiger, Ford, & Salas, 1993, p. 314)  Knowledge organisation  Cognitive strategies  Comprehension  Production  Indicators of lessons completed  Marks gained for exercises  Time to complete a task  Number of errors in a completed task Cognitive feedback in games, another important element of the MASELTOV services, may be supported through reflection, debriefing and articulation (Obikwelu & Read, 2012). During reflection the players may compare their own game progress with those of a peer, of an expert (e.g. a teacher) or with their own cognitive model of expertise. Debriefing mechanisms may be a key for connecting game experiences and feedback. The debrief may help players/learners to receive comments and insights about the game activity and link it with real-world situations. Furthermore, an analysis of misinterpretations and corrective actions could be realised encompassing strategies for connecting the gaming activity with other activities performed in face-to-face settings. The focus here is on progress through efforts which are energised by challenge (Obikwelu & Read, 2012). Articulation facilitates peer-interaction as means of cognitive growth. Articulation may be a useful activity especially in Massively Multiplayer Online Games (MMOGs) where social negotiation of ideas is fundamental for creating an identity, personal knowledge development and collaborative knowledge creation. Cornillie et al., (2012) argue that by using mini game remedial exercises, input enhancements or corrective feedback for language learning may improve learning outcomes. Feedback in this domain “enables individuals to understand and improve their judgments, improve their expertise in the judgment task, and reduce commitment to incorrect judgment strategies (Hogarth, 1981)”(Balzer, Doherty, & O'Connor, 1989, p. 412). Also see Section 5 “Types of Feedback and Progress Indicators”) for more detailed discussion. 4.3.2 AFFECTIVE Hurd writes about affect in the very relevant context of independent language learning: “Affect is about emotions and feelings, moods and attitudes, anxiety, tolerance of ambiguity and motivation”. For some it is also connected with dispositions and preferences (Oatley & Jenkins, 1996). It is generally accepted that “the affective domain encompasses a wide range of elements which reflect the human side of being, and play a part in conditioning MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 14 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services behaviour and influencing learning”(Hurd, 2008, p. 218).Like other researchers, Hurd also notes that affective factors such as a learner’s mood, confidence and anxiety have a significant effect on language learning. Note that here motivation is included as part of affect and that a learner’s beliefs about their learning capabilities are very important: “Efficacy expectations determine how much effort people will expend and how long they will persist in the face of obstacles and aversive experiences. The stronger the perceived self-efficacy, the more active the efforts” (Bandura, 1977, p. 194). Examples include:        Attitudes, motivations, goals Measurement of feelings: what the person likes /dislikes (which could be by selfreflection) o Direction of emotion (positive/ negative) scale o How deeply an emotion is felt Pre- and post- measures Motivational change Motivational disposition Confidence (a key affective concept) Anxiety Confidence, lack of anxiety and tolerance of short term failings in the context of long term success are particularly important for language learning. So an important consideration is how do we encourage confidence? It is likely that self-reporting of emotional state, perhaps in a very simple way will be important here, and feedback from the learner themselves about whether the learning is a positive, pleasant experience Mentors and peers can also play a role here, for example, by reminding the learner that language learning is a bumpy journey, with low points as well as high points (Pritchard-Newcombe, 2009) and boosting their confidence with positive feedback. Some online learning tools have adopted emotion indicators, e.g. happy, sad (Ferguson & Buckingham Shum, 2012) which could well be appropriate. Such emoticons are also often used in gaming. 4.3.3 SOCIAL The social category refers to interacting with others who may be peers, mentors or may be friends, members of the learner’s social network or people encountered in daily activities. We noted that these categories overlap significantly and of course mentors and peers have a significant role in affecting the learner’s level of confidence. Feedback from others may be cognitive or affective, or motivational. Examples include:  Interaction with others (peers, mentors)  Exchange of knowledge (learn about each other’s languages, e.g. via busuu.com language learning website)  Capturing the quality of interaction with others? (e.g. simple quiz/ report (multiple choice) on interactions, e.g. Please tick which of following applies: I feel more confident after talking with X; talking to X helped me towards my goal; etc.) MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 15 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services    Technical use of statistics Evaluative user rating of the value of the interaction Social tools or feedback such as Facebook’s “likes” ratings of other users, friending (Ferguson and Shum, 2012) and/or tagging. The sense of community, presence and online cooperation are also relevant: “participation is a way of belonging, where belonging is ‘not only a crucial condition for learning, but a constituent element of its content’” (Lave & Wenger, 1991, p. 35) 4.3.4 MOTIVATIONAL “A common classification of motivation is into intrinsic and extrinsic types (Reynolds, 2002); Intrinsic motivation being commonly described as self-motivation or internal drive to complete a task, whereas external motivation comes from interactions with individuals such as tutors, as well as the constructs and structures in place to scaffold the learning experience.” (Dunwell, Jarvis, & de Freitas, 2011, p. 6) Over the last twenty years, the most prominent perspective on research on motivation in learning has been the view that motivation is socially influenced (Zimmerman & Schunk, 2007). Some studies have investigated motivation and/or engagement in game-based contexts, notably Schwabe and Goth (2005), Huizenga et al. (2009) and Iacovides et al. (2012). Hurd, writing in the context of online distance learning, notes that “Motivation is the factor most frequently cited as critical to successful learning by distance learners themselves” (2008, p. 227). Often engagement and motivation are seen as similar or overlapping. Iacovides et al. (2012), considering the definition from a game based context, suggests that whilst motivation gets you started, engagement keeps you going. Thus, in MASELTOV it will be important to consider both of these aspects, and the FPIs considered across Section 4.3 should also contribute positively to learners’ motivation and engagement. Examples of motivational indicators and strategies to promote engagement:  Usage statistics give feedback on engagement in terms of persistence  Feedback from other people  Achievements: the award of certificates which can be publicly displayed (for detailed discussion see Section 5.2.6) 4.4 FPIS FOR INCIDENTAL LEARNING In this section we indicate the specific characteristics of FPIs for incidental learning. To determine the particular characteristics of this model of learning, we can identify the types of feedback available in comparison to other modes of learning. For example, when considering on how somebody can learn a language we might consider:  a formal, classroom based learning approach (traditional formal)  a more informal yet planned approach, studying via an online language learning course (planned informal)  the incidental approach of learning through everyday activities (incidental/unplanned) MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 16 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Table 1, originally created by Gaved et al. (2013) as part of the MASELTOV explorations of feedback and progress indicators suitable for incidental learning, describes the different forms of feedback typically available in these modes of learning. Type (examples) Cognitive (Quality of learning, results recognition) Traditional Formal Planned Informal Externally set curriculum, formative assessment (e.g. weekly tests), summative assessment (e.g. accredited certification) Planned goals, navigation maps, structured levels of difficulty, peer ratings of exercises completed, self-assessment of progress, summative assessment (unaccredited certification) Mentor feedback, peer encouragement and recognition, personal assessment of learning (e.g. satisfaction survey) Structured feedback from peers, group forum Affective (Praise, emotional reflection) Teacher feedback, personal assessment of learning (e.g. satisfaction survey) peer recognition Social (Peer support) Reading group, study buddies Incidental/ unplanned Successful resolution of incident, reflection on actions Personal reflection, Instigating discussion of achievement with peers Ad hoc / on request feedback from peers Table 1: Examples of types of progress and feedback indicators for different modes of learning (derived from Gaved et al., 2013) This comparison shows the scarcity of likely feedback resulting from personal, incidental learning. We can see a lack of goal planning, reflection on improving performance for specific activities, and structured feedback from peers. We can see that the MASELTOV system might consider the following areas more closely:  encouraging the learner to look forward and plan how to reconceptualise their immediate problem solving as part of a larger journey towards social inclusion, linguistic competence and cultural familiarity  reflecting upon what has been learnt in a situation and how the response might be improved in future similar scenarios  learning how to engage with and elicit interaction from peers We now turn to consider how feedback and progress indicators may be represented to the learners, and consider learner analytics. 4.5 LEARNER ANALYTICS AND FPI’S This section describes how feedback and progress indicators can be represented to learners, mentors, educators and system administrators. We refer to the emerging fields of ‘learner analytics’ and ‘social analytics’. These have grown from an appropriation of the business world’s ‘data analytics’ approach of gathering and analysing data flows for commercial MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 17 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services advantage. ‘Learner analytics’ explores how data collected on learners’ educational activities can be used to support learners’ own development, taking a learner-centred approach, and also how the collection of this data might help improve the learning environments (e.g. software tools) in which the learning takes place (Ferguson & Buckingham Shum, 2012, p. 23). They note that social learning analytics extends this approach to consider “how learners build knowledge together in their cultural and social settings”. This approach is very suited to MASELTOV’s focus on supporting incidental learning, occurring informal during everyday activities often as a social or collaborative activity, and we will consider how it might be applied to support our target audience’s broader learning goals. 4.5.1 LEARNER ANALYTICS Learner analytics “is about collecting traces that learners leave behind and using those traces to improve learning” (Duval, 2012). While using MASELTOV software tools, users’ activities can be captured by the MASELTOV system, for example registering when a tool was activated, how often each tool is used, what messages are sent and received and answers given to language quizzes. There are many ‘traces’ left by users when engaging with learning mediated by technology enhanced learning tools. Recently, learner analytics has begun to consider context analytics, identifying not only what but also when and where interactions occur (Aljohani & Davis, 2012; Ferguson & Buckingham Shum, 2012) which is particularly relevant to the MASELTOV system with its intention of providing services through mobile phones. This data can be used by developers of the software tools to enable them to improve their services, but it can also be used by learners and educators: “for the educator[s] to better evaluate the learning process, as well as for the learners to help them in their learning endeavour” (Zaïane & Luo, 2001, p. 60). Data collected on user activity can be presented back to each individual learner as feedback to enable them to reflect on what they have done in the past, how they might improve, and what they might do in the future. Learner analytics assumes an underlying engine that can collect and make sense of learner activity; in the MASELTOV project this role could be undertaken by the Work Package 5 User Profile and Recommendation services. The data then needs to be presented back to the individual learner as feedback in a form they can interact with and act upon for future decisions, and often takes the form of a dashboard, one of the potential feedback and progress indicator representations described in Section 5, following. 4.5.2 SOCIAL LEARNING ANALYTICS Social learning earning analytics, derives from the social-constructivist view that learning occurs not only as an individual exercise but is often underpinned and facilitated by social interactions, an approach that draws from educational theorists such as Dewey (1938) and Vygotsky (1978). Knowledge is frequently created and made sense of collaboratively, and often through participation in a community (Wenger, 1998). This ‘socialised’ approach to learning analytics, focussing “on elements of learning that are relevant when learning in a participatory online culture” (Ferguson, 2012, p. 307) aims to capture not only an individual learner’s activities but how they interact within a community. For example, a system might capture how often a learner posts to a forum, and who they communicate with, in order to identify isolated learners within a group who may benefit from additional support or potential MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 18 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services communities that learners might join with to further their goals. The content of the communications themselves can be explored as these may identify how and where learning or misunderstandings are happening (De Liddo, Buckingham Shum, Quinto, Bachler, & Cannavacciuolo, 2011). Social learning analytics “focuses on how learners build knowledge together in their cultural and social settings” (Ferguson & Buckingham Shum, 2012, p. 23) and is therefore very suited to the MASELTOV system’s intended audience and their circumstances. Like learner analytics, social learning analytics argues for learner-centred design, and aims to present the collected and analysed data for the benefit of the learners themselves, as well as the educators. As such, it emphasises the use presentation of data as feedback that will allow learners to make sense of their actions and enable them to become “better learners” (Zaïane & Luo, 2001, p. 63), and also the collation of this data for educators and system developers in order to improve the learning tools and intervene in the learning process as required. In Section 5, following, we will explore a number of concrete examples of how learner and social analytics may be presented to the learner. A common form for summarising a learner’s individual and social actions is the dashboard, described in Section 5.2.4. 5. REVIEW OF EXISTING APPROACHES 5.1 INTRODUCTION In this section we report on a range of the commonly used feedback and progress indicators encountered in online environments, with an emphasis on those provided in learning and gaming environments. We also introduce an ongoing investigation into the use of feedback and progress indicators in web based learning environments, as an illustration of current examples in practice. 5.2 TYPES OF FEEDBACK AND PROGRESS INDICATORS Feedback and progress indicators can be provided in a wide range of forms. These can serve a number of purposes and support the types of feedback indicated above (cognitive, affective, social, motivational). We will now describe a number of the most common presentations of FPI’s and offer examples of their use in practice. These are:  Learning guides  Progress bars  Levels of difficulty  Dashboards  Assessment tools  Achievements  Reputation and ratings  Virtual currency  In-game hints  Prompts MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 19 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 5.2.1 LEARNING GUIDES Learners can be supported by overviews of the learning tasks open to them when engaging with a tool or service. Novice learners particularly benefit from an organised pathway and the availability of an overview to refer to (Chalmers, 2003). These can be an indication of tasks they need to complete, but can also show tasks (or exercises) completed, and also indicate possible options. Learning guides scaffold the journey through the learning content. These may be presented as a list, or a visual representation such as a road map or navigation guide. Like a road map, they may show where the learner can aim towards, where they have travelled from, what they have achieved, and what options have been made available based on the exercises they have completed. Figure 1 shows an example from “duolingo”, a language learning website. This enables learners to review their progress, and also set goals for the future. These learning guides are also used in video games to indicate possible future actions and options. Figure 2 shows an example from “Civilization 3”, a strategy game, indicating what choices the player will make and what future options this will allow them to pursue. Figure 3: Example of road map or decision tree in web based learning environment. Here, the novice has just started, but can see what pathways can be taken later and what completed activities further open (duolingo.com) MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 20 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Figure 4: Example of a road map in a game: Civilization III technology tree © Firaxis Games Another form of learning guide used in video games is the ‘navigation map’, which enables a game player to traverse a playing environment, showing an overview. Like the learning guides described above, they are used to indicate the player’s location in the current content and nearby opportunities. However, they are used in a much more interactive manner, more frequently referred to at a micro level. The additional role these play is to scaffold a player’s cognitive load, enabling the player to focus on the immediate task in hand (O'Neil, Wainess, & Baker, 2005). Like the learning guides described above, these can guide a learner as they progress and be referred to in order to understand choices. 5.2.2 PROGRESS BARS Progress bars provide a simple, often one dimensional visual acknowledgment of what tasks have been completed by the user of a tool. A typical example would indicate the progression a learner has made though a larger learning activity (“20% completed”, or “5 out of 10 tasks now finished”). Figure 5: busuu.com language learning website: numerical progress bar indicating elements of an exercise completed These take a variety of forms, often quantitative summaries (see Figure 5), but also may show a more abstract representation, or show a graphical representation based on a metaphorical visualisation. For example, in the busuu language learning website the learner is shown an illustration of their “language learning garden” and as they progress, the garden flourishes, and if they do not continue to progress through the lessons, the garden is seen to ‘dry up’ or deteriorate (see Figure 6). Progress bars are intended to act as a motivator, showing successful completion of tasks, and also orientate users to help them understand where they are in a process. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 21 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Figure 6: busuu.com language learning website: visual indicator of progress using garden metaphor 5.2.3 LEVELS OF DIFFICULTY Content and activities provided in software and learning environments can be differentiated in complexity by the use of levels. These provide a framework to distinguish associated content and activities, graded by the difficulty of task or competency required to complete activities. These are a commonly used mode of distinguishing content and acting as a progress indicator in games, where simple tasks provided with high levels of guidance and support are offered to beginners in initial easy level, to be gradually replaced by more demanding and complex tasks with less support as the user progresses. Language learning environments such as busuu.com and duolingo also use a similar approach to differentiate and structure content and activities, drawing on language levels frameworks such as the Common European Framework for Languages (Council of Europe, 2001) to guide the coverage of their learning programmes. This can help create a coherent learning progression and inform teaching and assessment measures (English Australia, 2013). The use of explicit levels of difficulty may act as a cognitive progress indicator, and provide distinct goals to motivate learners. 5.2.4 DASHBOARDS Many services offer a “progress dashboard” which shows users a report of their progress across a number of tasks, often as a summary of all their activities. The information in this dashboard may be presented as a list, or as a graphical representation (e.g. busuu.com’s ‘garden’, see Figure 6) and often contains multiple elements representing different aspects of tasks achieved. The reporting information presented often consists of data collected by the system while interactions have taken place (e.g. indicating scores, achievements, frequency of use) but may also contain user inputted self-assessments, such as their reported emotional state while engaging with the learning materials. This may be presented in an initial ‘home page’ for the software or alternatively held on a personal profile page. Such dashboards enable learners to ‘own’ their user profile and use it as a tool for self-reflection and modifying future goal planning (Aljohani & Davis, 2012; Ferguson, 2012). MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 22 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Figure 7: Personal dashboard: a user’s home page, indicating level of learning, activities completed, badges achieved, virtual currency awarded and personal learning goal. In some environments, personal dashboards can be shared in part or in whole with other users, depending on the access the user gives to others (through configuration of their ‘privacy’ settings). An example is shown in Figure 8. This social aspect may engender communication, collaboration, competition and encouragement from other learners. Figure 8: Shared dashboard: example of a user choosing to reveal some information about themselves to other users of a service, indicating basic details and key achievements: busuu language learning website Dashboards can identify user progress that has been gathered through software systems’ data collection of activities, present user entered data, and also present the results of assessment. We now turn to consider how assessment tools can be used as feedback and progress indicators. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 23 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 5.2.5 ASSESSMENT TOOLS Assessment of a learner’s competence through structured exercises can be used as the basis for cognitive and affective feedback, providing concrete progress indicators to learners. This can encourage the learner and trigger further reflection and goal planning, and hence motivate them to continue learning. Assessment may occur the first time that a tool or service is used to assess the appropriate level of content and support to offer a participant. It can also be activated during use of a tool as ‘formative assessment’ that will “provide feedback on performance to improve and accelerate learning” (Sadler, 1998, p. 77), and at the conclusion of use to identify final levels of competency, and to encourage reflection (‘summative assessment’). Students undertaking formative practice tests have been found to perform better than students who only take a final summative test (Sly, 1999). Self-assessment can be used, where learners draw their own conclusions about their level of competency (Ross et al., 2006), or objective assessment approaches, where learners have to respond to questions and their responses are rated by another (the software, a fellow learner, or a domain expert such as a language teacher, or more experienced user of the particular tool or service). A wide range of media can be used, with tools using audio, text, graphics, and the learner may be offered the opportunity to respond using a range of media, including spoken answers, written answers, interaction with graphic elements, and taking a photo or video. The smart phones intended as the delivery platform for MASELTOV are ideal for these purposes, with a range of built in sensors that can enable multimedia input and responses, drawing on the immediate context and situations the learner may find themselves in (e.g. taking photos or recording audio as evidence of learning). Assessment tools can include peer-assessment with individualised feedback: these are likely to have additional value as “feedback that gives information about what is right and wrong, with hints for further improvement, is more effective than feedback that just tells a student whether their answer is correct or incorrect” (Jordan, 2012, p. 832). The language learning website, busuu.com, for example, has a peer-assessment mechanism where each learner can select up to five other learners to mark their work and offer feedback (see example in Figure 9). However, computer moderated assessment also has its merits, as it can be more costeffective for assessing large numbers of learners, and seen as being free from bias (Jordan, Jordan, & Jordan, 2012). MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 24 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Figure 9: busuu.com peer reviewed assessment: learner has their written exercise corrected by another learner who offers improved suggestions A range of assessment methods that have been explored within technology enhanced learning environments could be incorporated within the MASELTOV tools and services. E-assessment approaches include completion of diagrams, multiple-choice questions (see Figure 10), multiple-response, drag and drop of elements to the correct place (see Figure 11 and 12), clicking on hotspots or those requiring free-text entry of numbers, letters, words and sentences (Jordan, 2012; Jordan et al., 2012). Figure 10: busuu.com language learning multiple choice formative exercise MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 25 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Figure 11: busuu.com language learning exercise requiring student to drag icons into position to create correct phrases in correct sequence. Red circle at bottom of screen indicates one incorrect choice made, with three incorrect choices allowed before exercise is failed Figure 12: busuu.com language learning exercise requiring student to drag icons into position to create correct phrases 5.2.6 ACHIEVEMENTS A traditional form of identifying achievements in formal learning environments is the awarding of certificates. These are presented to the learner to enable them to confirm to their peers, future employers and others the level of learning that they have achieved. This concept has been embraced within the informal learning and video gaming communities with the concepts of “achievements” or “badging”. Like certificates in formal education, these are visible representations of learning achievements and indicators of progress made within an environment that are capable of being displayed publicly. Their goal is to enable a member of a community to communicate their achievements to others, to act as a form of external accreditation of their achievements (awarded to them by another body, such as the provider of the learning service or game), and aim to act as motivators, both to the learner themselves and also within a social sphere (encouraging others to match a similar level of progress). Achievements may help self-esteem by enabling learners to recognise they have made progress and encourage them to continue their engagement with the tool or service. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 26 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services In online tools, these are often presented as icons on screen, sometimes in a shared public space, or in a publicly visible aspect of a user’s profile. Figure 13: Example of video game presentation of achievements Figure 14: Example of informal learning presentation of achievements (‘badging’) Achievements are common in entertainment games to recognise players activities, scaffold learning activities, monitor progress, and provide direct feedback, and are often applied external to the game through a platform such as XBox Live or PlayStation Network, allowing players to readily compare their own achievements to their peers and the wider gaming community (see Figure 13 for examples). Clear overlap can be seen when considering the educational application of an achievement system (Dunwell et al., 2012; Heeter, Lee, Medler, & Magerko, 2011) as a means to provide concrete, gamified learning objectives. MASELTOV will build on previous work within the EU-funded ALICE project (Dunwell et al., 2012) to apply these within the MASELTOV system. An approach to achievements that is gaining interest in the online informal learning world is ‘badging’ – a term taking its lead from the traditional real world informal learning practice of awarding badges for achievements. Badges are being explored by learning environments such as MOOCs (massive online open courses), as an alternative means of encouraging learning and recognising achievement where formal accreditation is not offered (see Figure 14 for examples). Cross and Galley (2012), exploring badging for an Open University online open access course, proposed that achievements may encourage learning in three key ways: rewarding the attainment of a way point on a predetermined learning journey (passing a test, completing an activity); rewarding effort (cumulative or threshold reached, e.g. number of hours committed, exercises completed); and thirdly, for exploring outside the core material, rewarding deviation from the main ‘learning arc’ and hence encouraging and rewarding “exploration, deeper learning, and independence”. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 27 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Cross and Galley identify eleven different roles that ‘badges’ can play within an informal learning course (see Table 2). These roles would appear to align well with MASELTOV’s learning goals, and address an audience that has some similarities with the targets for massive open online courses: informal learners fitting their educational work around everyday tasks, a loose community that we wish to encourage to interact, a group of learners who may engage intermittently and need to be encouraged regularly to continue. As such, this table may offer a framework for considering achievement-based feedback and progress indicators for the MASELTOV system. Role of Badge Benefit of the badge Receiver (learner) Creator (awarder) 1. As a motivator / Greater sense and understanding of A solution to the 'motivation issue' for open 2. To promote engagement/ achievements, skills learnt and courses that have no formal (or at least 3. To prevent withdrawal progress being made. Can set teacher marked qualification-related intermediate milestones and assessment). The drop-out rate for such waypoints in the learning journey courses is much higher than for traditional courses (even up to 90-95% of registrations). 4. As a meaning maker / Badges help show learners what the Badges can help describe what is important 5. Signifier of learning awarder think are most important to and can be constructively aligned with objectives the subject/ competency being learning outcomes. Achievement of a badge studied. It can give greater meaning may be seen as evidence that an outcome to the learning 6. As a low-cost option has also been achieved The learner does not have to pay for Awarder does not need to mark, moderate, assessment, nor potentially even grade or award. This means less to no time study (or pay for) a course to spent on assessment and on maintaining the receive the badge structures that support assessment and award 7. As a low-effort option The learner does not have to enrol Courses are repeatable with less effort from in a course if they have previously the awarder (less to no time required for done something that demonstrates assessment) they meet the badge criteria. 8. As a valuer In a similar way to a qualification Both the existence of a badge and uptake of or certificate, a badge can help the it by learners, can help confer value to learner to value what is being learnt something that is clearly import to the MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 28 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services awarder (otherwise why create the badge) 9. As a symbol of identity / A shorthand to represent Helps ties the learner to the awarder and in 10. As a means of achievement, effort or skills and a so doing deepens the association between association way of associating with, and them and others holding the badge. For simultaneously distinguishing social or political groups for whom the goal oneself from others of a qualification/award is not the primary purpose, badges can provide shared goals (or other foci) around which badge-seekers and badge-achievers can associate. 11. As an empower Enables learner to gain status within Awarder gains status by being seen as an a group by achieving badges awarder and, potentially, this may help deemed of value to the group. challenge and shift the authority/power to recognise achievement/skills. The awarder does not need to have formal qualification granting powers to create badges. May also allow individuals to attempt to determine the identify of a group. 12. As an entrencher Those learners who can gain badges Institutions, individuals or groups with sooner (e.g. who already have the established authority, status, wealth skills) place themselves at a or power may use badges to entrench or competitive advantage over those even extend this. This privileges 'super- who cannot. Rather than empower, players' at the detriment of smal this may simply entrench an existing hierarchy or social/professional structure. Table 2: Roles of badges. Reproduced from Cross and Galley (2012) Achievements are related to the social indicator “reputation”, which we will now turn to discuss. 5.2.7 REPUTATION AND RATINGS Reputation is the socially awarded rating of a user’s activities by peers in the same environment. These are displayed publicly within a social software environment, and act as a social feedback mechanism. They are found in social environments such as Facebook to indicate likes or dislikes and approval of posts, to indicate trustworthiness in marketplace tools like eBay, and as a social feedback mechanism in learning environments (the quality of a learners’ contributions. These can be used as social motivators, and to offer affective MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 29 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services feedback. Within the MASELTOV environment, they might be used to offer encouragement to other learners, and to rate the quality of contributions or the help that volunteers provide. Such ‘ratings’ models can also be extended to artefacts, for example, users might be given the opportunity to rate the quality of information articles (see Figure 15). Furthermore, this can be useful to enable content providers to identify which articles are considered low quality, and so should be worked on, and also the popular and well received articles, which might indicate an interest from the users for more articles in the same area. Figure 15: Wire frame mock-up of a potential MASELTOV information page with feedback tool to enable users to rate the quality of an article 5.2.8 VIRTUAL CURRENCIES Virtual currencies are currencies accrued within a virtual environment. They are often awarded by the system for participants’ achievements, though they may also be transferred between participants. Virtual currencies are used for purchasing rewards within the environment that may be cosmetic, affect performance, or allow access to previously locked content and resources and can increase reported levels of immersion by participants (Nojima, 2007). Virtual currencies can in some cases be displayed publicly to other participants in the same environment, indicating one’s progress or comparative achievement, introducing an element of competition and social interaction (Hamari & Lehdonvirta, 2010). Virtual currencies are used in games, but also in virtual learning environments, such as the busuu.com web based learning environment’s use of “busuu berries” (see Figure 16). Busuu berries can be used in the busuu.com language learning environment to purchase virtual goods, can be displayed to other participants, transferred to friends, and won or lost in competitions with other learners. Figure 16: Virtual currency indicator within the busuu.com language learning website, indicating current value of learner’s reserve of ‘busuu berries’ MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 30 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Virtual currency has the potential to change real-world behaviour (Wang & Mainwaring, 2008), for example, motivating continued participation in a game or language learning service, and in MASELTOV we have the opportunity to explore how virtual currencies might be used across a range of tools and services. One potential avenue of exploration might be to investigate how currency awarded in one service might benefit use of another service by the same participant, or whether earned currency could be exchanged between participants. 5.2.9 IN-GAME HINTS In-game hints are an efficient strategy for providing players with guidance while they are using a tool or service, to help correct misconceptions or mistakes they have made and are likely to encounter again during the completion of a current task. Players may get in-game hints in appropriate moments, such as after losing points, or completing a sub-task. For example the World of Warcraft provides in-game hints after completing a quest or through creating a new character. These messages provide information about the current status of the challenge, and remind the player of what functions they have available or options they can take (so in a game, ths might be the properties and characteristics of the chosen character and the tasks they still have to complete, or what errors they have made). Figure 17: In-Game hint indicating the characteristics of an avatar in the World of WarCraft game By providing this kind of mini-feedback, designers can be encouraged to reflect on what the player or learner needs to know in order to achieve a specific learning goal using the tools or service, and at what times such hints will be appropriate, therefore helping structuring the feedback and support they provide during tool or service use. Providing timely feedback can make use of contextual information aware to the device (e.g. GPS sensor providing location based information, forum software understanding who else is online and may be able to support the learner with a task). In-game hints can be seen as a specific example of ‘prompts’ to which we now turn. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 31 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 5.2.10 PROMPTS A powerful type of feedback that systems can provide to encourage participation, learning and reflection is a prompt to the user. Prompts can notify users that action is required, a change in status, or a request for social interaction. They are widely used in social media, games software, and learning environments. They make take the form of an active prompt, such as a message on screen or notification sent via a different medium like SMS or email. busuu.com, for example, will send users an email if they have not logged in to the language learning website, and they have found 14% of users will log in to the website immediately after receiving an email prompt (personal communication May 2013). However, it is important not to overwhelm users with too many prompts, and more passive modes of prompting that require user interaction might be more successful at engaging users, particularly when considering mobile services: audio alerts on mobile phones in the middle of the night are not popular (Vihavainen & Väänänen-Vainio-Mattila, 2013). Furthermore, there is evidence that delayed feedback can help learning in some circumstances (Schroth, 1992). User configuration settings are important to enable users to decide how they will be prompted. Figure 18 shows the Facebook notification icons that ‘passively’ indicate a number of prompts that the user can choose to act upon. Figure 18: Facebook prompts: friend requests from others, notification of messages, and participation in a thread Question prompts are common practice in traditional learning environments and are now being used in online environments(Ge & Land, 2003), identified as “an effective way to elicit reflection since they provide cognitively complex ways learners think about, feel about, and make connections in experience” (Wu & Looi, 2012, p. 319). By encouraging reflection, question prompts can support learners’ planning and evaluation activities (Ge & Land, 2004). Prompts can be initiated by the system, more advanced learners, or peers. 5.3 REVIEW OF FPI’S USED IN LANGUAGE LEARNING To inform MASELTOV’s exploration of persuasive learning services (WP7), an expert evaluator at the Open University has been undertaking a review of two web based learning environments (busuu.com and duolingo.com). This work focuses on usability, however it has enabled us to identify the range and categories of feedback and progress indicators employed within these environments, and provides the MASELTOV project with examples of feedback and progress indicators in practice. This is an ongoing work, however it has informed the current document, providing examples of both the range of FPIs and their interaction in a large scale, informal learning environment. We therefore include as Appendix A an overview of this work, mapping functionalities across to categories and types of feedback and progress indicators. The research is not complete and hence this table should be considered a work in progress. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 32 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 6. CHALLENGES There will be challenges to overcome if the MASELTOV project is to provide effective feedback and progress indicators to the users of the systems. In this section, we describe a number of the key issues to be addressed and propose responses. 6.1 ASSESSMENT OF SKILL LEVELS To provide targeted learner support, it is necessary to understand the software users’ competency levels. This may be achieved through self-assessment, where the learner estimates their own abilities, or an externally examined objective test, where the learner undertakes a test that is marked by another (e.g. software system, mentor, domain expert or educator). Monitoring of user activity by the software by collecting data on how the users are engaging with the tools can also provide feedback that can be employed to both prompt feedback and guided support, and provide data that can be analysed to improve the tools themselves (described in Section 4.5, Learner Analytics and FPI’s). An alternative and often employed approach is to assume the users of the software are novices, and they are required to start as a novice, at the beginning of the process, supported with help guides and other prompting, which may be ‘scaffolded’ to fade into the background as the user progresses. Short cuts may allow users who are confident of their abilities to progress more rapidly. This approach (assuming no prior knowledge) is the approach taken by language learning websites such as busuu.com and duolingo, where the learner is required to start at absolute beginner level and work through all the elementary content, even if they have some knowledge of the language. The assumption is that learners will progress rapidly through the activities they are already competent at, and quickly move to the level which is appropriate for their learning. This has a disadvantage for advanced learners as they will need to cover potentially large amounts of material they are already capable of completing, and hence be seen as demotivating. However, in the MASELTOV situation where we expect learners to be at lower levels of competency this reviewing process may be seen as a confidence building exercise, revealing their levels of existing competence (and often displaying as achievements such as badges). If assessment is used to direct learners to the appropriate content and activities, it is likely to be employed at the initial stages of learner engagement. However, it may be used during the ongoing use of the software to enable continued monitoring and feedback on user competencies (providing cognitive feedback) and to understand how the learners affective state (anxious, confident, happy, etc.). Assessment can happen at the completion of content (“summative assessment”) to identify final levels of competency in engaging with specific activities, and potentially trigger the awarding of achievements such as badges or certificates. Assessment can be achieved via an objective test of skills, or self-assessment and we now describe these in more detail 6.1.1 OBJECTIVE TEST OF SKILLS An objective test of a learner’s skills is one that is an assessment marked by an external body (e.g. an educator or software system). This will enable the software system and educators who may be supporting the learning process to identify which material or content is appropriate for MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 33 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services the learner. Tests may be devised by the developers of the specific software system, or may draw on an externally created assessment tools and measures. In language learning, a wellknown assessment framework is the Common European Framework for Languages: the “CEFR” (Council of Europe, 2001). The framework is a practical framework for describing “what language learners/users can do at different stages of language acquisition, in a range of situations, contexts and fields of language use” (English Australia, 2013). Language ability is measured at six levels, from beginner (A1) through to high levels of proficiency (C2). Assessment of learner’s ability can be matched against these recognised scales, for example an A1 beginner is said to be able to “understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type” while a C2 highly proficient language student is said to “understand with ease virtually everything heard or read. Can summarise information from different spoken and written sources, reconstructing arguments and accounts in a coherent presentation” (Council of Europe, 2001, p. 24). These are further broken down into key skill categories such as understanding, speaking and writing, and aspects such as range, accuracy, fluency and coherence. An analysis of the CEFR framework suggests that the MASELTOV target audience may have language skills that could be considered “pre-A1”: more in limited their abilities than the competencies expected of an A1 learner. Hence it will be important to consider how to offer activities, support, and appropriate feedback for such an audience. This challenge has been recognised by other researchers, for example Alajärvi and Anttila (2012), exploring immigrants in Finnish vocational education, and Farinati et al. investigating the learning of Italian to enable the cultural integration of immigrants (Farinati, Masseroni, & Vimercati, 2012). An alternative approach to an objective test of ability is to invite the learner to offer their own opinions about their abilities and capacities: self assessment. 6.1.2 SELF ASSESSMENT This approach invites the learner undertake a self-assessment of their skills, abilities and goals as a means of judging the appropriate level of content and activities for their needs. Using this approach, the learner will be given prompts to elicit:  Their self-assessment of ability  What their learning goals are  What learning they hope to achieve As with objective assessment, this is likely to be carried out at the beginning of their engagement with a new tool or service, but could be prompted periodically to encourage reflection and re-assessment of goals and planning, as well as providing updates that could be used by educators/mentors and the system to ensure the correct activities are offered at an appropriate level. This assessment may be aligned with existing measures for indicating competency, so for the MASELTOV project the EurLIFE Quality of Life Indicators presented in the (http://www.eurofound.europa.eu/areas/qualityoflife/eurlife/index.php), MASELTOV Description of Work, Table 17, may be appropriate when considering broad scale social inclusion indicators.Tools for ongoing self-reflection and self-assessment of learning and capacity have been considered in learning analytics, for example the Effective MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 34 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Lifelong Learning Inventory (Buckingham Shum & Crick, 2012; Crick et al., 2004), and these may be useful for developing the basis of such tools within MASELTOV. 6.1.3 RECOMMENDATION The simplest solution for the MASELTOV project is to consider all users of services have no experience of the tools or services, and provide appropriate support for novice users. In this model all users are treated as a novice and need to progress through the use of the service to more advanced usage through the same set of steps. It is therefore important to make feedback on use optional to allow users to choose whether to receive response on their activities, so as not to slow down their usage. For services where it is appropriate to offer differentiated levels of service (e.g. beginner/ expert, or CEFR aligned levels of language materials) it may be appropriate to either ask the user which level of content they require or to set a test that can guide them to the appropriate level of use. For language learning content, we propose that content is aligned with CEFR levels. 6.2 ETHICAL ISSUES The collection of user data to generate feedback and progress indicators raises ethical challenges particularly when considering the MASELTOV target audience. “In the context of education, analytics are likely to include sensitive information about identity, status, background and achievements. Ethical use therefore involves making users aware of the data that is being collected, how it is being used and who has access to it” (Ferguson & Buckingham Shum, 2012, p. 31). To offer detailed, targeted recommendations and feedback will require users to agree to some data being collected, therefore it will be important to foster trust and have mechanisms in place to give users confidence about how it is being safeguarded (Dunwell, 2012). MASELTOV is addressing ethical challenges within Work Package 2. With respect to feedback and progress indicators, MASELTOV will need to be sensitive towards: - what is collected, and why - who has access to the data, and - allowing users to choose what they share The challenge of storing user data will be considered closely in Work Package 5, where it is anticipated that the practical implementation of the feedback systems will be operationalized (through the User Profile and Recommendation systems). We will have to find a good solution for storing the data, and the options made available for MASELTOV service users. 6.3 INFORMATION PRIORITISATION A key pedagogical and user design challenge faced when developing feedback and progress indicators is to consider how best to present such information to ensure it is effective, rather than distracting or detrimental to the overall experience. While feedback and progress indicators are intended to motivate, learners can be “overwhelmed and discouraged by the amount of information presented to them, confused by being presented with too many MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 35 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services visualizations” (Ferguson & Buckingham Shum, 2012). Feedback and progress indicators need to be presented at meaningful points in a learner’s journey, and a decision has to be made as to what represents a significant change and the appropriate time for this to be indicated. Timely feedback allows learning in context, however feedback may not be welcomed during critical incidents, or at inconvenient times (Vihavainen & VäänänenVainio-Mattila, 2013). FPI’S FOR MOBILE LEARNING 6.4 The challenges of learning on mobile devices have been well described elsewhere, by authors such as Kukulska-Hulme (2005). These will need to be considered when designing and implementing feedback and progress indicators for the MASELTOV services, which also bring in the additional challenge of supporting mobile incidental learning. Key challenges to be noted are:     Small screen size, limiting the amount of information that can be provided, and hence affecting the visualisations that can be provided Connectivity with central services may be sporadic, affecting when feedback can be provided Mobility and embedded sensors offer the opportunity for highly contextual feedback: which may however become outdated as the user moves on Learning activities might be shorter in length, and more sporadic, carried out between other daily activities. These challenges must be considered by pedagogical and user interface designers, as well as software developers. 7. FEEDBACK AND PROGRESS INDICATORS: OUR RECOMMENDATIONS In this document we have described cognitive, affective, social and motivational outcome measures suitable for the MASELTOV tools and services. Through an exploration of feedback and progress indicators in current practice, we note the following:  Feedback and progress indicators are required for learners (end users), mentors, and the system.  FPIs should be timely, but not overwhelm the user: “passive” indicators may be more effective than interrupting presentation, and in some cases delayed feedback may also be beneficial  FPIs should be considered to support the aspects of learning indicated as potentially lacking in an incidental learning mode of acquiring knowledge (for more detail on incidental learning, see deliverables D7.1.1 and D7.1.2), and encourage not only specific instrumental task completion but reflection on broader social inclusion goals. We therefore recommend that FPI’s should be integrated into software systems to support: o Goal setting o Planning o Reflection MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 36 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services o Structured social learning  We recommend that, where possible, the user profile (collection of data on the user’s progress) is made as transparent as possible to the user, which means representing the data collected through a user profile including a summary of actions undertaken and goals achieved. This should be presented in a mode suitable for the target MASELTOV audience. Considerations must be given according to their information presentation requirements (e.g. an option to present all feedback in their preferred language, not assuming high levels of computer competency).  User profiles could be shared with other MASELTOV community members, with users having the option to configure the level of information that they share. Work carried out in learning analytics suggests this may be socially beneficial: to gain support, to motivate others and to build self-esteem (Ferguson & Buckingham Shum, 2012) 8. SUMMARY/ CONCLUSIONS In this deliverable we described cognitive, affective, social and motivational outcome measures suitable for the MASELTOV tools and services. We provided an overview of key literature in the field, and explained key categories and types of feedback and progress indicators. We gave examples of these strategies in practice, in informal learning, language learning and gaming, and identified challenges that the MASELTOV developers should consider. This document has provided a level review, identifying significant literature and key examples of FPIs in practice. The document offers recommendations therefore in general terms. Decisions about specific FPIs to be implemented will be made in coordination with technical partners to identify which MASELTOV services and tools will support which specific feedback and progress indicators, and how they will be implemented within the system. Our research has established the importance of learner self-reflection, social and community interaction, and diverse mechanisms for monitoring progress, all of which can help encourage continuity of learning and sustained engagement throughout immigrants’ journeys towards social inclusion. Short-term learning gains and achievements are steps towards social integration, which is a gradual process but one that we believe can be accelerated through well designed learning experiences, in which FPI’s play a significant role. Data analytics approaches offer promising ways to monitor learner activity with a view to shedding light on its effectiveness. Key FPIs that should be considered by MASELTOV partners for inclusion in the MASELTOV tools and services are those that encourage:  Goal setting  Planning  Reflection  Structured social learning MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 37 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Learners will benefit from being able to review their progress periodically, and work carried out in learning analytics has indicated a ‘dashboard’ summarising progress may be a valuable progress indicator. MASELTOV services will contribute to the development of their users by providing learning experiences that should make them more able to self-regulate their learning. It is recognized that informal, incidental learning requires a good deal of self-motivation and self-direction. The structuring provided by FPIs should help develop the user’s confidence and support their transition from a being a learner who is strongly dependent on formal, teacher-led activity to becoming someone who is capable of recognising a wide range of learning opportunities and using available feedback and progress indicators to take full advantage of them. This process is aided by the fortuitous cross-overs between gaming and informal learning, particularly elements of fun and competition, as well as light-touch rewards such as visually attractive badges. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 38 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services 9. REFERENCES Alajärvi, K., & Anttila, S. (2012). Finnish of tourism for immigrants: integrating vocational contents and language studies in web-based learning environment. In Pixel (Ed.), ICT for Language Learning, Florence, Italy 2012: conference proceedings. Padova, Italy: liberiauniversitaria.it. Aljohani, N. R., & Davis, H. C. (2012). Learning analytics in mobile and ubiquitous learning environments. In M. Specht, M. Sharples & J. Multisilta (Eds.), Proceedings of the 11th International Conference on Mobile and Contextual Learning 2012 (mLearn2012) (pp. 70-77). Aachen: CEUR-WS.org. Balzer, W. K., Doherty, M. E., & O'Connor, R. (1989). Effects of cognitive feedback on performance. Psychological Bulletin, 106(3), 410-433. doi: 10.1037/00332909.106.3.410 Bandura, A. (1977). Self-efficacy: toward a unifying theory of behavioral change. Psychological Review, 41(191-215). Beetham, H., & Sharpe, H. (2013). Rethinking pedagogy for a digital age: designing for 21st century learning Abingdon, UK: Routledge. Boud, D., & Molloy, E. (Eds.). (2012). Feedback in higher and professional education: understanding it and doing it well. Abingdon, UK: Routledge. Buckingham Shum, S., & Crick, R. (2012). Learning dispositions and transferable competencies: pedagogy, modelling and learning analytics. In S. Buckingham Shum, D. Gasevic & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics & Knowledge, (29 Apr-2 May), Vancouver, BC (pp. 92-101). New York: ACM Press. Chalmers, P. (2003). The role of cognition theory in human-computer interface. Computers in Human Behavior, 19, 593-607. Cornillie, F., Clarebout, G., & Desmet, P. (2012). The role of feedback in foreign language learning through digital role playing games. Procedia - Social and Behavioral Sciences, 34, 49-53. doi: http://dx.doi.org/10.1016/j.sbspro.2012.02.011 Council of Europe. (2001). Common European Framework of Reference for Languages: learning, teaching, assessment. Cambridge: Cambridge University Press. Crick, R. D., Broadfoot, P., & Claxton, G. (2004). Developing an Effective Lifelong Learning Inventory: the ELLI project. Assessment in Education, 11(3), 247-272. Cross, S., & Galley, R. (2012). MOOC badging and the learning arc Retrieved 2013-06-20, from http://www.olds.ac.uk/blog/moocbadgingandthelearningarc Davies, C., & Lowe, T. (n/d). Kolb Learning Cycle tutorial - static version Retrieved 26/06/2013, from http://www.ldu.leeds.ac.uk/ldu/sddu_multimedia/kolb/static_version.php De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M., & Cannavacciuolo, L. (2011). Discourse-centric learning analytics Proceedings of LAK 2011: 1st International Conference on Learning Analytics & Knowledge, 27 Feb - 01 Mar 2011, Banff, Alberta Dewey, J. (1938). Experience and education. New York: Macmillan. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 39 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Dunwell, I. (2012). Conducting ethical research with a game-based intervention for groups at risk of social exclusion. In M. Herrlich, R. Malaka & M. Masuch (Eds.), Proceedings of the 11th international conference on Entertainment Computing (ICEC'12) (pp. 594599). Berlin: Springer-Verlag. Dunwell, I., Jarvis, S., & de Freitas, S. (2011). Four-dimensional consideration of feedback in serious games. In S. de Freitas & P. Maharg (Eds.), Digital games and learning (pp. 42062). London: Continuum. Dunwell, I., Petridis, P., Hendrix, M., Arnab, S., AL-Smadi, M., & Guetl, C. (2012). Guiding Intuitive Learning in Serious Games: An Achievement-Based Approach to Externalized Feedback and Assessment. Paper presented at the Proceedings of the 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS). Duval, E. (2012). Learning analytics and educational data mining Retrieved 26/06/2013, from http://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educationaldata-mining/ English Australia. (2013). CEFR resources Retrieved 2013-06-19, from http://www.englishaustralia.com.au/cefr-resources.html Farinati, L., Masseroni, M., & Vimercati, M. (2012). Parliamoci Chiaro: an online opportunity to learn Italian for immigrants. In Pixel (Ed.), ICT for Language Learning, Florence, Italy 2012: conference proceedings (pp. 65-68). Padova, Italy: liberiauniversitaria.it. Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304-317. Ferguson, R., & Buckingham Shum, S. (2012). Social Learning Analytics: five approaches. In S. Buckingham Shum, D. Gasevic & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics & Knowledge, (29 Apr-2 May), Vancouver, BC (pp. 23-33). New York: ACM Press. Gaved, M., Kukulska-Hulme, A., Jones, A., Scanlon, E., Dunwell, I., Lameras, P., & Akiki, O. (2013). Creating coherent incidental learning journeys on mobile devices through feedback and progress indicators Paper accepted for the 12th International Conference on Mobile and Contextual Learning 2013 (mLearn 2013). Ge, X., & Land, S. M. (2003). Scaffolding students’ problem-solving processes in an illstructured task using question prompts and peer interactions. Educational Technology Research and Development, 51(1), 21–38. Ge, X., & Land, S. M. (2004). A conceptual framework of scaffolding ill-structured problem solving processes using question prompts and peer interactions. Educational Technology Research and Development, 52(2), 2. Gibbs, G. (1988). Learning by doing: a guide to teaching and learning methods. London: Oxford Centre for Staff and Learning Development. Hamari, J., & Lehdonvirta, V. (2010). Game design as marketing: how game mechanics create demand for virtual goods. International Journal of Business Science and Applied Management, 5(1), 14-29. Heeter, C., Lee, Y.-H., Medler, B., & Magerko, B. (2011). Beyond player types: gaming achievement goal. Paper presented at the ACM SIGGRAPH 2011 Game Papers, Vancouver, British Columbia, Canada. Hogarth, R. M. (1981). Judgement and choice: The psychology of decision. New York: Wiley. Huizenga, J., Admiraal, W., Akkerman, S., & ten Dam, G. (2009). Mobile game-based learning in secondary education: engagement , motivation and learning in a mobile city game. JCAL, 29(2), 115-126. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 40 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Hurd, S. (2008). Affect and strategy use in independent language learning. In S. Hurd & T. Lewis (Eds.), Language learning strategies in independent settings (pp. 218–236). Bristol, UK: Multilingual Matters. Iacovides, I., Aczel, J., Scanlon, E., & Woods, W. (2012). Investigating the relationships between informal learning and player involvement in digital games. Learning, Media and Technology, 37(3), 321–327. Jordan, S. (2012). Student engagement with assessment and feedback: some lessons from short-answer free-text e-assessment questions Computers and Education, 58(2), 818– 834. Jordan, S., Jordan, H., & Jordan, R. (2012). Same but different, but is it fair? An analysis of the use of variants of interactive computer-marked questions. International Journal of eAssessment, 2(1). King, A. (1999). Discourse patterns for mediating peer learning. In A. O’Donnell & A. King (Eds.), Cognitive perspective on peer learning (pp. 87–115). Mahwah, NJ: Lawrence Erlbaum Associates. Kolb, D. A. (1984). Experiential learning: experience as the source of learning and development. Englewood Cliffs, NJ: Prentice Hall. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of applied psychology, 78(2), 311-328. Kukulska-Hulme, A. (2005). Mobile usability and user experience. In A. Kukulska-Hulme & J. Traxler (Eds.), Mobile Learning: A handbook for educators and trainers (pp. 4556). Abingdon, UK: Routledge. Lave, J., & Wenger, E. (1991). Situated learning: legitimate peripheral participation. Cambridge: Cambridge University Press. Lea, S. J., Stephenson, D., & Troy, J. (2003). Higher education students’ attitudes to studentcentred learning: beyond ‘educational bulimia’. Studies in Higher Education, 28(3), 321–334. Little, D. (2010). The linguistic integration of adult migrants: towards the evaluation of policy and practice: intergovernmental conference, Strasbourg, 24-25 June 2010: Council of Europe. Luckin, R. (2010). Re-designing learning contexts: technology-rich, learner-centred ecologies. London: Routledge. Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31, 199-218. Nix, I., & Wyllie, A. (2009). Exploring design features to enhance computer-based assessment: learners' views on using a confidence-indicator tool and computer-based feedback. British Journal of Educational Technology 42(1), 101-112. Nojima, M. (2007). Pricing models and Motivations for MMO play Proceedings of DiGRA 2007: Situated Play (pp. 672-681). Tokyo: Authors & Digital Games Research Association (DiGRA). O'Neil, H. F., Wainess, R., & Baker, E. L. (2005). Classification of learning outcomes: evidence from the computer games literature. The Curriculum Journal, 16(4), 455474. Oatley, K., & Jenkins, J. (1996). Understanding emotions. Cambridge, MA.: Blackwell. Obikwelu, C., & Read, J. C. (2012). The Serious Game Constructivist Framework for Children's Learning. Procedia Computer Science, 15, 32-37. doi: http://dx.doi.org/10.1016/j.procs.2012.10.055 MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 41 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Palinscar, A. S. (1998). Social constructivist perspectives on teaching and learning. Annual Review of Psychology, 49, 345–375. Pritchard-Newcombe, L. (2009). Think without limits: you can speak Welsh! : Carreg Gwalch Press. Ross, S., Jordan, S., & Butcher, P. (2006). Online instantaneous and targeted feedback for remote learners. In C. Bryan & K. Clegg (Eds.), Innovative assessment in Higher Education (pp. 123–131): Routledge/Taylor and Francis. Rowntree, D. (1987). Assessing students: how shall we know them? London: Kogan Page. Sadler, D. R. (1998). Formative assessment: revisiting the territory. Assessment in Education, 5(1), 77-85. Schon, D. A. (1983). The reflective practitioner: how professionals think in action. London: Maurice Temple Smith. Schroth, M. L. (1992). The effects of delay of feedback on a delayed concept formation transfer task. Contemporary educational psychology, 17(1), 78-82. Schwabe, G., & Goth, C. (2005). Mobile learning with a mobile game: design and motivational effect. Journal of Computer Assisted Learning, 21(3), 204-216. Sly, L. (1999). Practice tests as formative assessment improve student performance on computermanaged learning assessments. Assessment and Evaluation in Higher Education, 24(3), 339-343. Swaffield, S. (2011). Getting to the heart of authentic assessment for learning. Assessment in Education: Principles, Policies and Practice, 18(4), 433-449. Vihavainen, S., & Väänänen-Vainio-Mattila, K. (2013). The implications of mobile notifications for user experience of a social network service. International Journal of Interactive Mobile Technologies, 7(2). doi: http://dx.doi.org/10.3991/ijim.v7i2.2373 Vygotsky, L. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA.: Harvard University Press. Wang, Y., & Mainwaring, S. D. (2008). Human-Currency Interaction: learning from virtual currency use in China. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. Wenger, E. (1998). Communities of practice: learning, meaning, and identity. Cambridge: Cambridge University Press. Wu, L., & Looi, C.-K. (2012). Agent prompts: scaffolding for productive reflection in an intelligent learning environment. Educational Technology & Society, , 15(1), 339–353. Yorke, M. (2001). Formative assessment and its relevance to retention. Higher Education Research and Development, 20(2), 115-126. Zaïane, O. R., & Luo, J. (2001). Web usage mining for a better web-based learning environment. 4th IASTED International Conference on Advanced Technology for Education (CATE’01) Banff, Canada (27 – 28 June) (pp. 60-64). Zimmerman, B., & Schunk, D. (2007). Motivation: an essential dimension of self-regulated learning. In D. Schunk & B. Zimmerman (Eds.), Motivation and self-regulated learning: theory, research and applications. New York: Taylor & Francis. MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 42 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services APPENDIX A: REVIEW OF TWO WEB BASED LEARNING ENVIRONMENTS: BUSUU.COM AND DUOLINGO To inform MASELTOV’s exploration of persuasive learning services (WP7), an expert evaluator at the Open University has been undertaking a review of web based learning environments. This research has enabled us to identify the range and categories of feedback and progress indicators employed within these environments, and is therefore included as examples of current practice. The research is not complete and hence this table should be considered a work in progress. Function Form in www.busuu.com Domain(s) Type of FPI Getting started All new users are sent a welcome email with links to key aspects of the website and encouraging them to take a tour of the website. Cognitive Affective Social Prompt The website uses colour to make usage appealing. It looks like it will be fun to interact with. Form in Duo Lingo (http://www.duolingo.com) No welcome email received when users ‘sign up with email’ (i.e. the evaluator) Domain(s) Type of FPI Affective Prompt, Learning guide The login area of the website has usability/accessibility issues which are problematic. Affective The website is clearly laid out. The landing page makes effective use of colour, i.e. green for growth and the bright blue sky. The tree growing concept is used throughout the language learning process. It is easy to see how you can sign up either with Facebook or email and it is easy to see which languages you can learn with Duolingo. Affective It is easy to see where to start learning. The lesson to be studied is clearly highlighted in a bright colour MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 43 of 50 Prompt, Learning guide Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Encourages continued learning The concept of a language garden and growth. Right from the start of language learning users are informed they will earn gifts for their garden as they progress. Affective Levels of difficulty, Achievemen ts, Virtual currencies The longer term Premium Membership costs are far more cost effective than short terms Memberships. Cognitive Reputation 3 main progress indicators are used as feedback and progress indicators. These are clearly displayed on a user’s home page in the right hand menu Skill points Users can earn ‘skill points’ (which are used to track progress). Users earn skill points every day. They can sign up to a daily reminder email at a specific time to remind them to do learning. New users, who may not wish to become a Premium Member straight away (Free Members) are not provided with access to the website via a smartphone, cannot be rewarded with a busuu.com Certificate, and cannot access the grammar units. The can only do the vocabulary, reading and writing exercises and the interactive exams (formative assessment). Words A record is kept of how many words users learn (and what they are) in a separate Word page (see Vocabulary) Cognitive Affective Social (if shared) Cognitive Affective Social (if shared) Achievements Prompt Cognitive, Affective Achievement One (and more) day streaks. The learner is praised for committing time to learning on an uninterrupted period of days (learning 20 days in a row would be a “twenty day streak”) As learning activities are completed within a learning unit, the colour of the activity changes from red to green. The lessons are displayed in a route map or (‘French skill tree). Users can watch their tree’ growth as lessons completed turn from ‘greyed out’ to colour. Language learning is presented in a structured pathway enabling learners to develop language skills in a systematic way A positive ring tone can be heard every time a user gets an answer correct (and vice versa) Cognitive Premium users can select The claim that language learning is ‘scientifically proven’ Achievemen t MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 44 of 50 Affective Achievement Affective Affective Learning guide, Assessment tool Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Encourages continued learning vocabulary they wish to keep and save it as a .pdf file which they can then print if required. Personalised activity feeds showing the Exercises and Corrections users have done Social, Cognitive Assessment tool, Progress indicator (not presented as a progress bar) Levels of difficult, Dashboard Reflection Goal setting No specific tools could be found for reflection, but the multimodal approach to language learning would encourage reflection. Users can set their learning goal on their home page. Access to Users have to check every answer in each lesson before they can continue to the next question. If an answer is wrong, an explanation is provided. There is no option to re-do the question, but the evaluator thinks that errors are used in subsequent questions to ensure the user learns correctly. Affective Assessment tool The multi-modal approach to learning implicitly encourages reflective thinking, but a reflective tool could be an additional benefit if introduced at various stages in the learning process, for example, at the end of Lesson 5. Not found in duolingo MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 45 of 50 Cognitive Prompt Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services this feature is easy via an easily distinguished button in a specific block in the right hand menu. Study planning Participation Once set, ‘My goal’, the no. of units and the number of days that users have left to achieve the goal is displayed in the right hand menu. This allows users to easily track their progress. Users can easily change their goal. Once a user has set their learning goal they can see how long they have left to achieve their goal and how many units they have to achieve their goal so could use the tool to plan their study. However, no guidance on planning study is provided, it is left to users to work out how to plan their study. The website is easy to use. Cognitive Progress bar Cognitive Affective Cognitive It is not intuitive to access the Friend Request (see Friend Request) Progress bar Users can see their progress in their Skill tree, so could use this to plan their study, but no guidance is provided. The website is easy to use See also Encouraging continued learning. Social Friend requests that match the language learning needs that users have identified are displayed only as numbers and overlay icons without clear meanings. However, previous MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 46 of 50 Cognitive Affective Progress indicator Dashboard Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services Confidence building Confidence building Generating a sense of community experience of Facebook tells me that the red square indicates significance. Badges, busuu-berries and starrating (which displays level of activity on the website) all helps to develop confidence. Affective Premium users can practise speaking a language and using the correct pronunciation through voice-recording. Cognitive, Affective Confidence is built where patterns of dialogue interaction are reinforced as higher levels of language learning are accessed. Cognitive At the end of a learning unit users can listen to a podcast of the vocabulary and dialogue introduced. The busuu community is via a network of friends who help one another to develop their language skills. Friend requests are easy to access; to accept or ignore and a good deal of information is provided.about each request, e.g. language they speak and languages they are learning, recognition of Virtual currencies Users can see their skill points, Vocabulary development and streaks easily. Affective Virtual currencies A positive ring tone can be heard every time a correct answer is given and when a lesson is completed joyful music is played. Affective Progress indicator Finding friends at the start of learning is fairly easy. There are two options: 1) to invite friends to learn together and 2) to find friends on Facebook. Social Reputation Levels of difficulty Cognitive, Affective Levels of difficulty Cognitive, Affective Social Reputation Social Inviting friends in Duolingo means personal friends, i.e. friends you have contact with via email and/or through Facebook. You can only invite one friend at a time. These friends are, not MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 47 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services achievement in the form of starrating, blueberries and badges. Users can develop their writing skills by posting a message which is shared with selected others. If another user responds and corrects the message, this encourages wider participation in the community. friends who are already in the Duolingo community, as occurs in busuu.com. Social, Cognitive, Affective Users can ‘Join the Duolingo family’ by following people on Twitter or by liking Duolingo on Facebook. The indicators suggest there are 41,000 followers on Twitter and 94,000 on Facebook. Social, Cognitive, Affective Duolingo has a competitive element, which was not seen in busuu.com. Clearly visible on the right hand side of the screen is a section where users can ‘compete with your friends’. The pedagogy underlining this feature is unclear and would benefit from description. Users can develop dialogue skills by chatting with other conversation partners within the website Social, Affective, (may be cognitive) Social, Affective, Cognitive Social, affective Cognitive, social Prompts Email prompts are sent to users who have not accessed the website for X amount of time. Affective Prompts A tab next to the Skill Tree called ‘Stream’ presents an up-to-date list of a users’ communications, their level of study (as it progresses) and the number of skill points they have acquired. Social Users can upload, read and translate documents posted and give feedback Social, Affective Users can share feedback, present ideas in the Discussion area. The tutor agent (displayed as a green owl) pops up occasionally to remind users that they can always over a word to see a Social MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 48 of 50 Cognitive Prompts Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services translation. Email prompts are sent to users when a friend has posted a message. Recognition of achievement for self Shared recognition of achievement Fun/enjoyment The badges and busuu-berries demonstrate achievements. Progress towards ‘My goal’ is easily visible on a user’s home page. In a learning unit review users can self-assess their learning through a series of Q & A. They also receive direct feedback from busuu.com on whether their answers were correct or not. Users badges, busuu-berries and star ratings are all displayed on the profile window that pops up whenever a mouse is over a learner’s profile. Users can share their learning goal on Facebook or Tweet about it. The website is colourful and looks fun to use. A ‘language garden’ provides learners with a visual representation and an affective dimension to their learning and Affective Virtual currency Affective, Cognitive Reputation Cognitive, Affective Assessment tool Affective Virtual currency Duolingo’s algorithm informs users when they need to practice words they have learnt so they transfer to long-term memory Users can easily view their Skill points; their progress in the Skills tree, and the number of words they have learned at various locations on the website Cognitive Prompts Affective Social Reputation Users can share their learning goal on Facebook or Tweet about it. Affective Social Reputation Multi-media approach to learning Affective, Cognitive Affective Social Reputation Affective The website is appealing Affective Learning guide MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 49 of 50 Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Network Services development. It can be very rewarding to see things grow in addition to internal recognition. It is a form of extrinsic motivation. Drag and drop quizzes and multiple choices questions are presented throughout the learning activities. Fun and motivational. Users can practice their writing, speaking and listening skills with native speakers who use the website. Affective, cognitive Social, affective, cognitive Assessment tools Reputation MASELTOV – DELIVERABLE D7.2“Feedback and Progress Indicators” Page 50 of 50