Assessment for Learning Formative Assessment

OECD/CERI International Conference “Learning in the 21st Century: Research, Innovation and Policy” Assessment for Learning Formative Assessment ORGANISATION F OR E CONOMIC CO-OPERATION AND DEVELOPMENT ORGANISATION DE COOPÉRATION ET DE DÉVELOPPEMENT ÉCONOMIQUES ASSESSMENT FOR LEARNING – THE CASE FOR FORMATIVE ASSESSMENT This paper provides findings on assessment for learning, drawn from recent analyses undertaken by CERI. It begins with analysis of the formative approach in exemplary practice carried out in secondary schools in eight education systems. The second half of the paper comprises key analyses on formative assessment in adult language, literacy, and numeracy provision, and a framework for strengthening policy and practice across the sector as well as for building the evidence base. Assessment is vital to the education process. In schools, the most visible assessments are summative. Summative assessments are used to measure what students have learnt at the end of a unit, to promote students, to ensure they have met required standards on the way to earning certification for school completion or to enter certain occupations, or as a method for selecting students for entry into further education. Ministries or departments of education may use summative assessments and evaluations as a way to hold publicly funded schools accountable for providing quality education. Increasingly, international summative assessments – such as OECD’s Programme for International Student Assessment (PISA) – have been important for comparing national education systems to developments in other countries. But assessment may also serve a formative function. In classrooms, formative assessment refers to frequent, interactive assessments of student progress and understanding to identify learning needs and adjust teaching appropriately. Teachers using formative assessment approaches and techniques are better prepared to meet diverse students’ needs – through differentiation and adaptation of teaching to raise levels of student achievement and to achieve a greater equity of student outcomes. But there are major barriers to wider practice, including perceived tensions between classroom-based formative assessments, and high visibility summative tests to hold schools accountable for student achievement, and a lack of connection between systemic, school and classroom approaches to assessment and evaluation. The principles of formative assessment may be applied at the school and policy levels, to identify areas for improvement and to promote effective and constructive cultures of evaluation throughout education systems. More consistent use of formative assessment throughout education systems may help stakeholders address the very barriers to its wider practice in classrooms. This overview shows how formative assessment promotes the goals of lifelong learning, including higher levels of student achievement, greater equity of student outcomes, and improved learning to learn skills. The chapter then discusses barriers to wider practice of formative assessment and ways in which those barriers can be addressed, and outlines the study scope and methodology. Meeting goals for lifelong learning Each of the national and regional governments participating in this study promotes formative assessment as a means to meeting the goals of lifelong learning. They are motivated by quantitative and qualitative evidence that teaching which incorporates formative assessment has helped to raise levels of student achievement, and has better enabled teachers to meet the needs of increasingly diverse student populations, helping to close gaps in equity of student outcomes. Teachers using formative assessment 1 approaches guide students toward development of their own “learning to learn” skills – skills that are increasingly necessary as knowledge is quickly outdated in the information society. Promoting high-performance: raising levels of student achievement Formative assessment methods have been important to raising overall levels of student achievement. Quantitative and qualitative research on formative assessment has shown that it is perhaps one of the most important interventions for promoting high-performance ever studied. In their influential 1998 review of the English-language literature on formative assessment, Black and Wiliam concluded that: “… formative assessment does improve learning. The gains in achievement appear to be quite considerable, and as noted earlier, among the largest ever reported for educational interventions. As an illustration of just how big these gains are, an effect size of 0.7, if it could be achieved on a nationwide scale, would be equivalent to raising the mathematics attainment score of an ‘average’ country like England, New Zealand or the United States into the ‘top five’ after the Pacific Rim countries of Singapore, Korea, Japan and Hong Kong.” (Beaton et al., 1996, Black and Wiliam, 1998, p. 61) These findings provide a strong foundation for further research on effective teaching, learning and assessment strategies (including the present study). Promoting high-equity: education for all The “What Works” case studies support the idea that formative assessment methods may help create greater equity of student outcomes. Although Black and Wiliam (1998 and in Part III of this study) note that research on the effectiveness of formative assessment is lacking in regard to underachieving students or students’ race, class, or gender, it is worth noting that several of the case study schools with large percentages of “disadvantaged” students had moved from “failing” to exemplary status over the past several years. Case study schools featuring programmes specifically targeted to the needs of underachieving students also yielded positive results. Teachers in the case study schools used formative assessment to establish factors lying behind the variation in students’ achievements in specific subjects, and to adapt teaching to address identified needs. Such approaches represent a move away from models of equity that suggest that all children should receive exactly the same inputs (they are “indifferent to difference”, Perrenoud suggests [1998]), or “deficit” models that identify certain children as “disadvantaged”. Instead, teachers adjust methods to recognise individual, cultural, and linguistic differences between children (see for example, Bruner 1996; Bishop and Glynn, 1999). Building students’ skills for learning to learn Formative assessment builds students’ “learning to learn” skills by: x Placing emphasis on the process of teaching and learning, and actively involving students in that process. x Building students’ skills for peer- and self-assessment. x Helping students understand their own learning, and develop appropriate strategies for “learning to learn”. 2 Students who are actively building their understanding of new concepts (rather than merely absorbing information), who have developed a variety of strategies that enable them to place new ideas into a larger context, and who are learning to judge the quality of their own and their peer’s work against well-defined learning goals and criteria, are also developing skills that are invaluable for learning throughout their lives. Addressing barriers to wider practice The major (although not the only) barriers to wider practice of formative assessment that emerged from the case studies include: x The tension between classroom-based formative assessments of student learning, and high visibility summative tests – that is, large-scale national or regional assessments of student performance that are intended to hold schools accountable for meeting standards, and that may hold particular consequences for low or underperforming schools. Too often, highly visible summative tests used to hold schools accountable for student achievement drive what happens in classrooms. x A lack of connection between systemic, school and classroom approaches to assessment and evaluation. Too often, information gathered through national or regional monitoring systems, or even in school-based evaluations, is seen as irrelevant or unhelpful to the business of teaching. Too often, information gathered in classrooms is seen as irrelevant to the business of policy making. Addressing the formative-summative tension While teachers often express ambivalence or resistance to external summative tests, there is nothing inherent in summative assessment to prevent teachers from using formative methods. Indeed, summative results can be used formatively. Yet, in several countries, summative assessments have dominated political debate over education. Often, schools with poor results on public examinations face major consequences, such as threatened shut-downs, reconstitution, or firing of teachers. In environments where summative tests have high visibility, teachers often feel compelled to “teach to the test”, and students are encouraged to meet performance goals (to perform well on tests) at the expense of learning goals (that is, to understand and master new knowledge). Many – if not most – teachers perceive these external assessments as being in conflict with – or even inimical to – the practice of formative assessment. Poorly designed external tests, media league tables which use a narrow set of data to compare performance across schools, and lack of connection between tests and curriculum can also inhibit innovation. Note that, for the purposes of this study, assessment refers to judgments of student performance, while evaluation refers to judgements of programme or organisational effectiveness. In all cases, the use of data to inform teacher planning of future classroom activities, or at the national level to inform and adapt policies, might be considered as secondary levels of formative assessment. Strengthening cultures of evaluation One of the particular interests for this study has been in examining how teachers and school leaders create or strengthen cultures of evaluation. In a culture of evaluation, teachers and school leaders use information on students to generate new knowledge on what works and why, share their knowledge with colleagues, and build their ability to address a greater range of their students’ learning needs. 3 A culture of evaluation refers to the development of a shared language regarding the goals of learning and teaching, as well as a shared understanding of the purposes of evaluation in meeting these goals. Several OECD countries support school-based evaluation as a key component, either as the primary or only form of school-level evaluation, or as a complement to external testing, inspections and programme evaluation. All education stakeholders are thus focused on developing strategies for school improvement. School-based evaluation helps school leaders and teachers to focus their attention on resources and organisational challenges, and to develop solutions appropriate to their circumstances. The idea of school-based evaluation is quite appealing because it involves school staff directly, incorporates local knowledge, and potentially, directly shapes school improvement. However, schoolbased evaluation is not always well aligned with the work of schools. Evaluation tools may be more suited to the needs of policy officials than they are to schools and teachers. Moreover, the skills required for gathering and interpreting school or programme level data are quite different than those required for classroom assessment (Monsen, 2002; Simmons, 2002; Lander and Ekholm, 1998). Some countries that do not now have external examinations and/or inspection systems are considering adopting such approaches to ensure greater school accountability. By contrast, a few countries that have promoted external examinations are paying greater attention to the potential for school-based evaluation to shape school improvement. Policy officials can learn much from the experiences of their counterparts. No matter which approach is chosen, assessment and evaluation are only really effective if the data gathered at different levels are taken into account throughout systems. Ideally, information gathered in assessments and evaluations is used to shape strategies for improvement at each level of the education system. At the classroom level, teachers gather information on student understanding, and adjust teaching to meet identified learning needs. At the school level, school leaders use information to identify areas of strength and weakness across the school, and to develop strategies for improvement. At the policy level, officials use information gathered through national or regional tests, or through monitoring of school performance, to guide investments in training and support for schools and teachers, or to set broad priorities for education. In this way, summative information is used formatively at each level of the system (see Figure 1). Teachers, school leaders and policy officials are more likely to use assessment information when assessments are well coordinated, and it is clear why and how the information is relevant to their work. 4 Figure 1. Coordinating assessment and evaluation Assessment for student learning Evaluation for school improvement Evaluation for systemic improvement Note: Information gathered at each level of the system can be used to identify strengths and weaknesses, and to shape strategies for improvement. Source: Authors. Formative assessment – while not a “silver bullet” that can solve all educational challenges – offers a powerful means for meeting goals for high-performance, high-equity of student outcomes, and for providing students with knowledge and skills for lifelong learning. Systems that address tensions that prevent wider practice of formative assessment and that foster cultures of evaluation are likely to make much greater progress toward these goals. 5 THE ELEMENTS OF FORMATIVE ASSESSMENT: CASE STUDY FINDINGS AND SUPPORTING RESEARCH The key elements that have emerged from the case studies and related research are: 1. Establishment of a classroom culture that encourages interaction and the use of assessment tools. 2. Establishment of learning goals, and tracking of individual student progress toward those goals. 3. Use of varied instruction methods to meet diverse student needs. 4. Use of varied approaches to assessing student understanding. 5. Feedback on student performance and adaptation of instruction to meet identified needs. 6. Active involvement of students in the learning process. What is most striking about the case study findings is that in all cases, teachers had incorporated each of the six elements into regular practice. While teachers may have placed different emphases on the various elements (for example, some teachers placed greater stress on providing students with feedback; other teachers were more focused on providing students with a variety of learning opportunities), they used each of these elements to shape teaching and assessment. Teachers thus created a framework, language and tools, using the elements of formative assessment to shape their approach to teaching and learning. 6 Figure 2. The six key elements of formative assessment Note: Teachers across the case study schools used formative assessment as a framework for teaching and learning. Culture change was central to creating and sustaining regular practice of formative assessment. Teachers drew upon each of these elements to create a dynamic teaching and learning environment and to move students toward learning goals. Source: Authors. Element 1: Establishment of a classroom culture that encourages interaction and the use of assessment tools The concept of formative assessment was first introduced in 1971 by Bloom, Hastings and Maddaus. They formally introduced the idea that assessment need not be used solely to make summative evaluations of student performance, arguing that teachers should include episodes of formative assessment following phases of teaching. During these episodes teachers should provide students with feedback and correction as a way to remediate student work. Most experts now consider formative assessment as an ongoing part of the teaching and learning process. Formative assessment thus becomes a central element in teaching and learning. Teachers across the case study schools have integrated formative assessment into their teaching, establishing classroom cultures that encourage interaction and use of assessment tools. In each of the case studies, teachers noted the importance of helping students to feel safe to take risks and make mistakes in 7 the classroom. This is, in part, simply practical: children who feel safe to take risks are more likely to reveal what they do and don’t understand, an essential feature of the formative process. Research also highlights the importance of focusing students’ attention on mastering tasks, rather than on competition with peers, and in developing emotional competencies. Emotional competencies, such as self-awareness, self-control, compassion, co-operation, flexibility, and the ability to make judgments on the value of information serve students well in school and throughout their lives (OECD, 2002, p. 58). Emotions also affect the student’s self-esteem, motivation and ability to regulate his or her own learning. Element 2: Establishment of learning goals, and tracking of individual student progress toward those goals Several OECD countries have established general standards for student achievement, and monitor students’ progress toward those standards. Teachers in several of the case study schools worked together to define the standards in more detail, developing and sharing criteria with colleagues and students, and developing new internal systems to track individual student progress. Teachers in the case study schools look to these objective standards to set out learning goals for students, sometimes scaffolding these goals for weaker students. The teachers have also moved away from traditional systems of marking – which tend to rely on “social comparison” of student performance (that is, comparison of each students’ performance with that of their peers) toward methods that allowed them to track an individual student’s progress toward the learning goals, as judged through established criteria. International research supports the idea that tracking a student’s progress toward objective learning goals is more effective than is comparison with peers’ progress (Cameron and Pierce, 1994; Kluger and DeNisi, 1996; Heckhausen, 1989; and Rheinberg and Krug, 1999). In situations of comparison, weaker students absorb the idea that they lack ability, and thus lose motivation and confidence. Ames (1992) notes that teachers’ beliefs about the importance of effort, rather than ability, also play an important role in students’ beliefs about themselves. Appropriate reference to an individual student’s progress and opportunities to improve work based on feedback can help counter the negative impact of social comparisons. Mischo and Rheinberg (1995) and Köller (2001) also found positive effects in several experimental and field studies where teachers referred to student progress over time. Positive effects were identified for students’: x Intrinsic motivation. x Self-esteem. x Academic self-concept. x Causal attributions. x Learning [see particularly Krampen (1987)]. The establishment of learning goals and tracking of student progress toward those goals makes the learning process much more transparent; students do not need to guess what they need to do to perform well. Teachers also help students to track their own progress and to build confidence. 8 Element 3: Use of varied instruction methods to meet diverse student needs Teachers in the case study schools adjust their teaching methods to meet the needs of a variety of students. In some cases, this means that they adjust teaching to recognise different emotional styles. Teachers note that more vulnerable students need help in developing greater emotional competency. (For a more detailed discussion on emotions and cognition, see OECD, 2002.) These teachers are concerned with building students’ confidence in their own skills and knowledge and in their ability to manage their own learning. Social and cognitive psychologists, anthropologists and other social scientists have increasingly recognised that the knowledge and experiences children bring to school shape their learning experiences (Bruner, 1996; Bransford et al., 1999). Such prior knowledge is shaped, in part, by learners’ ethnicity, culture, socio-economic class, and/or gender. Teachers can help students learn new concepts and ideas in ways that connect to their prior understandings and ways of looking at the world. Teachers who are attuned to variations in cultural communication patterns and sensitive to individual ways of communicating are more likely to draw out what children understand, and how they develop their understanding of new ideas (Bishop and Glynn, 1999). Research has found that parents can play an important role here, too, because they share their children’s life experiences, are well acquainted with their abilities and interests, and can help their children make connections between ideas (Bransford et al., 1999). Swiss education scholar Philippe Perrenoud proposes that: “… [t]o the extent that pupils do not have the same abilities, nor the same needs or the same way of working, an optimal situation for one pupil will not be optimal for another …. One can write a simple equation: diversity in people + appropriate treatment for each = diversity in approach”. (Perrenoud, 1998, p. 93-94) Early research findings suggest that there is a need for a fundamental re-thinking of approaches to reaching equitable student outcomes. But there is also a need for more refined research on the impact of formative assessment methods for different students. Such research might address whether and how formative assessment can address the needs of students based on individual differences, such as emotional style, or ethnicity, culture, socio-economic class, and/or gender Element 4: Use of varied approaches to assessing student understanding Teachers in the case study schools use varied approaches to assessing individual student progress over time, in realistic settings, and in a variety of contexts. Students who may not perform well in certain tasks have the opportunity to demonstrate their knowledge and skills in others. Such varied assessments also draw out information on students’ ability to transfer learning to new situations – a skill emphasised as important to learning to learn – and on how student understanding might be corrected or deepened. These varied assessments may include tests and other summative forms of assessment, so long as the information on student performance gathered in the tests is used to inform further learning. Summative results, when embedded in the wider teaching and learning environment, are more likely to be used formatively. They also help to lower the stress of tests, which can have a have negative impact on the self-esteem of lower achieving students (See for example, a study conducted by the EPPI – Centre at the Institute of Education, University of London, June 2002). Element 5: Feedback on student performance and adaptation of instruction to meet identified needs Feedback is vital to formative assessment, but not all feedback is effective. Feedback needs to be timely and specific, and include suggestions for ways to improve future performance. Good feedback is 9 also tied to explicit criteria regarding expectations for student performance, making the learning process more transparent, and modelling “learning to learn” skills for students. In their review of the English-language literature, Black and Wiliam (1998) identified a number of studies, conducted under ecologically valid circumstances (that is, controlled experiments conducted in the student’s usual classroom setting and with their usual teacher) to support this point of view. For example, “ego-involving” feedback (even in the form of praise) rather than feedback on the task at hand appears to have a negative impact on performance (Boulet et al., 1990). Students also obtain better results when they are working toward process goals rather than product goals, and when tracking progress toward overall goals of learning (Schunk, 1996). Grades may actually undermine the positive help of specific feedback on tasks (Butler, 1995). Teachers also benefit from the feedback process. When providing feedback, teachers pay closer attention to what students do and do not understand well, and are better able to adjust teaching strategies to meet identified student needs. Element 6: Active involvement of students in the learning process Ultimately, the goal of formative assessment is to guide students toward the development of their own “learning to learn” skills (also sometimes referred to as “metacognitive” strategies). Students are thus equipped with their own language and tools for learning and are more likely to transfer and apply these skills for problem solving into daily life; they strengthen their ability to find answers or develop strategies for addressing problems with which they are not familiar. In other words, they develop strong “control” strategies for their own learning. “Metacognition” involves awareness of how one goes about learning and thinking about new subject matter and is sometimes referred to as “thinking about thinking”. The student who has an awareness of how he or she learns is better able to set goals, develop a variety of learning strategies, and control and evaluate his or her own learning process. As evidence of this, PISA 2000 found that: “… Within each country, students who use… [metacognitive and control strategies] more frequently tend to perform better on the combined PISA reading literacy scale than those who do not (although whether the learning strategies cause the better results cannot be established). … [T]he strategies are essential for effective self-regulation of learning because they help students to adapt their learning to particular features of the task on which they are working. Schools may need to give more explicit attention to allowing students to manage and control their learning in order to help them all to develop effective strategies, not only to support their learning at school but also to help them with the tools to manage their learning later in life”. (OECD, 2001, p. 110) Importantly, PISA also found that students are unlikely to use control strategies if they lack motivation or self-confidence (OECD, 2003). Students’ personal judgments about their ability to carry out a task (“self-efficacy”) also significantly influence task performance (Pajares, 1996). Thus, a key role for teachers is to help children build confidence, and develop a variety of learning strategies. Teachers in the case study schools model such learning behaviour, teach self-assessment skills and help students to analyse of how well different learning strategies have worked for them in the past. Such teaching approaches may be particularly important for children who do not have extra support for learning at home (OECD, 2003; Bransford et al., 1999). 10 POLICY IMPLICATIONS Policy principles of formative assessment to promote wider, deeper and more sustained practice are to: 1. Keep the focus on teaching and learning. 2. Align summative and formative assessment approaches. 3. Ensure that data gathered at classroom, school and system levels are linked and are used formatively. 4. Invest in training and support for formative assessment. 5. Encourage innovation. 6. Build stronger bridges between research, policy and practice. The aim of these principles is to ensure that the schools included in this study are no longer considered exceptional, but are representative of common practice. 11 INVESTMENTS IN FURTHER RESEARCH While there is evidence that formative assessment methods have a significant impact on student learning, there is a need for further research. Future research may address: x The impact of formative assessment on general student achievement. While there is convincing evidence that formative assessment is indeed highly effective in raising levels of student achievement (see Black and Wiliam, 1998; Natriello, 1987; Crooks, 1988), the research should be extended and strengthened. Further research in this area may include both quantitative and qualitative studies of formative methods, drawing upon a breadth of international educational experiences. x The relative impact of formative assessment methods for underachieving students. Several studies show that formative assessment methods have an even stronger impact for underachieving students. Selected studies focus on teaching which stresses the importance of effort over ability, or of task-centred feedback (as opposed to ego-involving feedback). These studies show relatively stronger improvements for previously underachieving students. Further research in this area may have significant implications for teachers working with larger groups of underachieving students or in “failing” schools. x Effective formative approaches for students based on gender, ethnicity, socio-economic status, or age. As noted earlier in this study, there is a need for more refined knowledge of what works for students in different socio-economic or demographic groups. Research in this area may explore the differential impact of methods on diverse learners. For example, research may explore the circumstances under which different students thrive on competition, or in more cooperative situations. Research may also explore the extent to which principles of teaching that work well for a defined group, such as the Maori Mainstream Programme (Te Kotahitanga) included in this study, transfer to other groups of students. Studies in this area may prove extremely important to addressing long-term challenges of closing equity gaps in student achievement. x Connections between students’ emotions and learning. The connections between positive emotions and improved learning are a major theme of neuro-scientific research on learning. This research, along with work in the area of educational psychology, can inform studies on the impact of different formative methods on student emotions, motivation, self-perceptions and achievement. x The expansion of teacher repertoires to meet identified student needs. As noted earlier, if teaching is limited, the quality of student assessment will also be limited. Teachers need a healthy repertoire of approaches to setting up learning situations and responding to student learning needs. Teachers and researchers may form a healthy partnership for research in this area. Formative assessment requires greater transparency in teaching and learning, and is also quite iterative. The approach is ideal for researchers who want to explore the process of teaching and learning in normal classroom settings. Teachers using formative assessment may also draw upon research to further build their repertoires. 12 x The challenges of deepening and broadening practice of effective formative assessment approaches and techniques. This study has asserted that formative assessment methods are more than a passing fad. Still, there are important challenges to deepening and broadening practice of effective formative assessment methods and techniques. Researchers should pay careful attention to the success of various dissemination and implementation strategies. Policy, in the formative spirit, can draw upon this knowledge to adapt and improve strategies and deepen impact. 13 TEACHING, LEARNING AND ASSESSMENT FOR ADULTS: IMPROVING FOUNDATION SKILLS Introduction In many OECD countries, large portions of populations do not have the skills needed to function fully and effectively in daily life. Adult foundation skill learning for those with low language, literacy and numeracy skills has therefore become an increasingly urgent policy issue. However, it is less certain that this issue is receiving the attention and priority it merits. Large-scale surveys, such as the International Adult Literacy (1994 1998), the Adult Literacy and Life Skills surveys (OECD and Statistics Canada, 2005), and the new Programme for International Assessment of Adult Competencies (PIAAC) provide vital data on adult skills in populations and the dimensions of the challenge. OECD publications such as Beyond Rhetoric (2003) and Promoting Adult Learning (2005) have provided analyses of how systems can make more effective investments and promote access to learning opportunities. This study complements these macro-level analyses, bringing a much-needed focus to effective teaching, learning and assessment within classrooms. With a better understanding of the factors that contribute to the quality of provision and outcomes, policy makers can provide more effective leadership and support. The study is part of the “What Works in Innovation in Education” series of the OECD’s Centre for Educational Research and Innovation (CERI). The series was initiated in 1993 to examine innovations related to common policy concerns across a small number of countries (between six and ten). Seven countries identified case studies of exemplary classroom practice: Belgium (Flemish Community), Denmark, England, France, Norway, Scotland and the United States. Nine countries contributed background reports detailing the challenges and policy responses for adult foundation skill learners. They include Australia, Belgium (Flemish Community), Denmark, England, Norway, New Zealand, Scotland, Spain and the United States. As with the previous “What Works” study (Formative Assessment: Improving Learning in Secondary Classrooms, CERI/OECD, 2005), this study includes reviews of research from different linguistic traditions. The reviews provide insight into different conceptual approaches to common challenges, and make accessible a broader range of empirical research. This approach has been particularly important for the understudied area of adult foundation skill learning. The combined reviews provide a clearer picture of the state of the art in this sector, and directions for future research and development. The focus on the individual learner, the shaping of the educational process to draw on the information about how well learners are progressing, and the engagement of learners in the learning process go well beyond techniques of “traditional” student assessment: it is about the whole approach to shaping teaching and learning. In short, formative assessment represents a fundamental shift in classroom relationships, and how instructors and learners work toward successful outcomes. 14 But what do we know about the impact of different teaching, learning and assessment practices for adult learners? How are innovative programmes addressing the very diverse needs and goals of adults with low foundation skills in language, literacy and numeracy? What can we learn from exemplary practices, and how can we get effective practices to happen on a wider basis? How do policies support or hinder effective practice? This study aims to address these questions by looking more closely into exemplary classroom practices, international research, and the broader policy environment. The approach of the study This study explores the state of the art in this field, drawing upon: x International research. x Innovative cases in a range of adult LLN learning settings, where teaching, learning and assessment have been adapted to meet the needs of this population. x Policies that support or inhibit effective practice. The research reviews The international reviews of literature from different linguistic traditions (English, French, German and Spanish) examine the conceptualisation of formative assessment and related teaching strategies in adult LLN education, the breadth and quality of evidence in the field, and what is known about the impact of specific approaches and techniques for different learners. The four reviews commissioned for this study identified a range of literature, including: x Guidelines and principles for teaching and learning based on practitioner wisdom and experience. x Debates on how values and goals for adult LLN learning – such as learner self-determination and autonomy, or democratic learning environments – should shape classroom relationships and approaches. x Interviews and surveys reflecting learner and instructor views on effective teaching and learning. x Case studies and classroom observations. x Evaluations and monitoring of specific assessment tools, including portfolios, recognition of prior learning, ICT programmes. It is important to note that the existing research in this sector does not include a significant number of controlled studies measuring the impact of specific formative approaches on adult LLN learning. In the compulsory sector, studies demonstrating the impact of formative assessment on achievement have been vital for garnering the support of policy officials as well as practitioners. The reviewers contributing to this study hypothesised that the absence of research specifically on formative assessment or related approaches for adult LLN learners (including the absence of controlled studies) is the result of several factors, including: x The compensatory mission of adult LLN education, which is seen as being incompatible with the selection process associated with assessment at the end of the learning process. Researchers pay 15 little attention to the impact of assessment, including alternative forms of assessment (Derrick and Ecclestone, Annex 3 on the Internet; Grotlüschen and Bonna, Annex 3 on the Internet). x Awareness of the fear and frustration experienced by many low achievers, leading to a certain over-protectiveness, and hesitation in regard to testing and validating innovative methods (Grotlüschen and Bonna, Annex 3 on the Internet). x The assumption that research from compulsory settings on issues such as motivation and feedback can be easily transferred to the adult LLN setting (Grotlüschen and Bonna, Annex 3 on the Internet). This wholesale adoption of school models developed for school-age learners, our reviewers counter, has stifled research as well as the development of interactive models of teaching and learning for adult LLN learners (Sanz, Annex 3 on the Internet). x A focus on advocacy. Funding and institutional instability in the adult LLN sector have meant that much of the literature has focused on advocacy, and arguing for the benefits of adult learning (including equity, access, and so on) rather than pedagogy (Derrick and Ecclestone, Annex 3 on the Internet). x Limited research on specific pedagogies for adult LLN education within the workplace as well. Instead, this literature has tended to focus on issues such as learners’ attitudes, motivation to participate, competencies for work, professional development for instructors, or resistance within the workplace to the development of foundation level training (Daniau and Bélanger, Annex 3 on the Internet). While much of the research on adult LLN teaching, learning and assessment is descriptive, it is nevertheless important for clarifying concepts and concerns, and providing insights into the nature and quality of interactions in learning settings (something that studies conducted in controlled environments cannot necessarily achieve). It can provide a valuable foundation for future research examining impact of different approaches on learner progression and autonomy. The case study sites The OECD case studies of exemplary practice include programmes offering primary or secondary school certification; community-based programmes not providing formal certification; programmes for immigrant and refugee learners; and, a prison-based programme. First steps: Diagnosing learning needs and setting goals When adult LLN learners first enter a new programme, they very typically have an initial interview to discuss their goals and motivations for learning, and how they expect they will use new skills in their daily lives. This first meeting may also include a diagnostic assessment to identify learners strengths and any potential barriers to learning (such as a disability), and to place them at an appropriate level. As a followup instructors and new learners set out goals for learning in a written document (an individual learning plan or contract). These first steps in the learning journey set the tone and direction for learning. Relationships within the classroom: Dialogue and peer assessment The second step of the OECD staircase highlights the centrality of relationships within the classroom, including dialogue and peer assessment in adult LLN learning. The following five elements emerge from the OECD case studies and reviews of the international literature as important for effective dialogue in adult LLN settings: 16 1. Building rapport and creating a “safe” environment 2. Using dialogue to promote participatory and democratic learning 3. Negotiating learning goals and methods 4. Structuring dialogue to meet specific learning goals 5. Using dialogue to establish what learners do and do not know and to adjust teaching to meet identified learning needs. Together, the five elements help to ensure that dialogue enriches the learning process. Techniques: Feedback, questioning, and scaffolding The third step in the OECD staircase explores the formative assessment techniques of feedback, questioning and scaffolding in the adult LLN context. These techniques are used to uncover learner understanding, to help instructors pitch learning to the right level, and to help learners progressively improve their skills. Research drawing on controlled studies at school and university levels points to the significant learning gains associated with these techniques. The lack of research featuring controlled studies in the adult LLN sector, on the other hand, has meant that little is known about the impact of different formative assessment techniques, or how they should be adjusted to meet the needs of learners in this population. Developing learner autonomy The fourth step in the OECD staircase explores the development of learner autonomy. Efforts to build autonomy begin with learners’ partnership in the assessment and learning process. Instructors in the OECD case study sites emphasised the importance of helping learners to “own” the assessment and learning process, and to develop the confidence to use their own judgment regarding the quality of their work, and for identifying gaps in their learning. These skills are also important as learners engage in non-formal and informal learning. Recognition of learner progress Recognition of learner progress is the final of the five steps for learning in the OECD staircase – although of course not the final step in the learning journey. The adult LLN programmes featured in the study place the focus on “measuring the distance travelled” throughout the learning journey. 17 CONCLUSIONS: Across countries, policy makers have responded to data on on adult skills in populations and the dimensions of the challenge with significant new investments and new programmes. New policies have been important for increasing the transparency, coherence as well as the status of the adult LLN sector. At the same time, investments are still insufficient to meet needs, and it is not clear that even current levels of funding will be sustained over time. These concerns are compounded by a lack of information on how effectively these new systems are promoting adult LLN learning. Data on outcomes other than certification are rare, and as we have mentioned previously, there is little attention to how data can be used to improve programmes and better meet learner needs. To address these gaps, our study has examined exemplary teaching, learning and assessment processes that lead to high quality outcomes. It has considered how policy can better promote and support effective practice across the sector. The study’s initial focus on formative assessment has also been useful for considering how individuals as well as systems learn – from what works, as well as from what doesn’t work. There are seven interrelated areas where policy could do more to help strengthen and develop effective practices and improve outcomes. 1. Promote active debate on the nature of teaching, learning and assessment Countries have developed several innovative approaches to adult LLN provision – including diagnostic assessment of learner needs, individual learning plans, and tracking tools to measure progress toward goals. Instructors in the exemplary programmes featured in the OECD study have built on these elements to transform practice. The process of transformation within programmes has involved debates and reflections on the nature of teaching, learning and assessment itself. Policy makers can promote wider debate on such fundamental concepts across the sector. Active debate invites exchange among stakeholders, critical analysis of evidence and ideas, and innovation. It moves beyond simple dissemination of tools or sharing of best practices, and places a stronger focus on transformation of teaching and cultures of learning. This process is essential if ideas are to gain any real traction in the sector. 2. Strengthen professionalism Effective teaching, learning and assessment centre on the quality of interactions between and among instructors and learners. High quality interactions involve a complex mix of skills in assessment, subjectmatter and pedagogical expertise, and softer skills such as humour, patience, flexibility and empathy. How can systems better support instructors in developing this sophisticated set of skills? Instructors will need opportunities for effective training and professional development, as well as the ongoing support of programme leaders and peer networks. As instructors develop their skills, they develop the capacity to think and act reflectively – diagnosing needs and developing appropriate responses. 18 A strong focus on building learners’ higher level skills is also vital if teaching and learning are to move beyond mechanistic approaches, where instructors focus on the technical aspects of assessment. Instructors need to challenge learners, and to ensure that they are genuinely involved in the process of learning and assessment. Instructors also need to identify their own values and goals for teaching and learning, to take “ownership” of new approaches, and to pay attention to the impact they are having on learning. With these challenges in mind, countries will need to continue in current direction of strengthening practice through more rigorous qualification and professional development requirements. In many countries, there are efforts to “catch-up” with the compulsory sector. Most training targeted to adult LLN instructors currently occurs through professional development, rather than in initial training. This is important and should be continued. However, given the ad hoc and short-term nature of professional development, it is not enough. There should be a range of professional development approaches. In addition, pathways – mirroring systems developed for LLN learners – would allow instructors to deepen their skills in progressively advanced courses. Instructors also need opportunities for peer support. The fact that many instructors in this sector work on a part-time basis makes it difficult to schedule time to meet and exchange ideas. Networks (both virtual and physical) provide opportunities for programme leaders and instructors to share experiences and insights and to raise questions with a large number of peers. But these networks should augment, rather than replace programme based exchange and support. 3. Balance structure and flexibility: formative assessment as a framework Well-designed tools and guidelines provide much-needed structure and enable instructors to be more systematic in their practice. At the same time, instructors need plenty of flexibility in order to adapt to the teaching context as well as the needs of diverse learners. Ideally, formative assessment frameworks will provide the right balance of structure and flexibility. Instructors use the tools and techniques of formative assessment to uncover learner understanding and progress toward goals. They then adapt teaching to meet identified needs, drawing upon a repertoire of learning tasks and challenges to help learners address gaps. This interactive approach to assessment and adaptation of the teaching and learning process calls upon the instructor’s pedagogical and subject knowledge, but also calls for a great deal of creativity and flexibility. Nevertheless, poorly designed (or poorly implemented) guidelines and tools for formative assessment will provide more structure than flexibility, and will do little to advance learner autonomy or skills for learning-to-learn. Assessments may be seen as a way to track learners toward meeting summative targets (an iterative process), rather than engaging with learners to build skills, knowledge and understanding (an interactive process). Policies to help structure classroom practice might include: broadly defined learning objectives, tools that can be adapted to context (for example, community-based or work-based programmes), and guidelines that provide insight into the process as well as the principles of formative assessment. In addition, training and professional development for formative assessment will help instructors to develop effective skills for guiding classroom interactions. 4. Strengthen learner-centred approaches The countries and programmes participating in this study promote “learner-centred” approaches to adult LLN provision. While the approaches vary across countries and programmes, the consistent principles are: to ensure that learners’ needs are diagnosed and addressed, individual motivations, interests 19 and goals are incorporated into teaching, and that learners may choose whether or not to pursue qualifications. There is a strong emphasis across countries on tailoring of content to meet individual needs. Yet many aspects of adult LLN provision are still more oriented to the needs of systems than they are to those of learners. Some approaches that could help to strengthen the focus on learner needs include: instruction that fits learners’ patterns of participation, smoother pathways for progression, and portable portfolios as well as improved national databases to register learners’ progress toward goals over time and across programmes, and a stronger focus on ensuring that formative and summative assessments are grounded in real-life contexts related to the learner’s goals. 5. Diversify and deepen approaches to programme evaluation for accountability Increased attention to the adult LLN sector has also meant increased accountability for programme performance. Performance measurements are vital for understanding the quality of programme delivery, and ensuring that investment of public money is yielding results. Information gathered on programme performance can also be used to make improvements in delivery and teaching. But there are a number of challenges for assessment and evaluation systems in the adult LLN sector. At the system level, key stakeholders, including learners, community advocates, instructors, programme leaders, policy officials from different agencies, and employers may have very different views of what counts as success and how to measure it. There are also practical data needs. Do the data gathered provide the kind of information that is needed to know whether adult LLN systems are meeting goals? At the programme level, do the data include the necessary context and details to identify programme strengths, effective practice, and areas for improvement? Do accountability systems provide room for programmes to learn from what works as well as what doesn’t work? Current approaches to accountability provide little insight as to whether programmes are really helping to prepare learners for employment, community integration, or individual and family well-being. Nor do many of these systems provide any indication of whether learners have gained skills for selfassessment, the ability to transfer knowledge and skills across domains, or whether they are integrating new skills in their daily and work lives. Given the range of stakeholder interests, no single approach can satisfy all needs. Systems that use diverse, well-aligned measures of learning processes, as well as outcomes will be better able to manage competing goals and interests – and to capture useful data. The mix of approaches to measurement might draw upon current approaches, including:a focus on outcomes, targets, national reporting systems,inspections and programme self-evaluations. 6. Devote the necessary resources – people, time and money This study has not included any detailed discussion on funding within the adult LLN sector. However, it has been concerned with providing insight on effective practice in the hopes that this may guide more effective investment of existing resources – including people, time and money – and suggest where additional resources might help to further professionalise the field and improve outcomes. People Several countries and programme leaders have reported the problem of high turnover of instructional staff (although in Denmark, it is common for instructors to have taught in the same programme for at least ten years). In addition, several countries report that a large group of instructors are over the age of 50, and therefore nearing retirement age. Two major challenges, therefore, are to attract new instructors to the sector, and to improve conditions of employment in order to increase instructor retention. The timing is 20 opportune in terms of being able to reach a new generation of incoming instructors who will be entering this sector. However, it will take time for new qualification requirements to filter through in the sector. In addition, the kind of initial training and professional development referred to in earlier chapters (see particularly Chapter 4) is expensive. Conditions of employment will also need to be commensurate with instructors’ status as professionals. The adult LLN sector is still far from having conditions equivalent to those in the compulsory sector, although professionals in the two sectors are undertaking similar work. Countries need to closely examine pay and benefit conditions, as well as career structures for advancement (and development of skills as “master” instructors). The current over-reliance on voluntary or part-time instructors presents a serious barrier to development of effective practice. That said, a core volunteer/paraprofessional workforce should be supported and sustained with appropriate training and conditions of service. Having volunteers/paraprofessionals is a particularly good strategy in classrooms focused on tailoring provision for different learners. Volunteers can give more time and attention to individual learners, and also bring different points of view and knowledge in explaining new concepts and ideas. Time Many instructors note that formative assessment approaches such as tracking tools, individual sessions with learners and so on, are much more time intensive. Given the amount of paperwork instructors are often required to complete, formative assessment and related strategies may appear as an additional burden. Pragmatic solutions such as ICT-based learning programmes with effective feedback (a few programmes have been found to be particularly effective), effective deployment of volunteer tutors, peer learning and other approaches may help to relieve some of this burden. A close look at how to cut back on official paperwork may also help. Curricula may also reinforce the importance of helping learners to focus on developing a few core skills, and for developing skills for self-assessment and learning-to-learn. As learners progress, they can take on more responsibility for their own learning and self-assesment. In addition, research pointing to the impact of different formative assessment approaches and techniques may also help to convince instructors of the benefits resulting from the extra time invested. Money While, as noted above, we have not made a detailed review of funding structures for adult LLN provision, the country background reports prepared for this study have noted that there is significantly more funding in this sector than there has been in the past – and that it is still not enough to meet needs. 7. Strengthen the knowledge-base Our list of research needs in the adult LLN field is wide-ranging. It includes very specific recommendations related to promising teaching and assessment practices, as well as more general recommendations related to policy and implementation. The list is built on the assumption that researchers in this field will need to broaden the range of methodologies used, and in particular to pay much greater attention to impact. Researchers may also engage more actively with policy makers and practitioners. We have identified the following as priority areas: research on classroom relationships – which are at the very heart of teaching, learning and assessment; research on specific formative assessment techniques – including feedback, questioning and scaffolding; validated assessment instruments (and training for assessors); more research on how adult foundation skill learners progress from basic to more complex levels of knowledge and autonomy; evaluation of how well specific learning objectives are aligned with overall goals for learner progression, and autonomy; more research on learner persistence; more extensive evaluations of promising e-based programmes as a complement to traditional learning; broader surveys of current teaching, learning and assessment practices; research on how instructors take on and adapt new 21 practices to meet needs of learners; further investigation on the adequacy of funding and effectiveness of investment for improving practice and social and economic outcomes; further international research; and use of data gathered for accountability purposes to learn more about what works and why. Concluding remarks Our review of teaching, learning and assessment for adult LLN learners has revealed a rich array of policies and practices within and across countries. Countries have taken very different approaches to balancing competing needs and goals of stakeholders, and can learn a great deal from each other. Indeed, while policy approaches and cultures vary, many of the goals are the same. Policy support for the complex work of teaching, learning and assessment has been important for ensuring higher quality within this sector. But there is still work to do, both at the conceptual and empirical levels. The fact is that knowledge on the impact of the different approaches to teaching and assessment is limited in the adult LLN sector. Without better knowledge of what works (and what doesn’t work), it will be difficult to attract much-needed funding, or to develop effective practices across the sector. The good news is that the existing research base (including research on the generic methods of formative assessment, as well as practitioner wisdom) provides clear direction for future research and development. The stakes of not taking on this challenge will be high for communities as well as learners. 22 REFERENCES Ames, C. (1992), “Classrooms: Goals, Structures, and Student Motivation”, Journal of Educational Psychology, Vol. 84, pp. 261-271. Beaton, A.E. et al. (1996), Mathematics Achievement in the Middle School Years, Boston College, Boston, MA. Bishop, R. and T. Glynn (1999), “Culture Counts: Changing Power Relations in Education”, Dunmore Press, Palmerston North, New Zealand. Black P. and D. Wiliam (1998), “Assessment and Classroom Learning”, Assessment in Education: Principles, Policy and Practice, CARFAX, Oxfordshire, Vol. 5, No. 1, pp. 7-74. Bloom, B. et al. (1971), Handbook on Formative and Summative Evaluation of Student Learning, McGraw-Hill Book Co., New York. Boulet, M.M. et al. (1990), “Formative Evaluation Effects on Learning Music”, Journal of Educational Research, Vol. 84, pp. 119-125. Bransford, J.D. et al. (eds.) (1999), How People Learn: Brain, Mind, Experience, and School, National Academy of Sciences, National Academy Press, Washington D.C. Bruner, J. (1996), The Culture of Education, Harvard University Press, Cambridge, MA. Butler, D.L. and P.H. Winne (1995), “Feedback and Self-regulated Learning: A Theoretical Synthesis”, Review of Educational Research, Vol. 65, No. 3, pp. 245-281. Cameron, J. and D.P. Pierce (1994), “Reinforcement, Reward, and Instrinsic Motivation: A Metaanalysis”, Review of Educational Research, Vol. 64, pp. 363-423. Crooks, T.J. (1988), “The Impact of Classroom Evaluation Practices on Students”, Review of Educational Research, 58, pp. 438-481. EPPI – Centre at the Institute of Education, University of London (2002), “A Systematic Review of the Impact of Summative Assessment and Tests on Students’ Motivation for Learning”, June. Heckhausen, H. (1989), Motivation und Handeln, Springer, Berlin. Kluger, A.N. and A. DeNisi (1996), “The Effects of Feedback Interventions on Performance: A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory”, Psychological Bulletin, Vol. 119, pp. 254-284. Krampen, G. (1987), “Differential Effects of Teacher Comments”, Journal of Educational Psychology, Vol. 79, No. 2, pp. 137-146. 23 Lander, R. and M. Ekholm (1998), “School Evaluation and Improvement: A Scandinavian view” in A. Hargreaves, A. Lieberman, M. Fullan and D. Hopkins (eds.), International Handbook of Educational Change, Kluwer Academic Publishers, Dondrecht, the Netherlands, pp. 1119 1134. Mischo, C. and F. Rheinberg (1995), “Erziehungsziele von Lehrern und individuelle Bezugsnormen der Leistungsbewertung”, Zeitschrift für Pädagogische Psychologie, Vol. 9, pp. 139-151. Monsen, L.I. (2002), “School-based Evaluation in Norway: Why is it so Difficult to Convince Teachers of its Usefulness?” in D. Nevo (ed.), School-based Evaluation: An International Perspective, JAI Press, Oxford, pp. 73-88. Natriello, G. (1987), “The Impact of Evaluation Process on Students”, Educational Psychologist, 22, pp. 155-175. OECD (2001), Knowledge and Skills for Life, OECD, Paris. OECD (2002), Understanding the Brain: Towards a New Learning Science, OECD, Paris. OECD (2003), Learners for Life: Student Approaches to Learning: Results from PISA 2000, OECD, Paris. Pajares, F. (1996), “Self-efficacy Beliefs in Academic Settings”, Review of Educational Research, Vol. 66, pp. 543-578. Perrenoud, P. (1998), “From Formative Evaluation to a Controlled Regulation of Learning Processes. Towards a Wider Conceptual Field”, Assessment in Education: Principles, Policy and Practice, CARFAX, Oxfordshire, Vol. 5, No. 1, pp. 85-102. Rheinberg, F. and S. Krug (1999), Motivationsförderung im Schulalltag (2. Auflg.), Hogrefe, Göttingen, Germany. Schunk, D.H. (1996), “Goal and Self-evaluative Influences during Children’s Cognitive Skill Learning”, American Educational Research Journal, 33, pp. 359-382. Simmons, H. (2002), “School Self-evaluation in a Democracy”, School-Based Evaluation in D. Nevo (ed.), School-based Evaluation: An International Perspective, JAI Press, Oxford, pp. 17-34. 24