Teaching Statistics ? Despite Its Applications

CURRICULUM CORNER Teaching Statistics – Despite Its Applications XX 2CURRICULUM 29 CORNER © 0141-982X Teaching TEST Blackwell Oxford, Teaching UK Statistics Publishing Statistics Ltd Trust 2007 Jim Ridgway, James Nicholson and Sean McCusker University of Durham, England. e-mail: [email protected] [email protected] [email protected] Summary Evidence-based policy requires sophisticated modelling and reasoning about complex social data. The current UK statistics curricula do not equip tomorrow’s citizens to understand such reasoning. We advocate radical curriculum reform, designed to require students to reason from complex data. 䉬 INTRODUCTION 䉬 Statistics . . . the most important science in the whole world: for upon it depends the practical application of every other science and of every art; the one science essential to all political and social administration, all education . . . –Florence Nightingale Things should be made as simple as possible, but not any simpler. –Albert Einstein he role of statistics within the curriculum generally, and within mathematics in particular, has generated considerable debate in the UK. The Smith Report (2004) has recommended that the community considers ways by which aspects of the teaching of statistics can be taken out of mathematics, and be taught as an integral part of the curriculum in other subject disciplines. This approach has a poor track record; it has been applied to ICT and (to a large extent) to citizenship, with rather little success. All subjects feel they have too little dedicated curriculum time, and many teachers view ‘crosscurricular’ themes as a burden rather than as an opportunity to enhance their subject. In statistics, the problems run far deeper than elsewhere. We can have little confidence that non-mathematicians have the requisite knowledge to teach statistics T 44 • Teaching Statistics. Volume 29, Number 2, Summer 2007 (one also might argue that mathematicians whose background in statistics is weak are also illprepared to teach statistics, or to design statistics curricula). We know little about the ways that statistical concepts develop. Even were we to have such knowledge, there would be considerable logistical problems in providing a coherent set of experiences for students in statistics, if teaching takes place in a number of different curriculum areas (see, for example Holmes (2000), and the forthcoming survey commissioned by the UK Qualifications and Curriculum Authority from the RSS Centre for Statistical Education). These arguments are elaborated in a recent Royal Statistical Society Report (2005). The locus of statistics in the curriculum is important, but the nature of the statistics curriculum is also important. Here, we provide evidence that the current statistics curriculum provides a very poor basis for understanding evidence-based arguments in social policy arenas (and indeed, is of very limited use in almost any situation where reasoning with evidence might be appropriate), and is urgently in need of a radical review. A broadly based discussion is needed to help resolve the problems we identify here, and we hope this article will contribute to this discussion. © 2007 The Authors Journal compilation © 2007 Teaching Statistics Trust Statistics is a branch of mathematical modelling that was invented, and continues to be created, in response to practical problems (see Gigerenzer et al. 1989 for a historical account). A challenge that modern democracies face is to adopt policies that address complex social problems, for example those that arise in areas such as health, crime and social renewal, and also to describe policy and social action in ways that citizens can understand and be sympathetic towards. This must be part of any attempt to develop ‘statistical literacy’. A starting point for the article, therefore, is to explore the sorts of implicit models that (some) politicians seek to use to better understand, and to act on, complex social situations. These implicit models give pointers to the sorts of formal methods of evidence gathering and modelling that are likely to be employed by government, and, therefore, to the pressures on the emerging discipline of statistics and to the problems of communicating complex ideas to a wider public. 䉬 MODELS IMPLICIT IN A 䉬 POLITICAL SPEECH In this section, we set out some of the ideas about modelling complex phenomena that are implicit in one speech by a politician who was widely regarded at the time as one of the most thoughtful around. In the next section, we explore high-stakes statistics examinations in the UK that are used to determine access to higher education, to see where and how these ideas are exemplified and examined in current curricula. The purpose of this analysis is to examine the match between the sophisticated arguments about evidence and action presented in political speeches that citizens are supposed to understand, and the statistical modelling that students are actually exposed to in the UK national curriculum. The speech was chosen opportunistically; it was given by David Blunkett, then the Home Secretary, to the Ash Institute in the USA (Blunkett 2004). Extracts from the minister’s speech are presented in the correct order. They are edited to make points about modelling that citizens should understand, not to reproduce specific arguments about what should be done – so the section is about the nature of modelling social phenomena, not about the qualities of one particular model. Extracts from the speech are presented in a small font. A fuller discussion of the speech and the issues it raises can be found at http://www.durham.ac.uk/smart.centre. © 2007 The Authors Journal compilation © 2007 Teaching Statistics Trust ‘[How can] democracy . . . be revitalised and renewed in an era of global capital, the World Wide Web, mobile phones and multichannel media[?].’ Society is influenced by multiple factors. ‘. . . the challenge is one of the most enormous change both in terms of scale but also rapidity.’ Effects occur on different scales of impact, and over different time lines. ‘I want therefore to explore for a moment what it is that determines the level of social capital in a society.’ Politicians need models of causality in order to be effective. ‘But of course Government should not be a purely passive player in this . . . more important still for long-term change is our approach to education – which brings not just personal strength, hope and the capacity to cope with change, but also gives people a stake in society.’ Government should act to bring about desirable social change. Psychological variables are important causal factors in social behaviour. Education has multiple goals (personal strength, hope, capacity to cope with change, giving a stake in society). ‘. . . the evidence that [cultural] diversity is correlated with a decline in social capital is . . . powerful. . . . I am convinced that instability through high mobility and therefore turnover of population is a central factor in contributing to the decline in social capital.’ Moderator variables need to be considered (here, high mobility and turnover of population might be the underlying reason (i.e. be moderator variables) for the association between cultural diversity and lack of social capital). The speech reveals sophisticated thinking about the use of evidence and modelling in tackling complex problems. A short list of principles can be deduced: • Every complex problem has a number of components, which are influenced by a variety of factors. Effects occur over a range of time scales, and at different magnitudes • Social variables can be measured in a variety of different ways (and almost never in just one way) • Models of social change need feedback loops • Politicians need robust models of causality in order to be effective. Correlation is not the same as causality. Possible moderator variables need to be considered • Where causality is demonstrated, we should look for ways to undo causal relationships which have negative consequences, and induce more powerful positive social effects. Government is not simply Teaching Statistics. Volume 29, Number 2, Summer 2007 • 45 about maintaining ‘what is’, but is also about supporting the development of desirable possible futures. Citizens need to understand these principles, if they are to understand political speeches, and if they are to use complex data to inform their decision making. If students are to become responsible citizens, they need to be exposed to these ideas in the curriculum. This implies (at least) that their experiences include: • Exposure to multivariate problems: • Exposure to non-linear relationships between variables • Working with time lags between causes and effects • Seeing different effect sizes • Analysis of possible causal links between variables, and an understanding of moderator variables • Linking data analysis to the situations from which the data were derived • Speculation about possible courses of action beyond the ones considered. HOW DOES THE CURRENT CURRICULUM EQUIP 䉬 STUDENTS TO REASON 䉬 WITH EVIDENCE? In the UK, advanced-level subjects are available for students in post-compulsory education, i.e. those aged 16 to 18 years, and are used to determine access to higher education as well as being qualifications for direct entry into the labour market. Students choose a range of specialized courses. Mathematics is one of the three most popular subjects (along with English and General Studies). Examinations are set by a number of different agencies (‘awarding bodies’) who define a curriculum and design appropriate examinations. Schools have some choice in the curriculum they follow, and there is no single curriculum. Data for the year 2003 from the UK’s largest Awarding Body (Edexcel) show that more than 85% of mathematics students took at least one statistics module, and that more than 50% of students took two statistics modules. Assessment systems have an important role in exemplifying curriculum goals, and have a profound effect as drivers of the curriculum. As part of this exemplification, awarding bodies publish descriptions of the curriculum and all the examination 46 • Teaching Statistics. Volume 29, Number 2, Summer 2007 papers, initially as exemplar materials (called ‘specimen assessment materials’), then (once live examinations are in place) every examination paper is published after it has been taken by students. We set out to analyse the examination questions of these statistics modules, in order to understand the curriculum to be ‘attained’ (e.g. Robitaille et al. 1993). Two raters independently analysed the 2004 specimen assessment materials used to assess statistical knowledge by each of the six awarding bodies in every A-level statistics examination (including the AS examinations typically taken at age 17 years). Raters were asked to categorize the questions according to whether they assessed computational or procedural knowledge, or could be broadly classified as modelling or interpretation. The latter category included: identifying the assumptions which underlie certain models (e.g. use of the Poisson distribution); making a choice about which probability distribution to use in a particular context; interpreting the conclusions of a hypothesis test in context; and all tasks where students are required to reason from data. Conversely, if a question explicitly instructs the student to use the Poisson distribution (for example), the candidate does not have to make a choice about the distribution as part of modelling the situation, and so such a question would be judged to be assessing statistical technique (supposing there were no interpretive element later in the question). An analysis of 11 exemplar questions can be found at http://www.durham.ac.uk/smart.centre. There was excellent inter-rater agreement; on no paper did their judgements on the proportion of marks in each category differ by more than a single mark. Differences were resolved successfully by discussion. The proportions of marks awarded for the broad interpretative categorization on every examination paper set by every examination authority are shown in table 1. The column headings S1 to S4 refer to different examinations, with S3 requiring knowledge of S1 and S2, and so on. It can be seen that the majority of marks on every examination paper are allocated for demonstrating computational and procedural skill, not conceptual understanding or interpretation. Raters were also asked to identify questions where students were required to work with three or more variables; none were found (this is important for the current argument, but unsurprising, given that the curriculum specifications do not require work with three or more variables). Raters were also asked to identify questions where students were faced with data © 2007 The Authors Journal compilation © 2007 Teaching Statistics Trust AQA OCR CCEA Edexcel WJEC OCR (MEI) S1 S2 S3 S4 25 24 12 27 3 25 20 29 33 25 28 31 25 29 23 14 33 21 28 22 AQA is the Assessment and Qualifications Alliance; OCR is Oxford Cambridge and RSA Examinations; CCEA is the Northern Ireland Council for the Curriculum, Examinations and Assessment; WJEC is the Welsh Joint Education Committee; OCR (MEI) is the Mathematics in Education and Industry project, with OCR the awarding body who accredit this specification. Table 1. Percentages of marks available for modelling and interpretation in statistics modules in A-level mathematics where the relationship between two variables was non-linear. Not a single example was found on any examination paper. An unsatisfactory pattern emerges from this analysis. Statistical education focuses on a rather narrow range of techniques applicable only to univariate and bivariate analyses; students are only exposed to linear relationships between variables. Students are rarely required to engage in modelling, or to interpret data. Some curriculum developments have taken place recently. New specifications in statistics are offered by AQA in a full A-level, and by MEI at AS-level only, which share a common first unit with their respective mathematics specifications. These place more emphasis on interpretation and less on purely computational proficiency, but the content is still largely dominated by traditional topics. GCSE Statistics (a high-stakes examination taken at age 16 years) requires students to perform simple transformations on non-linear data (inverse, square root and quadratic) so that linear models can be applied. These are changes for the better but do little to address the central concerns of this paper. It is almost certain that students in some programmes will enjoy a curriculum rich in statistical ideas such as discussions on measurement, appropriate sampling, generalizing from the data to hand, and modelling; however, the exemplification of the curriculum provided by the specimen papers suggests that the majority of students will not. When the role of education to prepare students for informed citizenship is considered, one is hardpressed to show where in the National Curriculum students acquire the skills necessary to understand policy speeches. The analysis reported here concerns only high-attaining students in post-compulsory © 2007 The Authors Journal compilation © 2007 Teaching Statistics Trust education who have chosen to take courses in statistics. The picture for students in other phases of education, and with other choices of course, is almost certain to be even less rosy. We suspect that neither Florence Nightingale nor Albert Einstein would be impressed by our current situation. REASONING WITH DATA 䉬 THROUGH THE CURRICULUM 䉬 There has been little tradition of data interpretation within the mathematics curriculum; the main emphasis has been on mechanical skills. We have little subject-specific pedagogic knowledge to guide our teaching – such as a sensible hierarchical structure of statistical concepts, which allows us to plan the development of pupils’ interpretative skills. Exposure to rich data sets, and to graphs that have something to say which is both accessible and of substance, offers a way forward. On an optimistic note, ICT can actually facilitate the development of new cognitive processes; we need to develop a pedagogy that capitalizes on the new affordances of ICT (see Ridgway and McCusker 2003). The curriculum in subjects other than mathematics generally highlights opportunities where data could be used, but stops short of a data-based approach (see Holmes 2000 for details). Any move to make a data-based approach compulsory would require considerable professional development before it could be made to work in representative classrooms. 䉬 DEVELOPING A BETTER 䉬 CURRICULUM We believe that the current statistics curriculum is inadequate to equip students to become informed citizens in the 21st century, where they will be required to reason from complex data sets. Indeed, the current situation poses a threat to democracy. If politicians claim to base policy on evidence, and use models that require some sophisticated statistical understanding, and if citizens are unable to understand these arguments, then a gap can emerge between the government and the governed. Clearly, the task of modelling social phenomena is inappropriate for most school children. However, the quantum jump in reasoning from evidence is that between problems involving two variables which are linearly related, to situations with three or Teaching Statistics. Volume 29, Number 2, Summer 2007 • 47 more variables with non-linear relations and interactions between variables. There is an urgent need to make space in the curriculum for reasoning from evidence in realistic situations, using models that represent the complexity of the situations they purport to describe. There is some evidence that primary school children can reason using three or more variables including non-linear relationships, given appropriate contexts and technologies that remove the technical demands of exploring the data. The recommendations of the Smith Report (2004) concerning the relationship of statistics to the mathematics curriculum, and the counter arguments in the RSS Report (2005), both miss the essential theme that there is an urgent need for a radical reform of the content of statistics education, which should permeate the entire curriculum via an emphasis on ‘reasoning from evidence’. du Feu (2005) reported that children as young as 5 and 6 years old can build concrete representations of data, using Lego, and can articulate relationships present in what was essentially a 3 × 2 contingency table. There is good evidence that appropriate ICT support can make the understanding of data far easier for quite young students, for example by presenting new sorts of tasks, where dynamic displays show changes in several variables over time. Interaction makes computers well suited to the presentation of tasks where students are required to discover rules and find relationships. Experience with the World Class Arena (for sample tasks see http://www.worldclassarena.org/) has shown that students can work with complex realistic data sets, using methods similar to those used by professionals in the field (Ridgway and McCusker 2003). This research is in an early stage; we have begun to explore both the qualitative and quantitative aspects of this phenomenon, as we create new computer interfaces. Other examples of appropriate curriculum materials can be found in Nicholson et al. (2006). References Blunkett, D. (2004). Renewing Democracy: Why Government Must Invest in Civil Renewal. A speech by Home Secretary David Blunkett to the Ash Institute, Boston, USA on 9 March 2004. Available at http://www.homeoffice. gov.uk/documents/Civil_Renewal.pdf ?view= Binary. du Feu, C. (2005). Bluebells and bias, stitchwort and statistics. Teaching Statistics, 27(2), 34– 6. Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J. and Kruger, L. (1989). The Empire of Chance. New York: Cambridge University Press. Holmes, P. (2000). Statistics Across the English National Curriculum. Royal Statistical Society Centre for Statistical Education. Available at http://www.rsscse.org.uk/resources/ natcur.html. Nicholson, J.R., Ridgway, J. and McCusker, S. (2006). Reasoning with data – time for a rethink? Teaching Statistics, 28(1), 2–9. Ridgway, J. and McCusker, S. (2003). Using computers to assess new educational goals. Assessment in Education, 10(3), 309–28. Royal Statistical Society. (2005). Teaching Statistics across the 14–19 Curriculum. London: Royal Statistical Society. Robitaille, D.F., Schmidt, W.H., Raizen, S., McKnight, C., Britton, E. and Nicol, C. (1993). Curriculum Frameworks for Mathematics and Science. TIMSS Monograph No. 1. Vancouver: Pacific Educational Press. Smith, A.F.M. (2004). Making Mathematics Count. London: HMSO. 䉬 CONCLUSIONS 䉬 Statistics education in the UK falls far short of the ambitions of Nightingale and Einstein. The simplifications made in statistics courses go against Einstein’s advice, and make things too simple. Realistic multivariate problems (which students can actually solve) are ignored, and the statistics curriculum is reduced to practising algorithms out of context, for which the range of applications is limited. ICT can be used to present realistic, multidimensional data to students. 48 • Teaching Statistics. Volume 29, Number 2, Summer 2007 © 2007 The Authors Journal compilation © 2007 Teaching Statistics Trust