EQAO: Ontarios Provincial Assessment Program

Its History and Influence

1996 2012

EQAOs Board of Directors

EQAOs Executive Team

Dr. Brian L. Desbiens, Chair June 1, 2005March 16, 2014

Marguerite Jackson, Chief Executive Officer

Jerry Ponikvar, Vice-Chair July 4, 1997May 16, 2014

Marianne Mazzorato, Chief Assessment Officer

Roland Boudreau June 2, 2011June 1, 2014

Richard Jones, Director, Assessment and Reporting

Hlne Chayer June 25, 2008May 11, 2013

Michael Kozlow, Director, Data and Support Services

Dave Cooke June 25, 2008May 11, 2013

Franois Lavictoire, Executive Coordinator, French Language

Dr. Dieudonn Detchou January 13, 2010January 12, 2013

Mary Low, Executive Coordinator, Operations

Dr. Elizabeth (Lee) Ford-Jones June 2, 2011June 1, 2014

Tony Saini, Director, Corporate and Public Affairs (Acting)

Abirami Jeyaratnam October 3, 2012October 2, 2014

Dr. Bette M. Stephenson, Director Emeritus

EQAO: Ontarios Provincial Assessment Program

Its History and Influence

1996 2012

2013 Queens Printer for Ontario, ISBN 978-1-4606-1449-5 (Online) I C15ann_ne_0313 Education Quality and Accountability Office I 2 Carlton Street, Suite 1200, Toronto ON M5B 2M9 I Telephone: 1-888-327-7377 I Web site: www.eqao.com

Contents
Introduction
Background Early Years (19962004) I Developing Effective Assessment Programs Evolution (20052012) I Refining Effective Assessment Programs Program Enhancements 10 12

3
4 6 8

Ensuring Best-of-Class Assessments Frameworks 13 13

Item Development Field Testing 13

Test Construction Questionnaires

14 14 14

Test Administration Scoring 15

Analyzing and Equating Student Scores and Accuracy of Assessment Results Reporting 17 18

16

Sharing Expertise Through Partnerships and Outreach Empowering with Good Information Public Support for Provincial Testing Educator Use of EQAO Data 22 23 21

20

School Board Use of EQAO Data Provincial Use of EQAO Data Researchers Use of Data

24

24 25 26

Improvements in Student Achievement Looking Ahead Summary 27

References

28

Timeline: Education Quality and Student Success Initiatives 2012 2011 2010 2009
Full release of EQAO reporting for elementary principals EQAO reports results for students tracked from Grade 3 to Grade 6 to secondary school Interactive Resource to Inform Leadership Action Auditor General confirms quality, reliability and value-formoney of EQAOs assessment program; School and board profile Web pages

2008 2007

Report on the Evaluation of the Student Success/Learning to 18 Strategy EQAO reports results for students tracked from Grade 3 to Grade 6 EQAOs ready-to-use school results reports; Ensuring Quality Assessment Review: The Move Forward; Establishment of the Literacy and Numeracy Secretariat First full-census administration of the OSSLT First full-census assessment of Grade 9 mathematics Trial administration of the OSSLT First full-census assessment of Grade 3; Sample assessment of Grade 6 mathematics EQAO established Report and recommendations from the Royal Commission on Learning Royal Commission on Learning established

2004
2002 2001 2000 1997

1996
1995

1993

The Education Quality and Accountability Office (EQAO) began its work in 1996, and over the past decade and a half it has distinguished itself as a world-class large-scale testing organization that provides valuable services to the people of Ontario with a focus on improving student learning. Given that 2012 brings with it the 15th anniversary of the first EQAO tests, it is timely to reflect on and describe the evolution of the work of the agency over the years, the knowledge that has been gained from its testing and research programs, its impact as well as its future directions. As with any dynamic organization, EQAO is committed to a continual review and renewal of its initiatives, resources and tasks to ensure it meets its strategic priorities. The timeline on page 2 provides an overview of the agencys key refinements that support improving student learning.

Introduction 3

Background

The Ontario government, with all-party support, established EQAO as a Crown agency through the Education Quality and Accountability Act of 1996 in response to recommendations from the 1994 Royal Commission on Learning. The Commission concluded that province-wide testing of all students was required to provide independent and public scrutiny of the education system (53). The main focus was to monitor students achievement at key points in their learning as a way of assuring the public that all students were being assessed in the same way and according to an established set of standards. The government also wanted to respond to the publics demand for more accountability and clearer and better information about the quality and effectiveness of the provinces publicly funded education system.

It seems obvious to us that the public school system is responsible to the public, and owes to the public to demonstrate how well its doing with our childrenSo we take a stand on behalf of close monitoring of every childs progress right from the earliest years, and of the system itself at every level, in order that both can learn to be even better.
Ontarios Royal Commission on Learning

EQAO is an agency of the Ontario government, led by a Chief Executive Officer and governed by a board of directors. Its mandate is to support and guide student improvement by providing credible information about the quality of the provinces publicly funded elementary and secondary education system. It does this by developing, administering, scoring and reporting on province-wide tests of elementary and secondary school students achievement in reading, writing and mathematics in relation to Ontario Curriculum learning expectations. The agency also manages and reports on the provinces participation in national and international testing programs.

Background 5

Early Years
19962004 I Developing Effective Assessment Programs

Prior to the implementation of its assessment program, EQAO consulted widely on elements of the programs design as well as the assessments purposes. This process involved feedback sessions with stakeholder groups, including Ontario educators and the public, and extensive commissioned research on large-scale testing programs in North America and around the world. A requirement of the resulting assessments was that they clearly link to The Ontario Curriculum expectations students are expected to meet at key stages of their education. The first EQAO large-scale educational assessment was of reading, writing and mathematics expectations for Grade 3 (i.e., at the end of the primary division) and was conducted during the 19961997 school year. All Grade 3 Ontario students participated in the assessment made up of performance tasks related to a particular theme (e.g., whales), which required approximately 12 hours of testing over two weeks. Over the years, EQAO added assessments to develop an annually administered assessment program. The Grade 6 Assessment of Reading, Writing and Mathematics (administered at the end of the junior division) was introduced in 19981999 and required the same amount of class time as the Grade 3 test; the first Grade 9 Assessment of Mathematics (administered in the first year of secondary school) was conducted in 20002001; and the first Ontario Secondary School Literacy Test (OSSLT) (a literacy graduation requirement) was administered in 2002. Each of the secondary school assessments required about five hours of testing over two days. Different versions of the Grade 9 assessment were developed for students taking the applied and academic mathematics courses, based on the relevant curriculum expectations. In addition, since some students took their Grade 9 mathematics course in the first semester, while others took their course in the second semester or over the full year, two test administrations were held each school year,

one in the winter and one in the spring. The OSSLT was designed to be administered over two half-days. The first administration took place in February 2002; subsequent tests were administered annually in October.

Since the agency began, the following key principles have formed the foundation of its work.

The assessments are curriculum-based; they are designed to measure student achievement of Ontario Curriculum learning expectations. The assessments are developed separately in Canadas two official languages (English and French); attention is given to developing Englishand French-language assessments that are parallel and of equal, high quality. Educators are involved in all aspects of assessments (e.g., development, administration, scoring) to ensure they are appropriate for provincial use, relevant to the curriculum, and build the professional capacity of educators in curriculum and assessment. All students are expected to participate in the assessments (i.e., they are census assessments), so that all students are accounted for and reports are available not only at the provincial, but also at the school board, school and individual student levels for reflection and improvement purposes. Student achievement information is comparable year over year in order to track performance over time. The performance nature of the assessments is maintained by including open-response as well as multiple-choice items (questions), so that students have a variety of ways to demonstrate what they know and can do. Being of service is an EQAO core value, which the agency can fulfill only if there is clear, ongoing communication and collaboration with all stakeholders, including students, parents, educators (at all levels) and the Ontario government.

Early Years (19962004): Developing Effective Assessment Programs

Evolution
20052012 I Refining Effective Assessment Programs

In order to ensure continual success, it is important for any organization to reflect periodically on the effectiveness of its practices. In November 2002, five years after the administration of its first assessment, EQAOs board of directors launched the Ensuring Quality Assessments review. The purpose of this initiative was to conduct a comprehensive review of all aspects of the EQAO assessment programs to confirm that current international standards of best practice in large-scale assessment were being met, and that the agencys reporting satisfied user needs for accountability, improvement planning and staff development.

The review process had three main components:

A study of best practices (processes and procedures) used by exemplary testing organizations and jurisdictions worldwide, conducted by EQAO staff. An external review of EQAOs assessment processes conducted by a team of experts from the Ontario Institute for Studies in Education of the University of Toronto, who worked with a team of internationally recognized large-scale assessment specialists. Consultation with stakeholders, including the agencys Assessment Advisory Committee, and representatives of groups such as directors of education, supervisory officers, principals, teachers, school trustees, parents and students. Through a series of dialogue forums, information was gathered about the assessments impact in schools and school boards, the usefulness of EQAO data reports for accountability and improvement planning and the impact of the assessments on educator staff development.

EQAO responded to the key findings and implemented the initial refinements to its assessment programs, protocols and procedures within a very focused time frame in order to optimize the positive impact of these changes.

Evolution (20052012): Refining Effective Assessment Programs

Program Enhancements
EQAO considered all of the information gathered from the review, with a particular focus on the feasibility of various test-design options. The review confirmed that many of EQAOs practices were sound, and it found that the implementation of some refinements would make the assessments more helpful and less burdensome for schools. The framework document that underpins each assessment was updated to identify the purpose of the assessment and its links to The Ontario Curriculum more specifically. The enhanced test designs reduced the amount of class time for test administration and continued to provide quality data and useful feedback on student performance.

Brief Overview of the Redesigned Grades 3, 6 and 9 Assessments and the OSSLT
For the Grade 3 and Grade 6 assessments of reading, writing and mathematics, there are three test booklets, two for language (reading and writing) and one for mathematics. The amount of testing time has been reduced by half, to six hours (or two hours per booklet). The name of the assessment program has been changed to the Assessments of Reading, Writing and Mathematics, Primary Division (Grades 13) and Junior Division (Grades 46) in recognition that the entire primary and junior divisions are responsible for students learning progress. For the Grade 9 Assessment of Mathematics, there are two test booklets, and the total amount of testing time was reduced by half to two hours (two 60-minute time blocks). For the OSSLT, there are two test booklets, and the total amount of testing time has been reduced by half, to two and a half hours (two 75-minute blocks) administered on one day. In addition, the administration has been moved from the fall to the spring of the school year.

10 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

Principles applied to all assessments

All-student testing, as opposed to sample testing, was reaffirmed. Each test booklet contains common items (questions) that all students attempt, which count toward their results, and a small number of matrix items used for field testing and year-to-year comparisons (equating), which do not count toward students results. Open-response and multiple-choice items are presented throughout the tests. Results are reported as overall achievement levels (e.g., from a low of Level 1 to a high of Level 4) for the primary, junior and Grade 9 assessments, and successful or unsuccessful results for the OSSLT. For the primary, junior and Grade 9 assessments, five subcategories (e.g., for Level 1: 1.1, 1.3, 1.5, 1.7 and 1.9) are created to indicate where student performance is located within achievement levels.

The enhanced test designs provide a range of positive outcomes and benefits:

Enhanced measurement precision: Increasing the number of short open-response and multiplechoice items, while retaining longer open-response items, enhances data precision and strengthens year-to-year comparability. Enhanced reporting: Reporting scale scores together with levels of proficiency allows students to see where they were performing within performance levels. This type of reporting also allows for more precise and relevant tracking of individual student, school, school board and provincial results to show more effectively where the system is succeeding in closing the gap between low- and high-achieving students.

Efficiencies for schools and school boards: The collection of student demographic data is accomplished electronically, thereby reducing the need for the manual completion of forms by school personnel. The enhanced test designs allow for reductions in the amount of testing time yet maintain a high caliber of assessment information. Data are reported to schools in a timely manner to support schools decisions regarding student placement, intervention programs and school organization and program planning.

Evolution (20052012): Refining Effective Assessment Programs 11

Ensuring Best-of-Class Assessments

EQAO constantly reviews its large-scale assessment practices to ensure they remain best in class. When the office was formed, EQAO established an Assessment Advisory Committee (AAC), which included representatives of various stakeholder groups (e.g., directors of education, superintendents, principals, teachers, school trustees, parents, students). The AAC provides a forum for the sharing of expertise and advice related to the agencys assessment programs. EQAO also established a Psychometric Expert Panel, a group of Ontario, national and international assessment experts who continually review EQAOs assessment methodologies and provide technical advice and mentoring to the agency. In addition, EQAO expanded the involvement of educators in all aspects of the development, implementation and scoring of its assessments. As well, over the years, EQAO has developed significant in-house expertise in test development and psychometrics and continues to conduct assessment-related research in order to stay at the forefront of this field. Finally, EQAO is committed to sharing this expertise and rich research information with the educational community and the public.

The rigorous processes in place for test development scoring and equating the EQAO student assessments enable EQAO to be fully confident that the assessment results reflect the intended academic content and to report the results with a high level of accuracy and year-to-year consistency.
Dr. Mark Reckase, Professor, Measurement and Quantitative Methods, Michigan State University

During the seven years since the test-design changes were introduced, EQAO has continued to consult with its advisory bodies and to reflect on and review its approaches in the spirit of a learning organization that wants to improve continuously. EQAO also regularly conducts systematic peer reviews of its key processes. For instance, on numerous occasions, EQAO has contracted national and international experts to conduct quality assurance audits of its test administration policies, practices and materials and to review its scoring procedures. Test review projects have been conducted by external experts to provide feedback on the quality of reading, writing and mathematics test items and the overall content of the assessments in relation to the curriculum and the tests purposes. External experts have been contracted by the agency to provide training and mentoring to EQAO staff in item and test development.
12 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

Since the changes, the agency has adopted best-of-class methods and has evolved to become a world-class large-scale testing organization. The following section of this paper describes the learning that the agency has acquired over the years by explaining the steps it takes to produce high-quality assessments. Frameworks
In the interest of transparency, EQAO has developed a framework document that provides the foundation of each assessment. The frameworks provide information about EQAO; a description of the relevant large-scale assessment and how it aligns with national and international assessments; discussions of the purposes and benefits of the assessment; information about the assessment blueprint and how the assessment aligns with The Ontario Curriculum; descriptions of the assessment processes, content and accommodations for students with special education needs and information about the scoring of student work and how the assessment is comparable from year to year. an extensive process of generating content, inclass trials, revisions and editing before they can be considered for field testing. An Assessment Development Committee reviews the items to ensure they are appropriate for the age and grade of the student and are directly linked to Ontario Curriculum learning expectations. A Sensitivity Committee reviews the items for fairness and accessibility to the broadest range of Ontario students by considering equity issues such as gender, culture, English-language learners and students with special education needs. It is important that students be neither advantaged nor disadvantaged by the content of the test to allow them to best demonstrate their learning. In addition to the committee reviews, EQAO engages separate groups of expert Ontario educators who carefully review all test items before they are considered for field testing. The same type of review is conducted by the agencys directors and managers prior to field testing.

Item Development
It is critical to ensure that the tests are designed based on The Ontario Curriculum and that the learning expectations assessed and the difficulty of the tests are similar each year. Therefore, all new items are developed based on detailed blueprints that are directly linked to the curriculum. EQAO assessment blueprints are used to develop the multiple-choice and open-response items for each assessment, so that the assessment has the same characteristics from year to year. EQAO recruits and trains Ontario educators from across the province with expertise in the areas of English and French literacy (reading and writing) and mathematics to work on item-writing and review committees. Each year, new items are developed and field tested to become operational items on the following years assessment. Test items go through

Field Testing
Field-test items are embedded in operational assessments. During the writing of an assessment, each student responds to a small number of fieldtest items that do not count toward their final results. Data generated from field testing ensures that items selected for the following years operational assessment have strong statistical properties and are fair for all students. Field testing also provides data that are used to equate each years assessment with the previous years, so that results can be compared over time.

Evolution (20052012): Refining Effective Assessment Programs 13

Test Construction
A number of guidelines must be adhered to when operational tests are constructed, including item fairness and level of difficulty, blueprint coverage and balance for subject content, gender representation and other demographic aspects. Both pedagogical considerations and the psychometric properties of items are taken into consideration as the agencys assessment and psychometric staff collaborate to construct the test forms. EQAOs Psychometric Expert Panel approves all test forms.

and public hearings, the Ontario Governments Standing Committee on Public Accounts recommended that EQAO continue its policy of testing every child. Administration guides outline the procedures that ensure the administration of the assessments is consistent and fair for all students. Each assessment has an administration and an accommodation guide that specifies administrators and teachers roles and responsibilities and gives instructions about preparing students for the test, what to say to students throughout the administration and student participation (i.e., accommodations for students with special education needs, special provisions for English language learners, exemptions and deferrals). EQAO pays attention to quality assurance in test administration in several ways. Quality assurance monitors are assigned to visit a random selection of approximately 5% of schools during testing. The monitors report on the degree to which schools follow test administration guidelines. In addition, the agency statistically analyzes students responses to identify unusual response patterns. If unusual response patterns are identified, the school is contacted for further information. EQAO also examines overall school results over time to identify large improvements in achievement results in the current year. In cases where there are large improvements, school superintendents are asked to document explanations for these changes.

Questionnaires
Contextual information is used to analyze student achievement data by subgroup and to focus recommendations about improving student learning. EQAO develops student, teacher and principal questionnaires to collect contextual data as well as information about instructional practices and attitudes toward literacy and mathematics. The questions reflect a number of factors that are associated with student achievement. The data from these questionnaires are shared with the educational community to inform school improvement practices.

Test Administration
As noted previously, there is an expectation that all students working on Ontario Curriculum learning expectations will participate in the provincial assessments. In December 2010, following a review

The Standing Committee on Public Accounts recommends that the Education Quality and Accountability Office continue its policy of every-child testing.
Legislative Assembly of Ontario. Standing Committee on Public Accounts (2010)

14 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

Scoring
Rigorous scoring procedures ensure the validity and reliability of the scored results. Multiple-choice items are machine scored. Written responses to openresponse reading and mathematics items and writing prompts are carefully and systematically scored by qualified scorers, who are mostly Ontario educators. A generic rubric for each type of open-response item describes student work at each performance level and maintains consistency across items and years. A rubric specific to each open-response item is developed based on a generic rubric. Anchor papers (carefully chosen samples of student work) illustrate the descriptors for each of the score points (codes) in the rubrics. Extensive training and monitoring procedures maintain high levels of scoring accuracy and consistency. All scorers participate in extensive training to develop a clear and common understanding of the scoring materials, so that each scoring leader, scoring supervisor and scorer interprets and applies the scoring materials in the same way. Other quality-control procedures include the following:

The quality assurance measures described are designed to help ensure that a particular student response will receive the same score regardless of the scorer.

Each scorer, scoring supervisor and scoring leader must pass a qualifying test before he or she can score students work. Each scorer is responsible for scoring from one to four items. Scoring accuracy and consistency is enhanced when scorers focus on scoring a small number of items. Daily large-group training activities ensure scoring consistency is maintained. Validity and reliability statistics are compiled for individual scorers and groups of scorers throughout the scoring process. Scoring leaders and supervisors closely monitor scoring accuracy and consistency and provide individual and group retraining when required. All OSSLT student work is double-scored.

Evolution (20052012): Refining Effective Assessment Programs 15

Analyzing and Equating Student Scores and Accuracy of Assessment Results

EQAO uses various statistical methods to ensure the comparability of the assessment results from one administration to the next. The equating process places student scores in two adjacent years on a common scale. This controls for small differences in test difficulty from year to year and ensures that valid comparisons can be made across years and that students in one year are not given an unfair advantage over students in another. Cut scores are then determined to mark the dividing points between the levels of achievement used in reporting EQAO results (four levels for primary, junior and Grade 9 and successful or unsuccessful for the OSSLT). Students assigned a given result in the current year will have demonstrated the same level of skill and knowledge as the students assigned the same result in previous years. EQAO uses Item Response Theory for analyzing and equating student results. Equating procedures and cut-score determinations are conducted separately for English and French versions of the assessments. All analyses are performed twice: once by EQAO psychometric staff and once by an external contractor. The results are compared and any differences resolved, thereby ensuring the accuracy of the results.

The EQAO employs a number of quality assurance measures to provide credibility to its processes and procedures. These measures help to ensure that the questions presented to students are appropriate for their grade level and represent fairly the Ministrys curriculum expectations, and that the tests are consistent in their level of difficulty from one year to the next.
2009 Annual Report, Office of the Auditor General of Ontario

EQAO generates a variety of test statistics to assess the accuracy of the scores provided by its assessments. Overall the statistics show that EQAO test results are highly precise. A number of factors contribute to test accuracy but the quality of the test items and the accuracy and consistency of scoring are of paramount importance. The psychometric and data management teams continually update their use of technology and refine their processes and procedures to reflect best practices and ensure that EQAO test scores are valid and accurate.

16 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

Reporting
Assessment results, which provide valuable information to support improvement planning at all levels, are reported at the student, school, school board and provincial levels. Students and their parents or guardians receive Individual Student Reports, which provide the overall results for each student. Reports for schools and school boards are posted on EQAOs public Web site, at www.eqao.com. Also on its Web site, EQAO publishes English- and French-language elementary and secondary provincial reports for education stakeholders and the general public. Provincial results are published for each assessment within 10 to 12 weeks of administration. In addition, in the interest of transparency, EQAO publishes an annual technical report that describes in detail all processes and procedures associated with its assessment programs for the given year. Complete statistical reports related to scoring and equating are included in the report. EQAO is the only Canadian jurisdiction that publishes an annual technical report of this nature. The agency develops and publishes interpretative reports based on its analysis of data and the findings of the various research projects it conducts to draw attention to trends and areas of concern revealed by the assessment results. Examples of the latter are Grade 9 mathematics results for applied courses, 21st-century skills, traits of high-performing schools and cohort tracking. These reports build capacity in the educational community, inform classroom practice and support improvement planning. A recent initiative to further support administrators and schools in making effective use of the assessment data has been the development of the online EQAO Reporting application. This resource, which maximizes the use of technology based on input from principals, allows principals and school teams to create meaningful learning profiles of their students and focus a whole-school response to data.

Using EQAO data has allowed us to start conversations and to be very specific in targeting the areas of need and making the most efficient use of time, when teaching kids, so we make sure all students are moving forward.
Suzie Robertson, Principal, Elmdale Public School, Ottawa-Carleton District School Board

Evolution (20052012): Refining Effective Assessment Programs 17

Sharing Expertise Through Partnerships and Outreach

EQAO has built a strong staff team to perform the specialized tasks of the agency, maintain effective day-to-day operations and develop extensive partnerships with the educational community and the public. Each department provides effective support while evolving and refining its practices. Appropriate technology and efficient systems have been introduced, which have led to improved communication among teams and departments. The agency is committed to providing relevant and current professional development to its staff and partners in educational assessment. EQAO shares its expertise to allow parents, teachers and administrators to be informed partners in students education. The agency continues to use conventional forms of communication and is increasing its Web-based approaches (e.g., Web magazines, online conferences/seminars, videos) to expand its contact with all stakeholders. Various resources and forums that EQAO has developed help parents, classroom teachers and schools better understand the provincial testing program and engage them in effectively responding to EQAO assessment results to improve student learning. One of the priorities for EQAO continues to be to build capacity for the use of as well as respect for data among teachers, administrators, parents and other stakeholders. The agency addresses this priority through attention to outreach and the involvement of educators from across the province in all aspects of the provincial assessment program. EQAO works with school and school board administrators and other stakeholders to facilitate the use of assessment data for improvement planning. The outreach program creates opportunities to meet and work with a broad crosssection of the educator community and to showcase successful evidence-based strategies that have led to improved student achievement. EQAO approaches outreach collaboratively and involves many of its departments through specific activities. Indeed, EQAO outreach is considered to be everyones responsibility. In 2004, the first principal was seconded to the agency to conduct outreach activities with principals across Ontario. This principals role was to share information with the field about the Ensuring Quality Assessments Review and the subsequent changes. Over the years, the School Support and Outreach Team has grown and now consists of approximately seven credible and influential principals. Today, the teams main role is to help principals, teachers and parents understand, interpret and use EQAO data as a source of evidence for improvement.

18 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

The School Support and Outreach Team is a positive presence in the field, with its members acting as ambassadors and supporting and engaging colleagues in the work of the agency. They are able to draw on personal knowledge of the effectiveness of using data to inform practice and to support school improvement planning in order to guide discussions. School principals and vice-principals have regular opportunities to work with EQAO to enhance their understanding of how to use EQAO data as a basis for supporting student achievement. EQAO continues to promote the concept of a school team that includes teachers, administrators and parents, with students central to all interactions. Visiting schools across the province, Outreach gains firsthand knowledge of the accomplishments of school staff as they work together to support student achievement. Province-wide presentations, seminars, forums and workshops on the topics of EQAO assessment practices, student achievement data as well as the interpretation and use of that data are other strategies for engaging educators and stakeholders. These formal and informal contexts allow the School Support and Outreach Team to gather input

and feedback from principals to inform updates to reports for principals, to respond to the needs of the field and to engage in field research that supports the research priorities of the agency. For example, in response to the feedback, EQAO has developed the Interactive Reporting Resource to Inform Leadership Action, which is available to directors of education and superintendents. This resource encourages schools and school boards to use data in meaningful ways relevant to each educational context. According to EQAOs 20112012 Annual Report, through its numerous presentations, information sessions, learning seminars and workshops, the agency has connected with nearly 1000 parents, 5000 educators and 3500 pre-service teachers during the past year. Based on information gained through outreach activity, EQAO has prepared success stories from the field that allow Ontario principals and other stakeholders to learn about best practices in working with EQAO data to inform next steps for student achievement and school improvement planning.

Evolution (20052012): Refining Effective Assessment Programs 19

Empowering with Good Information

EQAO is mandated to support and guide student improvement by providing credible information about the quality of the provinces publicly funded elementary and secondary education system. In terms of providing information, EQAOs Web site complements EQAOs reporting and outreach efforts. Increasingly, the Web site is used as a gateway to communicating important ideas and information about EQAO and its various assessment programs. By sharing its knowledge, expertise and resources with educators, parents, the Ontario Ministry of Education and the public in order to build common understandings about effective assessment and instructional practices, EQAO maintains a critical role in helping more students achieve the provincial standard.

Over the past fifteen years, provincial student testing has gained broadbased support from parents and the rest of the public. According to the Ontario Institute for Studies in Education at the University of Toronto (OISE/UT) study Public Attitudes Toward Education in Ontario (2012),

64% 70%

of the public supports province wide testing at the elementary level and of the public supports province wide testing of all students at the secondary level.

Public Support for Provincial Testing

Ontarios provincial testing program was established for two main reasons:

Parents and the public demanded greater

accountability in the public education system, including all independent, objective and reliable measurement of students learning at key points in their schooling.

The EQAO study Parents Perspectives: The Importance of Provincial Testing and the Information It Provides About Childrens Learning (2010) revealed that

Valid and reliable information about student

achievement at all levels was considered essential for effective education system improvement planning.

88 74% 69%
%

of parents consider the provincial testing program important, believe EQAO results are used to improve education quality and believe provincial testing makes the  education system accountable to parents and other taxpayers.

Information in this section of the paper is excerpted or adapted from The Power of Ontarios Provincial Testing Program (2012), located on EQAOs Web site at http://www.eqao.com/pdf_e/12/PowerOntProv_TestingProg_en.PDF.

Empowering with Good Information 21

Educator Use of EQAO Data

Since test results are available for every student, EQAO data have become a key ingredient in helping students, schools, school boards and the province identify strengths and areas where attention and resources are needed to make improvements. A strong culture of evidence-based assessment has evolved over the years, both locally and provincially, in large part because of EQAO testing. The data have been used in grade-level, divisional and wholeschool discussions about student work to help identify strengths and areas for growth for individual students and groups and to plan next steps for instruction to improve student learning. EQAO data have also commonly been used as a source of information to strengthen the family-ofschools and feeder-school approaches to providing continuous kindergarten to Grade 12 support for students as they progress from elementary to secondary school. Ontario educators value the information provided by the provincial tests and use it to improve student outcomes.

In 2011, EQAO surveyed more than 3400 elementary school principals and 8500 Grade 3 and 7300 Grade 6 teachers and found that

questionnaire data to guide school improvement initiatives for reading, writing and mathematics;

96% 96 82
%

of principals reported they use EQAO achievement results and

of principals said they use EQAO data to identify program strengths and

areas for improvement in the subjects;

of Grade 3 and 80% of Grade 6 teachers said they use EQAO data to

identify areas of program strength and areas for improvement in the same three subjects and

to identify how well students are meeting curriculum expectations.

80%

of Grade 3 and 78% of Grade 6 teachers reported using EQAO data

22 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

School Board Use of EQAO Data

The Student Achievement and School Board Governance Act (2009) articulates the roles of school boards, trustees, board chairs and directors of education in improving student achievement. EQAO data are one important source of information to use in developing school board improvement plans. EQAO data provide consistent reference points that can be used to complement other sources of information about student learning at the classroom, school and board levels. In addition, directors of education are required to produce and publish annual school board reports for their school communities. Again, EQAO data are important sources of student achievement information for these reports. The Council of Ontario Directors of Education states in a 2011 advisory that using EQAO data to develop and monitor school board and school improvement plans is an effective practice that contributes to student achievement and program quality.

Through review and careful analysis of district results, boards can make general conclusions as to how they are progressing in relation to other districts in the province, and whether they are in line to meet provincial standards. The data that individual schools gather around student achievement provide the most current information that teachers need in order to plan for studentcentered programming, assessment, evaluation and reporting.
Council of Ontario Directors of Education

Empowering with Good Information 23

Provincial Use of EQAO Data

Ontarios provincial test results have helped the province identify students academic strengths and where attention and resources are needed. Government initiatives developed based on EQAOs full-census data have had a measurable impact for many thousands of Ontario students, as evidenced by the significant improvement in achievement in schools across the province. Data collected through EQAO tests have contributed to the establishment of the Ministry of Educations Literacy and Numeracy Secretariat and have enabled the Ministry to establish successful initiatives such as Think Literacy and Learning to 18 in secondary schools and the Ontario Focused Intervention Partnership and Schools in the Middle in elementary schools. EQAO data provide evidence for evaluating and monitoring the implementation and ongoing effectiveness of these initiatives. EQAO data also support the Ministrys work on a range of online and print resources and educator professional development, targeted funding and collaboration with school boards on local strategies to ensure each child learns effectively.

Researchers Use of Data

The EQAO Web site includes a Data Portal that can be used by the public to support and facilitate various research projects. An application form is posted, which also includes descriptions of the data files available and descriptions of the fields contained in each data file. Individuals wishing to access data files submit a request, including a brief description of how the data will be used (research) and the specific files that they want. The files are then provided through a secure site. During 2007 to 2009, 44 requests for EQAO data files were processed. Ten were from university and college students, 13 from professors, 13 from researchers with institutions and organizations, two from school-board staff and six from the media. From 2010 to 2012, 40 requests for EQAO data files have been processed. Thirteen were from university and college students, 11 from professors, 10 from researchers with institutions and organizations, three from school-board staff and three from the media. EQAO data have become reference points in supporting the positions of numerous authors. These works are often the basis for discussions in pre-service, graduate and post-graduate programs, in-school learning groups and at educational conferences. Notably these books and articles include

Lyn Sharratts and Michael Fullans 2012 book, Putting Faces on the Data, relies heavily on EQAO data. The book offers information to those who want to help schools do a better job. They say that evidence of success is needed for informed decision making. EQAO is one major source of data. Michael Fullans book The Moral Imperative Realized refers to EQAOs primary, junior and OSSLT data (as one important source of data) about student achievement over time.

24 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

Improvements in Student Achievement

Over the years, a culture of evidence-based decision making has evolved in Ontario as the province has come to appreciate the value of large-scale assessment. The availability of reliable student achievement information from Ontarios province-wide tests has enabled schools, school boards and the province to develop and implement targeted improvement plans. The result of this focused attention on evidence has been that Ontario students have made notable progress over the years in meeting the provincial standard in reading, writing and mathematics. For example, from 20012011, more than 21 000 additional students have met the provincial standard in reading. Similarly, several recent national and international assessments have shown high levels of achievement among Ontario students. When EQAO was established, Ontario students did not perform as strongly on these assessments as they do today. For example, Ontario was the only jurisdiction in Canada whose students were above the national average in all three subjects (reading, mathematics and science) on the recent administration of the Pan-Canadian Assessment Program. In addition, fully 92% of Ontario students met or exceeded the key achievement benchmark on the 2009 administration of the Programme for International Student Assessment. The focused attention on literacy and numeracy, strongly supported by Ontarios provincial testing program, has led to demonstrated improvements in student achievement that have been confirmed on provincial, national and international assessments.

benchmark on the 2009 administration of the Programme for International Student Assessment.

92

of Ontario students met or exceeded the key achievement

Empowering with Good Information 25

Looking Ahead

As EQAO moves forward, it will hold true to its fundamental principles and established priorities. Although the work of the agency will continue, its methods will evolve to maintain the exemplary status of its assessment programs and reflect the changing educational context. For instance, the agency will explore how it can incorporate technology to best advantage and assist schools and school boards in identifying key learnings using local and provincial data. It will consider how best to support student improvement, and it will continually look for ways to better serve the citizens of Ontario.

Summary
EQAO has distinguished itself as a world-class large-scale testing organization. Over the past 15 years since the agency began testing all students in Ontario at key stages in their learning, our education system has become more robust, and more students are achieving the provincial standard. Because of EQAOs full-census assessment program, school system educators, administrators, parents and the public have had access to detailed information on student achievement in their local schools and school boards. Teachers, principals and administrators have shown that they use the information provided by the provincial tests in conjunction with the results of school- and classroombased assessments to guide decisions in order to improve learning programs for all students. Parents and taxpayers have demanded and deserve this kind of local accountability and detailed information about student achievement. A comprehensive and well-balanced assessment program, anchored by EQAOs full-census assessments, has been a key element of the progress witnessed in Ontario student achievement.

We are all in this together. The bottom line is that we are morally obligated to improve student achievement. EQAO data helps us do this.
Bernadette Gillis, Vice-Principal, St. Barbara Catholic Elementary School, Dufferin-Peel Catholic District School Board

Looking Ahead 27

References
Council of Ontario Directors of Education. (2011). CODE Advisory 11. Retrieved from http://www.ontariodirectors.ca/CODE_Advisories/Downloads/CODE%20Advisory%20 No%2011%20WEB.pdf. Education Quality and Accountability Office. (2012). The power of Ontarios provincial testing program. Toronto: Queens Printer for Ontario. Retrieved from http://www.eqao. com/pdf_e/12/PowerOntProv_TestingProg_en.PDF. Education Quality and Accountability Office. (2012). 20112012 annual report. Toronto: Queens Printer for Ontario. Education Quality and Accountability Office. (2011). EQAOs technical report for the 20102011 assessments. Toronto: Queens Printer for Ontario. Education Quality and Accountability Office. (2010). Parents perspectives: The importance of provincial testing and the information it provides about childrens learning. Retrieved from http://www.eqao.com/Research/pdf/E/Parent_Research_ Findings_ENG.pdf. Education Quality and Accountability Office. (2004). Ensuring quality assessments: Enhancements to EQAOs assessment program. Toronto: Queens Printer for Ontario. Fullan, M. (2011). The moral imperative realized. Thousand Oaks, CA: Corwin. Hart, D. (2012). The 18th OISE Survey of Educational Issues: Public Attitudes Toward Education in Ontario 2012. Retrieved from http://www.oise.utoronto.ca/oise/UserFiles/ File/OISE%20Survey/18th_OISE_Survey/OISE%20SURVEY%2018.pdf.

28 EQAO: Ontarios Provincial Assessment ProgramIts History and Influence

References (continued)
Legislative Assembly of Ontario. (1996). Education Quality and Accountability Office act. Retrieved from http://www.e-laws.gov.on.ca/html/statutes/english/elaws_ statutes_96e11_e.htm. Legislative Assembly of Ontario. Standing Committee on Public Accounts. (2010). The Education Quality and Accountability Office (Section 3.04, 2009 annual report of the auditor general of Ontario). Retrieved from http://www.ontla.on.ca/committeeproceedings/committee-reports/files_pdf/EQAO_S3.04AR2009_EN.pdf. Ministry of Education. (2009). Student achievement and school board governance act. Retrieved from http://www.e-laws.gov.on.ca/html/source/statutes/english/2009/ elaws_src_s09025_e.htm. Office of the Auditor General of Ontario. (2009). 2009 annual report. Toronto: Queens Printer for Ontario. Royal Commission on Learning. (1994). For the love of learning: Report of the royal commission on learning [Short Version]. Toronto: Queens Printer for Ontario. Sharratt, L., & Fullan, M. (2012). Putting faces on the data. Thousand Oaks, CA: Corwin. Wolfe, R., Childs, R., & Elgie, S. (2004). Ensuring quality assessments: The EQAO assessment review projects findings and recommendations. Toronto: Queens Printer for Ontario. Report prepared under contract for the Education Quality and Accountability Office.

References 29

2 Carlton Street, Suite 1200, Toronto ON M5B 2M9 Telephone: 1-888-327-7377 I Web site: www.eqao.com 2013 Queens Printer for Ontario I C15ann_ne_0313