provided suggestions that strengthened the Executive Summary. Shirley Schwartz reviewed an early ... more provided suggestions that strengthened the Executive Summary. Shirley Schwartz reviewed an early draft of the full report, providing an especially useful practitioner's perspective. Thomas Smith's insights into the implementation of the Talent Development High School Model in the School District of Philadelphia are greatly appreciated. Thanks are also due many colleagues at MDRC. Cindy Willner diligently tracked down facts and references and prepared the tables and figures. Thoughtful comments from
evaluation reports present objective information on the conditions of implementation and impacts ... more evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the report. To order copies of this report,
VOL. 19 / NO. 1 / SPRING 2009 3 Approximately 16 million students attend more than 40,000 high sc... more VOL. 19 / NO. 1 / SPRING 2009 3 Approximately 16 million students attend more than 40,000 high schools in the United States. The vast majority of these students (more than 90 percent) attend public schools.1 And yet by most accounts, the typical American high school is failing its students in terms both of excellence and of equity. Although the math and reading achievement scores of both fourthand eighth-grade American youngsters have improved over the past seventeen years according to the nation’s “report card,” the National Assessment of Educational Progress (NAEP), the math and reading scores of twelfth graders have been stagnant or even falling over roughly the same period. As another way to think about it, the overall U.S. achievement goal is for all students to score at or above the proficient level—the level at which they demonstrate solid academic performance exhibiting competency over challenging subject matter.2 And yet in 2005 only 35 percent of the nation’s high school s...
Department of Education published research findings on Reading First, a centerpiece of the No Chi... more Department of Education published research findings on Reading First, a centerpiece of the No Child Left Behind (NCLB) Act that provided $1 billion per year to help all children read at or above grade level by the end of third grade. 1 The findings were interpreted by many in the media and the policy community as saying that Reading First did not work. Although the story is more nuanced than that, funding for the program was eliminated in the fiscal 2009 spending bill that was signed by President Obama in March. NCLB is up for reauthorization in 2009. In the meantime, the American Recovery and Reinvestment Act provides tens of billions of dollars to states and localities for spending on education, meaning that federal, state, and local policymakers face critical choices today about how best to use this money to support early reading instruction and achievement.
The impact analysis in this report is based on a design known as a comparative interrupted time s... more The impact analysis in this report is based on a design known as a comparative interrupted time series, a method used widely in education research and evaluation to assess the impact of school-wide programs and systemic policies on student outcomes. 1 There are several potential influences on math test scores that must be controlled for by the comparative interrupted time series analysis: The central strength of this methodology is that it accounts for many factors that may have produced changes in math achievement in the So1 schools instead of or in addition to the implementation of the school-wide So1 program in the 2010-2011 school year. The goal of accounting for these factors is to construct the best estimate of math achievement levels that were likely to have occurred in the So1 schools in the absence of the program. This alternative is known as a counterfactual. The analyses conducted for this chapter are based on a particularly strong counterfactual in that it accounts for many important alternative influences on student test scores that may have been present over and above the implementation of So1 in the 2010-2011 school year. A strong counterfactual increases confidence that the findings from the analyses constitute rigorous evidence of effects, or lack of effects, from the program. 1 This statistical methodology has been used widely in education research and evaluation (see Bloom, 1999 and Shadish, Cook, & Campbell, 2002). As in this paper, comparative interrupted time series analyses have been applied primarily to study broad systemic policies and interventions such as the federal No
The analyses of test score trends utilize school-level data compiled from the New York State Educ... more The analyses of test score trends utilize school-level data compiled from the New York State Education Department (NYSED), the DOE, and the Common Core of Data (CCD) from the National Center for Education Statistics (NCES) at the U.S. Department of Education. 6 NYSED data on grade 4 and grade 8 ELA and math test scores are available for the school years 1998-99 through 2009-10 and include both the average scale score for each school and the percentage of students in each school that scored at designated performance levels aligned with statewide performance standards (NYSED, 2010). 7 10 The sample of schools used for these analyses represents 81% of all schools that conducted testing of 4 th or 8 th grade students at any time during the period from 1999 through 2009. 11 In controlling for these factors, it is important that the comparison districts and schools be as similar as possible to New York City in their test score trends prior to 2003. The primary findings in this paper are based on analyses that focus on test score trends for New York State's Big Four school districts. As shown in Table 1, these four districts exhibited quite similar test scores prior to 2003. As urban school districts with high percentages of students with similar characteristics, these districts also experienced many of the same challenges and opportunities presented by national and state education policy initiatives and secular trends. 12 See Schwartz and Stiefel, this volume.
This report has been more than two years in the making and benefited greatly from guidance, insig... more This report has been more than two years in the making and benefited greatly from guidance, insights, and critique provided by colleagues at the Research Alliance and elsewhere. At the Research Alliance, Michael Segeritz was instrumental in clarifying the statistical models and many features of the analytic design. Christy Baker-Smith provided many hours of assistance with data management, early versions of the analysis, and the initial presentations of the empirical findings. Adriana Villavicencio offered constructive feedback on multiple iterations of the findings and drafts of the report. The author is especially grateful for the innumerable discussions with Saskia Levy Thompson about the broader context of high school reform in New York City over the past decade. Saskia's extraordinary insights were drawn from her more than 15 years of work with the City's schools as a practitioner at the Urban Assembly,
More than 69,000 educators logged on to ARIS at least once during the 2010-2011 school year. This... more More than 69,000 educators logged on to ARIS at least once during the 2010-2011 school year. This represents 73 percent of all school-based administrators and teachers, including 94 percent of principals, 84 percent of assistant principals, 67 percent of teachers, and 60 percent of other administrative staff. The average user logged on to the system 21 times during the year, for just under five minutes per session.
In 1987, Florida anticipated the federal Job Opportunities and Basic Skills Program with Project ... more In 1987, Florida anticipated the federal Job Opportunities and Basic Skills Program with Project Independence, a statewide welfare-to-work program. It was structured to increase employment among Aid to Families with Dependent Children (AFDC) recipients as quickly as possible, primarily through job search activities. An evaluation was begun in 9 counties randomly selected from among the state's 25 largest in terms of AFDC caseloads; the research sample consisted of more than 18,000 ',ingle parents. The organizational capacity of local programs differed in several important dimensions, including number and characteristics of staff, caseload sizes, service availability and quality, and child care availability. The project achieved substantial compliance with its participation mandate-75 percent of those required to participate in the program attended orientation. Fifty-six percent of those who attended orientation went on to participate in a job search, education, or training activity. The project increased first-year earnings by nearly 7 percent and reduced first-year AFDC payments by nearly 7 percent; The program's effects were concentrated among single parents with school-age children; their earnings increased by 11 percent. Single parents with younger children (aged 3-5) experienced a 5 percent reduction in welfare payments but no significant increase in earnings. (Appendixes include supplemental tables and 18 references.) (YLB)
National Center For Education Evaluation and Regional Assistance, Apr 1, 2008
IES evaluation reports present objective information on the conditions of implementation and impa... more IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the fi ndings in the reports.
This paper argues that the federal government's role in vocational and technical education (VTE) ... more This paper argues that the federal government's role in vocational and technical education (VTE) should include a greater emphasis on systematically testing promising strategies and interventions and subjecting them to rigorous evaluations of their effects, implementation, costs, and benefits. Section 1 presents a rationale for rethinking the federal role in VTE at the secondary level and highlights several potential benefits of moving to a greater focus on systematic innovation and rigorous evaluation. Section 2 outlines a set of guiding principles for what systematic innovation and rigorous evaluation ought to look like. Section 3 touches on several strategies that might be used to help shift the federal role toward a greater focus on these principles. (Throughout, the paper implies that a federal commitment to systematic innovation and rigorous evaluation should not be confined to VTE. It contends that the current administration's focus on ensuring that "no child is left behind" appears to be particularly wellsuited to using federal resources to leverage state and local investments in promising strategies to improve schools and to rigorously evaluate them in ways that assess the impacts of the reforms and help develop accountability systems for further improvement.) (YLB) Reproductions supplied by EDRS are the best that can be made from the original document.
The Talent Development High School model is an education reform initiative that aims to improve t... more The Talent Development High School model is an education reform initiative that aims to improve the academic achievement of students in large, nonselective, comprehensive high schools. In operation at 33 high schools in 12 states across the country, the approach encompasses five main features: small learning communities, organized around interdisciplinary teacher teams that share the same students and have common daily planning time; curricula leading to advanced English and mathematics coursework; academic extra-help sessions; staff professional development strategies; and parent-and community-involvement in activities that foster students' career and college development.
provided suggestions that strengthened the Executive Summary. Shirley Schwartz reviewed an early ... more provided suggestions that strengthened the Executive Summary. Shirley Schwartz reviewed an early draft of the full report, providing an especially useful practitioner's perspective. Thomas Smith's insights into the implementation of the Talent Development High School Model in the School District of Philadelphia are greatly appreciated. Thanks are also due many colleagues at MDRC. Cindy Willner diligently tracked down facts and references and prepared the tables and figures. Thoughtful comments from
evaluation reports present objective information on the conditions of implementation and impacts ... more evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the report. To order copies of this report,
VOL. 19 / NO. 1 / SPRING 2009 3 Approximately 16 million students attend more than 40,000 high sc... more VOL. 19 / NO. 1 / SPRING 2009 3 Approximately 16 million students attend more than 40,000 high schools in the United States. The vast majority of these students (more than 90 percent) attend public schools.1 And yet by most accounts, the typical American high school is failing its students in terms both of excellence and of equity. Although the math and reading achievement scores of both fourthand eighth-grade American youngsters have improved over the past seventeen years according to the nation’s “report card,” the National Assessment of Educational Progress (NAEP), the math and reading scores of twelfth graders have been stagnant or even falling over roughly the same period. As another way to think about it, the overall U.S. achievement goal is for all students to score at or above the proficient level—the level at which they demonstrate solid academic performance exhibiting competency over challenging subject matter.2 And yet in 2005 only 35 percent of the nation’s high school s...
Department of Education published research findings on Reading First, a centerpiece of the No Chi... more Department of Education published research findings on Reading First, a centerpiece of the No Child Left Behind (NCLB) Act that provided $1 billion per year to help all children read at or above grade level by the end of third grade. 1 The findings were interpreted by many in the media and the policy community as saying that Reading First did not work. Although the story is more nuanced than that, funding for the program was eliminated in the fiscal 2009 spending bill that was signed by President Obama in March. NCLB is up for reauthorization in 2009. In the meantime, the American Recovery and Reinvestment Act provides tens of billions of dollars to states and localities for spending on education, meaning that federal, state, and local policymakers face critical choices today about how best to use this money to support early reading instruction and achievement.
The impact analysis in this report is based on a design known as a comparative interrupted time s... more The impact analysis in this report is based on a design known as a comparative interrupted time series, a method used widely in education research and evaluation to assess the impact of school-wide programs and systemic policies on student outcomes. 1 There are several potential influences on math test scores that must be controlled for by the comparative interrupted time series analysis: The central strength of this methodology is that it accounts for many factors that may have produced changes in math achievement in the So1 schools instead of or in addition to the implementation of the school-wide So1 program in the 2010-2011 school year. The goal of accounting for these factors is to construct the best estimate of math achievement levels that were likely to have occurred in the So1 schools in the absence of the program. This alternative is known as a counterfactual. The analyses conducted for this chapter are based on a particularly strong counterfactual in that it accounts for many important alternative influences on student test scores that may have been present over and above the implementation of So1 in the 2010-2011 school year. A strong counterfactual increases confidence that the findings from the analyses constitute rigorous evidence of effects, or lack of effects, from the program. 1 This statistical methodology has been used widely in education research and evaluation (see Bloom, 1999 and Shadish, Cook, & Campbell, 2002). As in this paper, comparative interrupted time series analyses have been applied primarily to study broad systemic policies and interventions such as the federal No
The analyses of test score trends utilize school-level data compiled from the New York State Educ... more The analyses of test score trends utilize school-level data compiled from the New York State Education Department (NYSED), the DOE, and the Common Core of Data (CCD) from the National Center for Education Statistics (NCES) at the U.S. Department of Education. 6 NYSED data on grade 4 and grade 8 ELA and math test scores are available for the school years 1998-99 through 2009-10 and include both the average scale score for each school and the percentage of students in each school that scored at designated performance levels aligned with statewide performance standards (NYSED, 2010). 7 10 The sample of schools used for these analyses represents 81% of all schools that conducted testing of 4 th or 8 th grade students at any time during the period from 1999 through 2009. 11 In controlling for these factors, it is important that the comparison districts and schools be as similar as possible to New York City in their test score trends prior to 2003. The primary findings in this paper are based on analyses that focus on test score trends for New York State's Big Four school districts. As shown in Table 1, these four districts exhibited quite similar test scores prior to 2003. As urban school districts with high percentages of students with similar characteristics, these districts also experienced many of the same challenges and opportunities presented by national and state education policy initiatives and secular trends. 12 See Schwartz and Stiefel, this volume.
This report has been more than two years in the making and benefited greatly from guidance, insig... more This report has been more than two years in the making and benefited greatly from guidance, insights, and critique provided by colleagues at the Research Alliance and elsewhere. At the Research Alliance, Michael Segeritz was instrumental in clarifying the statistical models and many features of the analytic design. Christy Baker-Smith provided many hours of assistance with data management, early versions of the analysis, and the initial presentations of the empirical findings. Adriana Villavicencio offered constructive feedback on multiple iterations of the findings and drafts of the report. The author is especially grateful for the innumerable discussions with Saskia Levy Thompson about the broader context of high school reform in New York City over the past decade. Saskia's extraordinary insights were drawn from her more than 15 years of work with the City's schools as a practitioner at the Urban Assembly,
More than 69,000 educators logged on to ARIS at least once during the 2010-2011 school year. This... more More than 69,000 educators logged on to ARIS at least once during the 2010-2011 school year. This represents 73 percent of all school-based administrators and teachers, including 94 percent of principals, 84 percent of assistant principals, 67 percent of teachers, and 60 percent of other administrative staff. The average user logged on to the system 21 times during the year, for just under five minutes per session.
In 1987, Florida anticipated the federal Job Opportunities and Basic Skills Program with Project ... more In 1987, Florida anticipated the federal Job Opportunities and Basic Skills Program with Project Independence, a statewide welfare-to-work program. It was structured to increase employment among Aid to Families with Dependent Children (AFDC) recipients as quickly as possible, primarily through job search activities. An evaluation was begun in 9 counties randomly selected from among the state's 25 largest in terms of AFDC caseloads; the research sample consisted of more than 18,000 ',ingle parents. The organizational capacity of local programs differed in several important dimensions, including number and characteristics of staff, caseload sizes, service availability and quality, and child care availability. The project achieved substantial compliance with its participation mandate-75 percent of those required to participate in the program attended orientation. Fifty-six percent of those who attended orientation went on to participate in a job search, education, or training activity. The project increased first-year earnings by nearly 7 percent and reduced first-year AFDC payments by nearly 7 percent; The program's effects were concentrated among single parents with school-age children; their earnings increased by 11 percent. Single parents with younger children (aged 3-5) experienced a 5 percent reduction in welfare payments but no significant increase in earnings. (Appendixes include supplemental tables and 18 references.) (YLB)
National Center For Education Evaluation and Regional Assistance, Apr 1, 2008
IES evaluation reports present objective information on the conditions of implementation and impa... more IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the fi ndings in the reports.
This paper argues that the federal government's role in vocational and technical education (VTE) ... more This paper argues that the federal government's role in vocational and technical education (VTE) should include a greater emphasis on systematically testing promising strategies and interventions and subjecting them to rigorous evaluations of their effects, implementation, costs, and benefits. Section 1 presents a rationale for rethinking the federal role in VTE at the secondary level and highlights several potential benefits of moving to a greater focus on systematic innovation and rigorous evaluation. Section 2 outlines a set of guiding principles for what systematic innovation and rigorous evaluation ought to look like. Section 3 touches on several strategies that might be used to help shift the federal role toward a greater focus on these principles. (Throughout, the paper implies that a federal commitment to systematic innovation and rigorous evaluation should not be confined to VTE. It contends that the current administration's focus on ensuring that "no child is left behind" appears to be particularly wellsuited to using federal resources to leverage state and local investments in promising strategies to improve schools and to rigorously evaluate them in ways that assess the impacts of the reforms and help develop accountability systems for further improvement.) (YLB) Reproductions supplied by EDRS are the best that can be made from the original document.
The Talent Development High School model is an education reform initiative that aims to improve t... more The Talent Development High School model is an education reform initiative that aims to improve the academic achievement of students in large, nonselective, comprehensive high schools. In operation at 33 high schools in 12 states across the country, the approach encompasses five main features: small learning communities, organized around interdisciplinary teacher teams that share the same students and have common daily planning time; curricula leading to advanced English and mathematics coursework; academic extra-help sessions; staff professional development strategies; and parent-and community-involvement in activities that foster students' career and college development.
Uploads
Papers by J. Kemple