Educational Testing Service

Tort law offers an appropriate vehicle for handling the increased prevalence of standardized testing in the wake of the No Child Left Behind Act and the potential miss-scoring problems that arise with standardized tests. This incentivizes the use of reasonable care in scoring standardized tests and justly compensates for miss-scoring harms when doing so does not unduly burden testing agencies.  Neither contract law nor the existing Truth-in-Testing law adequately affords the sort of remedies and protections the issue of standardized testing and miss-scoring pose.

The ability for tort liability to adequately hold testing agencies accountable for miss-scoring errors and afford relief to test-takers will vary based on the actions or inactions of the testing agencies. The culpability of the testing agency and other considerations potentially give rise to actions for negligence and infliction of emotional distress, misrepresentation, defamation and false-light invasion of privacy, tortious interference with prospective advantage, or injurious falsehood—also known as disparagement. While some theories will rarely offer viable avenues for recovery, others, given particular facts, may provide a basis for relief.

Given the great, new weight placed on standardized tests in modern times, tort law posses the ability to best afford remedies to parties injured by miss-scorings on the part of testing agencies. However, relief should only be attained after a painstaking review of the facts of each case in light of principles of tort liability, seeking to incentivize the use of reasonable care in scoring standardized tests and compensating harm in meritorious cases.

This paper presents a socio-cognitive framework for connecting writing pedagogy and writing assessment with modern social and cognitive theories of writing. It focuses on providing a general framework that highlights the connections between writing competency and other literacy skills; identifies key connections between literacy instruction, writing assessment, and activity and genre theories; and presents a specific proposal about how writing assessment can be organized to promote best practices in writing instruction.

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning.

As part of its educational and social mission and in fulfilling the organization's non-profit Charter and Bylaws, ETS has and continues to learn from and also to lead research that furthers educational and measurement research to advance quality and equity in education and assessment for all users of the organization's products and services.

This paper outlines an approach to improving learning and teaching that combines two strong research bases: The research on formative assessment or assessment for learning provides information about what to change; research on teacher learning communities guides decisions about how to change. In this paper we describe the content and process for one model of teacher professional development. Two case studies for two school districts engaged in this process of teacher change are then presented along with descriptions of the observed effects on both the participating teachers and their larger school contexts.

This report shows that the deterministic-input noisy-AND (DINA) model is a special case of more general compensatory diagnostic models by means of a reparameterization of the skill space and the design (Q-) matrix of item by skills associations. This reparameterization produces a compensatory model that is equivalent to the (conjunctive) DINA model, and is valid for all types of complex structure Q-matrices, not only for trivial cases. This equivalency uses the GDM as the basis, is not based on recent developments of diagnosis models such as G-DINA or LCDM. Model equivalency is a topic of some relevance as soon as researchers want to draw conclusions derived from any particular model-based estimates.

This paper identifies three noncognitive domains relevant for academic achievement in K-12—student engagement, behavioral learning strategies, and school climate. The paper also documents empirical findings that show relationships between these three noncognitive domains and academic achievement, especially in the areas of reading and mathematics. Key words: K-12, noncognitive, academic achievement, reading, mathematics

This paper undertakes a review of a model developed as part of the Cognitively Based Assessment of, for, and as Learning (CBAL) initiative, an ongoing research and development project at ETS that is exploring new ways to assess students in kindergarten through Grade 12 (K -12). The model is:

This paper presents a socio-cognitive framework for connecting writing pedagogy and writing assessment with modern social and cognitive theories of writing. It focuses on providing a general framework that highlights the connections between writing competency and other literacy skills; identifies key connections between literacy instruction, writing assessment, and activity and genre theories; and presents a specific proposal about how writing assessment can be organized to promote best practices in writing instruction.

LANGUAGE is a trademark of ETS. As part of its nonprofit mission, ETS conducts and disseminates the results of research to advance quality and equity in education and assessment for the benefit of ETS’s constituents and the field. ETS Research Reports provide preliminary and limited dissemination of ETS research prior to publication. To obtain a PDF or a print copy of a report, please visit:

Large-scale educational surveys are low-stakes assessments of educational outcomes conducted using nationally representative samples. In these surveys, students do not receive individual scores, and the outcome of the assessment is inconsequential for respondents. The low-stakes nature of these surveys, as well as variations in average performance across countries and other factors such as different testing traditions, are contributing factors to the amount of omitted responses in these assessments. While underlying reasons for omissions are not completely understood, common practice in international assessments is to employ a deterministic treatment of omissions, either as missing observations or as responses that are considered wrong. Both approaches appear problematic. In this project, we analyzed the effects of treating omitted responses either as missing or as wrong, as is done in the majority of international studies, and compared these data-treatment solutions to model-based approaches to treating omitted responses. The two types of model-based approaches used in this study are: (a) extensions of multidimensional item response theory (IRT) with an additional dimension based on response indicator variables defined and calibrated together with the set of items containing the observed responses and (b) multidimensional, multiple-group IRT models with a grouping variable representing the within-country stratification of respondents by the amount of omitted responses.

Interest in setting high educational standards is at the forefront of many school reform programs in the United States. States' pursuit of high standards can be advanced through implementation of programs such as High Schools That Work (HSTW), which emphasizes high expectations and rigorous academics for all students, including Career and Technical Education students. High Schools That Work is the largest comprehensive school reform program for high schools in the United States, with over 1,000 schools in more than 30 states currently participating. This report describes the program, previous research, and recent research studies conducted by ETS to ensure validity for the HSTW program.

This paper examines item response theory (IRT) scale transformations and IRT scale linking methods used in the Non-Equivalent Groups with Anchor Test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean-mean, mean-var linking, concurrent calibration, Stocking and Lord and Haebara characteristic curves approaches, and fixed-item parameters scale linkage.

The United States has recently seen falling test scores for mathematics problem solving in comparison to other countries. This paper reviews current approaches to diagnosing mathematics problem solving, and then introduces emerging technologies being developed at Educational Testing Service that address open areas found in the review. The application of these technologies to assessment design must be weighed against concerns for construct validity, equity, and access. The validity question is addressed by using evidence- ...

No Child Left Behind has highlighted the need for new types of assessments that not only provide high-quality evidence about what students know and can do, but also help to move learning forward. This paper describes a linked set of formative and summative reading assessments designed to address the tradeoffs inherent in these two goals. Targeted skills include the full range of competencies underlying proficient reading at the middle-school level, including both lower-order skills such as oral reading fluency and decoding, and higher-order skills such as the ability to integrate and synthesize information from multiple texts. Data collected in pilot administrations of two prototype test forms are presented. Analyses suggest that this new approach yields acceptable measurement properties while simultaneously addressing crucial learning outcomes. This paper was presented June 23, 2009 at the National Conference on Student Assessment,

Providing information to test takers and test score users about the abilities of test takers at different score levels has been a persistent problem in educational and psychological measurement (Carroll, 1993). Scale anchoring (Beaton & Allen, 1992), a technique that describes what students at different points on a score scale know and can do, is a tool to provide such information. Scale anchoring for a test involves substantial amount of work, both by the statistical analysts and test developers involved with the test. In addition, scale anchoring involves considerable use of subjective judgment, so its conclusions may be questionable. This paper describes statistical procedures that can be used to determine if scale anchoring is likely to be successful for a test. If these procedures indicate that scale anchoring is unlikely to be successful, then there is little reason to perform a detailed scale anchoring study. The procedures are applied to several data sets from a teacher licensing test.

The United States has recently seen falling test scores for mathematics problem solving in comparison to other countries. This paper reviews current approaches to diagnosing mathematics problem solving, and then introduces emerging technologies being developed at Educational Testing Service that address open areas found in the review. The application of these technologies to assessment design must be weighed against concerns for construct validity, equity, and access. The validity question is addressed by using evidence-centered design (ECD) methods to build an evidentiary argument. An innovative project called Mathematics Intervention Module (MIM) for helping students improve their mathematical problem solving skills is described that uses ECD methods in concert with the emerging technologies, with a focus on diagnosis, feedback, practice, and items that integrate targeted knowledge and skills.

support and help on the project. It has been argued that in the knowledge economy, a graduate degree will become the new bachelor's degree, the minimal education credential that high-skills employers require. If that is so, then the United States is in peril of losing its competitive edge, with long-term consequences for our economy, our quality of life, and our global standing. This report examines the data behind these assertions, and proposes a set of recommendations to strengthen U.S. graduate education in partnership with industry and government. The United States' system of graduate education is a strategic national asset. Like all valuable assets, it must be attended to and nurtured in order to remain viable and strong. Other countries and regions of the world have recognized the value of graduate education as a vital component of economic development and are making investments accordingly. Europe already produces more doctorates in science and engineering than are pr...

The L2 field's first concerted effort in terms of computer-based testing (CBT) emerged in the mid-80s with the 1985 LTRC. The conference proceedings were published under the title Technology and Language Testing (Stansfield, 1986). The proceedings indicate that several papers presented at the conference dealt with CBT and the application of latent trait models to item-bank construction, item selection, and computer adaptive testing (CAT). The general measurement profession had been working with CBT and, more specifically, with CAT since the early 70s. The first conference on CAT was held in 1975. Perhaps the main reason the L2 field has lagged behind in this area is because it has long promoted performance-based assessment, a form of assessment that does not lend itself as readily to computerized administration as do more traditional test formats. In fact, the second section of the Stansfield volume deals primarily with performance-based assessment. So, whereas general measureme...

As part of its educational and social mission and in fulfilling the organization's non-profit Charter and Bylaws, ETS has and continues to learn from and also to lead research that furthers educational and measurement research to advance quality and equity in education and assessment for all users of the organization's products and services.

Hierarchical linear modelling (HLM) and data from the New Brunswick School Climate Study were used to examine student background, school context, and school climate effects on Grade 6 student achievement in mathematics, science, reading, and writing. Gender, socioeconomic status (SES), and Native ethnicity were significant predictors of academic achievement. Schools showed the smallest variation in reading, the largest in mathematics. School mean SES was significant in mathematics, reading, and writing achievement, as was disciplinary climate in mathematics, science, and writing. School size and parental involvement significantly affected only the relationship between mathematics achievement and individual SES.

This paper undertakes a review of a model developed as part of the Cognitively Based Assessment of, for, and as Learning (CBAL) initiative, an ongoing research and development project at ETS that is exploring new ways to assess students in kindergarten through Grade 12 (K -12). The model is:

This paper outlines an approach to improving learning and teaching that combines two strong research bases: The research on formative assessment or assessment for learning provides information about what to change; research on teacher learning communities guides decisions about how to change. In this paper we describe the content and process for one model of teacher professional development. Two case studies for two school districts engaged in this process of teacher change are then presented along with descriptions of the observed effects on both the participating teachers and their larger school contexts.

Abstract: This study responds to the Test of English as a Foreign Language [TM](TOEFL [R]) research agenda concerning the need to understand the processes and knowledge that test-takers utilize. Specifically, it investigates the strategic behaviors test-takers reported using ...

Interest in setting high educational standards is at the forefront of many school reform programs in the United States. States' pursuit of high standards can be advanced through implementation of programs such as High Schools That Work (HSTW), which emphasizes high expectations and rigorous academics for all students, including Career and Technical Education students. High Schools That Work is the largest comprehensive school reform program for high schools in the United States, with over 1,000 schools in more than 30 states currently participating. This report describes the program, previous research, and recent research studies conducted by ETS to ensure validity for the HSTW program.

As part of its educational and social mission and in fulfilling the organization's non-profit Charter and Bylaws, ETS has and continues to learn from and also to lead research that furthers educational and measurement research to advance quality and equity in education and assessment for all users of the organization's products and services.

Developing a comprehensive strategy for postsecondary education that will meet the needs of America's diverse population and help ensure our ability to compete in the global economy is vital to the growth of our nation. Key to improving performance of our colleges and ...

It has been argued that in the knowledge economy, a graduate degree will become the new bachelor's degree, the minimal education credential that high-skills employers require. If that is so, then the United States is in peril of losing its competitive edge, with long-term ...

This investigation of the comparability of writing assessment prompts was conducted in two phases. In an exploratory Phase I, 47 writing prompts administered in the computer-based Test of English as a Foreign Language™ (TOEFL  CBT) from July through December 1998 were examined. Logistic regression procedures were used to estimate prompt difficulty and gender effects. A panel of experts reviewed selected prompts, and a taxonomy of prompt characteristics was developed and related to prompt difficulty and gender differences. In Phase II, 87 prompts administered from July 1998 through March 2000 were analyzed. All of the prompts used in Phase I, together with 40 new prompts, were analyzed using the larger Phase II database.

E-rater® has been used by the Educational Testing Service for automated essay scoring since 1999. This paper describes a new version of e-rater (V.2) that is different from other automated essay scoring systems in several important respects. The main innovations of e-rater V.2 are a small, intuitive, and meaningful set of features used for scoring; a single scoring model and standards can be used across all prompts of an assessment; modeling procedures that are transparent and flexible, and can be based entirely on expert judgment. The paper describes this new system and presents evidence on the validity and reliability of its scores.

In May 2016, an invitational symposium on the Reading for Understanding (RfU) initiative was held in Alexandria, VA. Co-hosted by ETS and the Council of Chief State School Officers (CCSSO), the symposium brought together some 160 state and local education leaders to examine the results of the Reading for Understanding Initiative. The goal of the Institute of Education Sciences-sponsored initiative was to accelerate the research on reading across grades preK-12. In 2010, five grant projects were awarded to focus on learning and instruction, and one project on assessment. In this policy report, the authors begin by framing the need for a focused, national effort to achieve universal, advanced reading literacy, arguing that it is an issue of equity as well as individual and national prosperity. The second portion presents a discussion of practical recommendations for lead practitioners, summarizing key insights from the RfU Research Initiative, remaining research challenges, and policy and practice recommendations for enhancing reading achievement across the educational developmental span from pre-K to secondary school graduation. The presentations of the research teams and videos of concluding panel discussions from the symposium can be found here: https://www.ets.org/research/events /reading_for_understanding_symposium. Two of the authors of this report (John Sabatini and Tenaha O'Reilly) were members of the RfU assessment project. While insiders in the RfU initiative, they were also outside observers of the other curricula and instruction-focused teams that were designing, implementing, and evaluating instructional programs in schools. This gave them a distinct perspective on how the research teams interacted with schools to develop content, support professional teacher development, and implement curriculum and instructional interventions. We would like to give special thanks to the CCSSO and the many members of their State Collaboratives on Assessment and Student Standards who participated in the conference and provided valuable insights into the major themes of this report. In addition, the CCSSO advisory group of state leaders who reviewed this paper helped us focus it on issues of high importance to state and district leaders. Their expertise and extensive experience are greatly appreciated.

Due to some progress on the natural language processing (NLP) front, researchers are able to pursue the problem of automatic content assessment for free text responses with some success. In particular, a concept-based scoring method implemented in c-rater, Educational Testing Service’s (ETS) technology for content scoring of short free-text answers makes c-rater capable of giving instantaneous formative individualized feedback without

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning.

This paper undertakes a review of a model developed as part of the Cognitively Based Assessment of, for, and as Learning (CBAL) initiative, an ongoing research and development project at ETS that is exploring new ways to assess students in kindergarten through Grade 12 (K-12). The model is:  based on modern cognitive understandings;  built to include integrated constructed-response tasks that can be equally useful for assessment and for instruction;  and structured to allow multiple measurements over the course of a school year. The model described in this paper, which emerges from a review of the literature on writing, places a strong emphasis on writing as an integrated, socially situated skill. Such skills cannot be assessed properly without taking into account the fact that most writing tasks involve management of a complex array of skills over the course of a writing project. A skilled writer integrates language and literacy skills, document creation, document-management skills and critical-thinking skills, among others, and has strategies for coordinating them effectively. More than anything else, this paper is about connections. We are concerned with connections among writing, reading and critical thinking; with relating writing to its social context; and, above all, with making strong connections to how writing is tested and how writing is taught. The context is an ongoing effort at ETS to develop a new approach to K-12 writing assessment in which these connections are not only respected but deeply embedded into the very design of assessments. 1 Writing is not an isolated skill. It builds upon a broad foundation of prerequisite literacy skills, both supports and requires the development of critical-thinking skills, and requires the writer to address a complicated array of rhetorical, conceptual, and linguistic problems. None of these themes are new in and of themselves. To point out a few of the more salient discussions, Shanahan (2006) examines complex interconnections and interdependencies among reading, writing and oral language. Applebee (1984) reviews older literatures connecting writing to the development of critical thought, while Hillocks (1987; 1995; 2002; 2003b) emphasizes the importance of inquiry in writing, noting that students need above all to learn strategies that will enable them to think about the subject 1 The project reported in this paper reflects the work of many people at ETS. The larger project of which this is a part was initiated under Randy Bennett's leadership and reflects his vision for an integrated assessment system. Nora Odendahl played a major role in the original conceptualization and development, and key features of the design reflect her insights. Mary Fowles has been an equal partner in the work at every stage, and the assessment designs reported here reflect her leadership and the work of many test developers at ETS, including

The Common Core Standards call for students to be exposed to a much greater level of text complexity than has been the norm in schools for the past 40 years. Textbook publishers, teachers, and assessment developers are being asked to refocus materials and methods to ensure that students are challenged to read texts at steadily increasing complexity levels as they progress through school so that all students remain on track to achieve college and career readiness by the end of 12th grade. Although automated text analysis tools have been proposed as one method for helping educators achieve this goal, research suggests that existing tools are subject to three limitations: inadequate construct coverage, overly narrow criterion variables, and inappropriate treatment of genre effects. Modeling approaches developed to address these limitations are described. Recommended approaches are incorporated into a new text analysis system called SourceRater. Validity analyses implemented on an independent sample of texts suggest that, compared to existing approaches, SourceRater's estimates of text complexity are more reflective of the complexity classifications given in the new standards. Implications for the development of learning progressions designed to help educators organize curriculum, instruction, and assessment in reading are discussed.

Estimation of parameters of random effects models from samples collected via complex multistage designs is considered. One way to reduce estimation bias due to unequal probabilities of selection is to incorporate sampling weights. Many researchers have been proposed various weighting methods (Korn, & Graubard, 2003; Pfeffermann, Skinner, Holmes, Goldstein, & Rasbash, 1998) in estimating the parameters of hierarchical models, including random effects models as a special case. In this paper, the bias of the weighted analysis of variance (ANOVA) estimators of the variance components for a two-level, one-way random effects model is evaluated. For these estimators, analytic bias expressions are first developed, the expressions are then used to examine the impact of sample size, intraclass correlation coefficient (ICC), and the sampling design on the bias of the estimators. In addition, two-stage sampling designs are considered, with a general probability design at the first stage (Level 2) and simple random sampling without replacement (SRS) at the second stage (Level 1). The study shows that firstorder weighted variance component estimators perform well when for moderate cluster sizes and ICC values. However, noticeable estimation bias can be found with this weighting method for small cluster sizes (less than 20), particularly when ICC is small (less than 0.2). In such scenarios, scaled first-order weighted estimators can be an alternative. This paper is discussed in the context of National Assessment of Educational Progress (NAEP) 2003 4th Grade Reading National and State Assessment data, with Level 1 being the student level and Level 2 being the school level.

One of the major objectives of large-scale educational surveys is reporting trends in academic achievement. For this purpose, a substantial number of items are carried from one assessment cycle to the next. The linking process that places academic abilities measured in different assessments on a common scale is usually based on a concurrent calibration of adjacent assessments using item response theory (IRT) models. It can be conjectured that the selection of common items has a direct effect on the estimation error of academic abilities due to item misfit, small changes in the common items, position effect, and other sources of constructirrelevant changes between measurement occasions. Hence, the error due to the common-item sampling could be a major source of error for the ability estimates. In operational analyses, generally two sources of error are accounted for in variance estimation: student sampling error and measurement error. A double jackknifing procedure is proposed to include a third source of the estimation error, the error due to common-item sampling. Three different versions of the double jackknifing were implemented and compared. The data used in this study were item responses from Grade 4 students who took the NAEP 2004 and 2008 math long-term trend (LTT) assessments. These student samples used in this study are representative samples of Grade 4 student population in 2004 and in 2008 across the US. The results showed that these three double jackknifing approaches resulted in similar standard error estimates that were slightly higher than the estimates from the traditional approach, regardless of whether an item sampling scheme was used or items were dropped at random.