Educational Testing Service

This paper identifies three noncognitive domains relevant for academic achievement in K-12—student engagement, behavioral learning strategies, and school climate. The paper also documents empirical findings that show relationships between these three noncognitive domains and academic achievement, especially in the areas of reading and mathematics. Key words: K-12, noncognitive, academic achievement, reading, mathematics

Large-scale educational surveys are low-stakes assessments of educational outcomes conducted using nationally representative samples. In these surveys, students do not receive individual scores, and the outcome of the assessment is inconsequential for respondents. The low-stakes nature of these surveys, as well as variations in average performance across countries and other factors such as different testing traditions, are contributing factors to the amount of omitted responses in these assessments. While underlying reasons for omissions are not completely understood, common practice in international assessments is to employ a deterministic treatment of omissions, either as missing observations or as responses that are considered wrong. Both approaches appear problematic. In this project, we analyzed the effects of treating omitted responses either as missing or as wrong, as is done in the majority of international studies, and compared these data-treatment solutions to model-based approaches to treating omitted responses. The two types of model-based approaches used in this study are: (a) extensions of multidimensional item response theory (IRT) with an additional dimension based on response indicator variables defined and calibrated together with the set of items containing the observed responses and (b) multidimensional, multiple-group IRT models with a grouping variable representing the within-country stratification of respondents by the amount of omitted responses.

Developing a comprehensive strategy for postsecondary education that will meet the needs of America's diverse population and help ensure our ability to compete in the global economy is vital to the growth of our nation. Key to improving performance of our colleges and ...

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning.

No Child Left Behind has highlighted the need for new types of assessments that not only provide high-quality evidence about what students know and can do, but also help to move learning forward. This paper describes a linked set of formative and summative reading assessments designed to address the tradeoffs inherent in these two goals. Targeted skills include the full range of competencies underlying proficient reading at the middle-school level, including both lower-order skills such as oral reading fluency and decoding, and higher-order skills such as the ability to integrate and synthesize information from multiple texts. Data collected in pilot administrations of two prototype test forms are presented. Analyses suggest that this new approach yields acceptable measurement properties while simultaneously addressing crucial learning outcomes. This paper was presented June 23, 2009 at the National Conference on Student Assessment,

This paper presents an application of a jackknifing approach to variance estimation of ability inferences for groups of students, using a multidimensional discrete model for item response data. The data utilized to demonstrate the approach come from the National Assessment of Educational Progress (NAEP). In contrast to the operational approach used in NAEP, where plausible values are used to make ability inferences, the approach presented in this paper reestimates all parameters of the model, and makes ability inferences based on replicate samples of the jackknife without using plausible values.

The minute-to-minuteand day-by-day use of assessment for learning holds great potential to change the trajectory of studentlearning in U.S. classrooms. But without effective and scalable systems of professional development that actuallylead teachers to adopt these practices, the utility and impact of assessment for learning will be quite limited. Drawing on learning theory, expertise research, and research on effective professional development, this paper presents the theoretical and empirical basis behind an evolving program,of professional development,that employs school-embedded teacher learning communities as a central component, called Keeping Learning on Track ®

This investigation of the comparability of writing assessment prompts was conducted in two phases. In an exploratory Phase I, 47 writing prompts administered in the computer-based Test of English as a Foreign Language™ (TOEFL  CBT) from July through December 1998 were examined. Logistic regression procedures were used to estimate prompt difficulty and gender effects. A panel of experts reviewed selected prompts, and a taxonomy of prompt characteristics was developed and related to prompt difficulty and gender differences. In Phase II, 87 prompts administered from July 1998 through March 2000 were analyzed. All of the prompts used in Phase I, together with 40 new prompts, were analyzed using the larger Phase II database.

This paper outlines an approach to improving learning and teaching that combines two strong research bases: The research on formative assessment or assessment for learning provides information about what to change; research on teacher learning communities guides decisions about how to change. In this paper we describe the content and process for one model of teacher professional development. Two case studies for two school districts engaged in this process of teacher change are then presented along with descriptions of the observed effects on both the participating teachers and their larger school contexts.

Estimation of parameters of random effects models from samples collected via complex multistage designs is considered. One way to reduce estimation bias due to unequal probabilities of selection is to incorporate sampling weights. Many researchers have been proposed various weighting methods (Korn, & Graubard, 2003; Pfeffermann, Skinner, Holmes, Goldstein, & Rasbash, 1998) in estimating the parameters of hierarchical models, including random effects models as a special case. In this paper, the bias of the weighted analysis of variance (ANOVA) estimators of the variance components for a two-level, one-way random effects model is evaluated. For these estimators, analytic bias expressions are first developed, the expressions are then used to examine the impact of sample size, intraclass correlation coefficient (ICC), and the sampling design on the bias of the estimators. In addition, two-stage sampling designs are considered, with a general probability design at the first stage (Level 2) and simple random sampling without replacement (SRS) at the second stage (Level 1). The study shows that firstorder weighted variance component estimators perform well when for moderate cluster sizes and ICC values. However, noticeable estimation bias can be found with this weighting method for small cluster sizes (less than 20), particularly when ICC is small (less than 0.2). In such scenarios, scaled first-order weighted estimators can be an alternative. This paper is discussed in the context of National Assessment of Educational Progress (NAEP) 2003 4th Grade Reading National and State Assessment data, with Level 1 being the student level and Level 2 being the school level.

As part of its educational and social mission and in fulfilling the organization's non-profit Charter and Bylaws, ETS has and continues to learn from and also to lead research that furthers educational and measurement research to advance quality and equity in education and assessment for all users of the organization's products and services.

Computer simulations are one promising tool for supporting learning at all levels – from elementary school to workplace learning. Learning through simulations, either by creating simulations or by exploring existing ones, helps learners develop a deeper understanding of the ...

This report shows that the deterministic-input noisy-AND (DINA) model is a special case of more general compensatory diagnostic models by means of a reparameterization of the skill space and the design (Q-) matrix of item by skills associations. This reparameterization produces a compensatory model that is equivalent to the (conjunctive) DINA model, and is valid for all types of complex structure Q-matrices, not only for trivial cases. This equivalency uses the GDM as the basis, is not based on recent developments of diagnosis models such as G-DINA or LCDM. Model equivalency is a topic of some relevance as soon as researchers want to draw conclusions derived from any particular model-based estimates.

One of the major objectives of large-scale educational surveys is reporting trends in academic achievement. For this purpose, a substantial number of items are carried from one assessment cycle to the next. The linking process that places academic abilities measured in different assessments on a common scale is usually based on a concurrent calibration of adjacent assessments using item response theory (IRT) models. It can be conjectured that the selection of common items has a direct effect on the estimation error of academic abilities due to item misfit, small changes in the common items, position effect, and other sources of constructirrelevant changes between measurement occasions. Hence, the error due to the common-item sampling could be a major source of error for the ability estimates. In operational analyses, generally two sources of error are accounted for in variance estimation: student sampling error and measurement error. A double jackknifing procedure is proposed to include a third source of the estimation error, the error due to common-item sampling. Three different versions of the double jackknifing were implemented and compared. The data used in this study were item responses from Grade 4 students who took the NAEP 2004 and 2008 math long-term trend (LTT) assessments. These student samples used in this study are representative samples of Grade 4 student population in 2004 and in 2008 across the US. The results showed that these three double jackknifing approaches resulted in similar standard error estimates that were slightly higher than the estimates from the traditional approach, regardless of whether an item sampling scheme was used or items were dropped at random.

This paper presents a socio-cognitive framework for connecting writing pedagogy and writing assessment with modern social and cognitive theories of writing. It focuses on providing a general framework that highlights the connections between writing competency and other literacy skills; identifies key connections between literacy instruction, writing assessment, and activity and genre theories; and presents a specific proposal about how writing assessment can be organized to promote best practices in writing instruction.

Tort law offers an appropriate vehicle for handling the increased prevalence of standardized testing in the wake of the No Child Left Behind Act and the potential miss-scoring problems that arise with standardized tests. This incentivizes the use of reasonable care in scoring standardized tests and justly compensates for miss-scoring harms when doing so does not unduly burden testing agencies.  Neither contract law nor the existing Truth-in-Testing law adequately affords the sort of remedies and protections the issue of standardized testing and miss-scoring pose.

The ability for tort liability to adequately hold testing agencies accountable for miss-scoring errors and afford relief to test-takers will vary based on the actions or inactions of the testing agencies. The culpability of the testing agency and other considerations potentially give rise to actions for negligence and infliction of emotional distress, misrepresentation, defamation and false-light invasion of privacy, tortious interference with prospective advantage, or injurious falsehood—also known as disparagement. While some theories will rarely offer viable avenues for recovery, others, given particular facts, may provide a basis for relief.

Given the great, new weight placed on standardized tests in modern times, tort law posses the ability to best afford remedies to parties injured by miss-scorings on the part of testing agencies. However, relief should only be attained after a painstaking review of the facts of each case in light of principles of tort liability, seeking to incentivize the use of reasonable care in scoring standardized tests and compensating harm in meritorious cases.

In May 2016, an invitational symposium on the Reading for Understanding (RfU) initiative was held in Alexandria, VA. Co-hosted by ETS and the Council of Chief State School Officers (CCSSO), the symposium brought together some 160 state and local education leaders to examine the results of the Reading for Understanding Initiative. The goal of the Institute of Education Sciences-sponsored initiative was to accelerate the research on reading across grades preK-12. In 2010, five grant projects were awarded to focus on learning and instruction, and one project on assessment. In this policy report, the authors begin by framing the need for a focused, national effort to achieve universal, advanced reading literacy, arguing that it is an issue of equity as well as individual and national prosperity. The second portion presents a discussion of practical recommendations for lead practitioners, summarizing key insights from the RfU Research Initiative, remaining research challenges, and policy and practice recommendations for enhancing reading achievement across the educational developmental span from pre-K to secondary school graduation. The presentations of the research teams and videos of concluding panel discussions from the symposium can be found here: https://www.ets.org/research/events /reading_for_understanding_symposium. Two of the authors of this report (John Sabatini and Tenaha O'Reilly) were members of the RfU assessment project. While insiders in the RfU initiative, they were also outside observers of the other curricula and instruction-focused teams that were designing, implementing, and evaluating instructional programs in schools. This gave them a distinct perspective on how the research teams interacted with schools to develop content, support professional teacher development, and implement curriculum and instructional interventions. We would like to give special thanks to the CCSSO and the many members of their State Collaboratives on Assessment and Student Standards who participated in the conference and provided valuable insights into the major themes of this report. In addition, the CCSSO advisory group of state leaders who reviewed this paper helped us focus it on issues of high importance to state and district leaders. Their expertise and extensive experience are greatly appreciated.

This paper undertakes a review of a model developed as part of the Cognitively Based Assessment of, for, and as Learning (CBAL) initiative, an ongoing research and development project at ETS that is exploring new ways to assess students in kindergarten through Grade 12 (K -12). The model is: