Search

The "Elementary and Secondary Education Act" ("ESEA"), as amended by the "No Child Left Behind Act of 2001" inaugurated important changes in assessment and accountability for English Learner (EL) students. Specifically, Title III of the law required states to develop or adopt English-language proficiency (ELP) standards aligned with language demands of academic content standards. An annually administered ELP assessment based on those standards was also required by the "ESEA" (NCLB 2002). Title III also instituted new accountability requirements for districts and states. These new EL accountability provisions required states to define criteria for progress in learning English, establish a performance standard for English proficiency, and set annually increasing performance targets for the number and percentage of ELs meeting these criteria. As has been well documented, the new law's requirements exceeded the technical capacity of many states and districts to comply with it (Abedi 2004; Government Accountability Office 2006). Over the past several years, empirical research with more rigorous ELP assessments, systematic technical assistance efforts, and federal guidance have helped to reduce confusion and increase coherence in state Title III accountability systems (Abedi 2007; Linquanti and George 2007; Cook et al. 2008; Federal Register 2008). Nevertheless, a significant need remains to develop the capacity of state and technical assistance providers to utilize empirical data for performance standard setting and accountability policy development in these areas. This document describes and illustrates several empirical methods and conceptual or theoretical rationales to help state policy-makers, standard-setting panels, and the technical advisory panels and assistance providers to (1) determine a meaningful ELP performance standard; (2) establish a realistic, empirically anchored time frame for attaining given ELP performance standard; and (3) take into count an EL's ELP level when setting academic progress and proficiency expectations. This is "by design" a technical document intended to assist those charged with providing empirical information germane to developing or revising EL accountability models, using ELP and academic assessments. This volume does not focus on several additional basic issues around EL student achievement because there is a companion volume (Taylor, Chinen, et al., forthcoming) that analyzes similar state- and district-provided student-level longitudinal achievement data and descriptively addresses those issues. That companion volume describes (1) the heterogeneity of the EL population and the different achievement statuses and trajectories of ELs with different characteristics; (2) the estimated achievement gaps among ELs, former-ELs and non-ELs; (3) a basic description of the typical time frame for attaining English language proficiency; and (4) the nature of the relationship between assessment scores measuring language acquisition and academic-content-area learning. The chapters of this current volume represent logical extensions building on those more basic descriptive analyses. Chapter I positions data-analysis methods illustrated in the report within a larger deliberative process of setting meaningful, ambitious, and realistic performance standards and accountability criteria for EL students. The chapter offers guidelines for enacting best practices in standard setting, and highlights limitations in using empirical data. Chapter II illustrates three methods (decision consistency, logistic regression, and descriptive box plots) for analyzing empirical data to assist policymakers in determining an ELP performance standard for English Learners. These methods are used in conjunction in three states, and the results--which vary in their degree of convergence within each state--are interpreted to show how they might be utilized to support each state's decision-making process. Chapter III illustrates methods for conducting empirical analyses to inform setting expected time frames for EL students to attain the ELP performance standard. Specifically, a descriptive approach and an event history approach are applied with various adjustments. Results are compared for EL students in different grade clusters with different levels of initial English language proficiency. As students' initial ELP level influences the expected time frame for their attaining the English-proficient criterion, these data are used to illustrate the ways in which more refined time-to-English-proficiency criteria could be derived. Finally, Chapter IV discusses two methods for taking into account an EL's ELP level when setting academic progress and proficiency expectations, and one method that explicitly ignores it. First, progressive benchmarking methods are illustrated that adjust either EL students' content achievement scale scores or their weight (individual "count"), based on each student's ELP level relative to their initial ELP level and time in the state school system. Second, an indexed progress method utilizes ELs' ELP growth as a proxy for English language arts performance on a weighted, time-sensitive basis for more newly-arrived ELs who enter the state's school system at lower initial ELP levels. Third, a status and growth accountability matrix method credits both a predetermined level of student academic growth as well as attainment of academic proficiency, without considering an English Learner's ELP level. Each method is carefully described and applied using the same education agency's sample data set. All the approaches presented in this document--many of which have been employed by the principal authors in working with states on their EL accountability systems--are intended to stimulate discussion and further exploration of additional methods among state data analysts, technical assistance providers, and researchers. The ultimate goal is to support the development and regular use of empirical methods that inform ambitious, realistic, and meaningful performance standards and accountability policies, which will foster EL students' linguistic and academic progress and attainment. Appended are: (1) Decision Consistency Method; (2) Education Agency 1; (3) Education Agency 2; (4) Education Agency 3; (5) Event History Analysis; and (6) Education Agency 1. (Contains 76 exhibits and 35 footnotes.) [For "National Evaluation of Title III Implementation Supplemental Report: Exploring Approaches to Setting English Language Proficiency Performance Criteria and Monitoring English Learner Progress. Draft," see ED529268.]

This study compares alignment of the ACT and the American Diploma Project (ADP) national college readiness standards sets with the Texas Essential Knowledge and Skills for English language arts and reading (TEKS ELAR) standards for grades 9-12 and analyzes their cognitive complexity. It finds that a majority of the content in the ACT and ADP standards sets is addressed to some extent by the TEKS ELAR standards and that the TEKS ELAR standards demand higher levels of cognitive complexity than do the other two standards sets. The study has two key limitations. First, the definition of partial alignment was very broad, covering cases of just one element of an ACT or ADP statement that was addressed by a TEKS ELAR statement or statements as well as cases when all but one of multiple elements of an ACT or ADP statement were addressed. Second, the determination of content alignment and the evaluation of standards included just two dimensions for evaluating alignment (content and cognitive complexity). Other dimensions, such as breadth and specificity, might provide additional content detail that state standards writing teams or assessment writing teams could find useful. Appendices include: (1) Methodology; (2) Reviewer qualifications and roles and interrater reliability findings; (3) Examples of fully and partially aligned statements; (4) Content alignment findings by strand; (5) Nonaligned standards statements; (6) Other standards-to-standards alignment study findings; (7) Webb's cognitive complexity level descriptions and example statements; (8) Cognitive complexity by strand; and (9) Cognitive complexity comparison for fully and partially aligned statements. (Contains 2 boxes, 22 figures, 19 tables, and 20 notes.)