Search

This is the executive summary for the report, "Survey of Evidence in Education for Schools (SEE-S) Descriptive Report." In the United States, increased pressure through accountability policy and through the production and dissemination of scientifically based research are intended to create conditions for improving research use. Concurrently, researchers, education agencies, and funders have mounted efforts to strengthen relationships between research and educational practice to improve decisions about and outcomes for children. Accompanying these efforts is a need to understand, at scale, educational decision-making, and the role of research in it. To date, studies of research use in the United States have tended to focus on various stakeholders' research use (e.g., Biddle & Saha, 2002; Dagenais et al., 2012), case studies of schools or districts (e.g., Asen et al., 2013; Finnigan et al., 2013), or case studies of specific education policies or practices (e.g., Hopkins et al., 2019; Scott et al., 2017). The purpose of the report is to broadly portray research use in U.S. schools at scale to better understand where we are as an educational system in the more than forty-year journey to improve the role of research in education policy and practice. [For the full report, see ED628007. For the technical report, see ED628010.]

In this technical report, the Center for Research Use in Education (CRUE) presents the methodological design of a large-scale quantitative investigation of research use by school-based practitioners through the "Survey of Evidence in Education for Schools (SEE-S)." It documents the major technical aspects of the development of SEE-S, including item development, sample selection, and reliability and validity assessment. Descriptive statistics for data collected during the survey field trial are also detailed in this report. Through the development and validation of multiple survey measures, this study aims to deepen the fields' understanding of the actions and activities that educators are involved in concerning the use of evidence in decision-making. [For the descriptive report, see ED628007. For the executive summary, see ED628009.]

In the United States, increased pressure through accountability policy and through the production and dissemination of scientifically based research are intended to create conditions for improving research use. Concurrently, researchers, education agencies, and funders have mounted efforts to strengthen relationships between research and educational practice to improve decisions about and outcomes for children. Accompanying these efforts is a need to understand, at scale, educational decision-making, and the role of research in it. To date, studies of research use in the United States have tended to focus on various stakeholders' research use (e.g., Biddle & Saha, 2002; Dagenais et al., 2012), case studies of schools or districts (e.g., Asen et al., 2013; Finnigan et al., 2013), or case studies of specific education policies or practices (e.g., Hopkins et al., 2019; Scott et al., 2017). The purpose of this report is to broadly portray research use in U.S. schools at scale to better understand where we are as an educational system in the more than forty-year journey to improve the role of research in education policy and practice. [For the executive summary, see ED628009. For the technical report, see ED628010.]

To estimate the long-term effects of the Reading Recovery® intervention, a regression discontinuity design (RD) was implemented in a randomly selected sample of Reading Recovery schools during each year of the federally-funded i3 Scale-Up external evaluation (2011-2015) and also in one additional cohort during the 2016-17 school year. Long-term outcomes were measured by collecting scale scores on state achievement tests in reading or English language arts (ELA) that were administered to students during 3rd and 4th grades (2013-2019). We were able to record and link 3rd grade state test score data in reading/ELA for 9,879 students, and 4th grade state test data for 6,345 students. Results suggest that the long-term impact of Reading Recovery on students' reading/ELA test scores in 3rd and 4th grades is statistically significant and substantially negative; students who participated in Reading Recovery in first grade had 3rd and 4th grade state test scores in reading/ELA that were, on average, 0.19 to 0.43 standard deviations (about one-half to one full grade level) below the state test scores of similar students who did not participate in Reading Recovery.

The paper presents the results of a randomized experimental study of a writing curriculum for college developmental writing courses based on strategy instruction with self-regulation integrated with practices common in college composition. Students in a full semester course learned strategies for planning and revising based on rhetorical analysis and genres. In addition, they learned metacognitive, self-regulation strategies for goal setting, task management, self-evaluation, and reflection. A prior quasi-experiment found positive effects of the curriculum on writing quality, self-efficacy, and mastery motivation. The current study included 19 instructors and 207 students across two colleges. Using hierarchical linear modeling (HLM) with students nested within instructors and with condition and college as factors and pretest scores as covariates, analyses found positive effects of the treatment for quality of argumentative writing (ES = 1.18), quality of writing on an independent writing assessment (ES =0.67), and several motivation outcomes, including self-efficacy (for tasks and processes (ES = 0.50), for grammar (ES = 0.36), and for self-regulation (ES = 0.40)), affect (ES = 0.32), and beliefs about the importance of content (ES = 0.29). No significant effects were found for grammar/conventions or reading comprehension. Teachers in the treatment condition commented positively on the approach and noted improvements in student writing and motivation. Students also shared positive experiences and noted improvement in their writing. [The paper will be published in "Journal of Educational Psychology."]

Supporting Strategic Writers (SSW) is an instructional approach and curriculum for college developmental English and integrated reading and writing courses. SSW is based on strategy instruction integrated with practices common in college English. Students learn strategies for writing and critical reading of sources based on rhetorical analysis and genre. In addition, they learn metacognitive, self-regulation strategies such as goal setting, task management, progress monitoring, and reflection. Research over 10 years, funded by the US Department of Education, included 3 experimental studies that found large positive effects on quality of writing and motivation. The purpose of this cost analysis is to help college personnel, including both administrators and faculty, make decisions about whether to use the SSW curriculum in their developmental courses and to guide them in allocating sufficient resources for implementation.

Beginning with the "No Child Left Behind Act" of 2001 (NCLB) and reinforced by the "Education Sciences Reform Act" of 2002 (ESRA), the federal government established explicit expectations for the role of research in informing decisions about education programs, policies, and practice, and it also specified new expectations for what constituted research knowledge that was worth using. Unfortunately, it is unknown whether these changes have actually led schools and individual educators to make better and more frequent use of research to inform their decisions. This paper documents early results from a nation-wide study of when and how educators use research in decisions about policy and practice. To motivate new approaches to increasing research use and evidence-based decision-making in schools, the authors are conducting a series of research studies, the first of which involves developing and validating a survey to measure depth of research use by schools and teachers through collection of multilevel data from district staff, school administrators, and classroom teachers. In this paper, the authors present results from the first pilot of this survey with over 500 educators from 32 schools. The results presented in this paper focus on the influence of scientifically-based research and IES-funded supports for research use (e.g., WWC, RELs) on school-wide and classroom-level decisions about practice. This paper serves to highlight the likely scope and severity of the apparent divide between educators' decisions and the effectiveness research intended inform those decisions. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

This study examines and compares the recruitment, employment, and retention of minority and nonminority school teachers over the past quarter century. The objective of the study is to empirically ground the debate over minority teacher shortages. The data analyzed are from the National Center for Education Statistics' nationally representative Schools and Staffing Survey (SASS) and its longitudinal supplement, the Teacher Follow-Up Survey (TFS). Data analyses show that a gap persists between the percentage of minority students and the percentage of minority teachers in the U.S. school system. The data suggest that widespread efforts over the past several decades to recruit more minority teachers and employ them in hard-to-staff and disadvantaged schools have been very successful. However, the data also show that over the past two and a half decades, turnover rates among minority teachers have been significantly higher than among nonminority teachers. Organizational and working conditions in schools were strongly related to minority teacher departures. Once organizational conditions were held constant, there was no significant difference in the rates of minority and nonminority teacher turnover. While the number of minority teachers has increased, the schools in which they have disproportionately been employed have had, on average, less positive organizational conditions than the schools where nonminority teachers are more likely to work, resulting in disproportionate losses of minority teachers. [For the research brief, see ED606771.]

Research increasingly links low literacy levels in the early grades with a range of poor outcomes; for instance, students who read below grade level at the end of third grade are about four times less likely than their higher-achieving peers to graduate from high school (Annie E. Casey Foundation, 2010, 2011; Balfanz, Bridgeland, Bruce & Fox, 2012). In a four-year study, researchers from the Consortium for Policy Research in Education (CPRE) at the University of Pennsylvania and the Center for Research on Education and Social Policy (CRESP) at the University of Delaware examined the effectiveness of Reading Recovery at helping struggling early readers catch up. The study's findings offer promise for intensive early literacy intervention.

This brief summarizes the results from a study of the recruitment, employment, and retention of minority k-12 teachers. The study examines the extent and sources of the minority teacher shortage--the low proportion of minority teachers in comparison to the increasing numbers of minority students in the school system. Using the National Center for Education Statistics' Schools and Staffing Survey/Teacher Follow-Up Survey, we found that efforts over recent decades to recruit more minority teachers and place them in disadvantaged schools have been very successful. But these efforts have been undermined by the high turnover rates of minority teachers--largely because of poor working conditions in their schools. The conditions most strongly related to minority teacher turnover were the degree of teachers' classroom autonomy and input into school decisions--both increasingly important when coupled with accountability pressures. [For the full report, see ED606772.]

This report presents the final results of a four-year independent external evaluation of the impacts and implementation of the scale-up of Reading Recovery, a literacy intervention targeting struggling 1st-grade students. The evaluation of Reading Recovery includes parallel rigorous experimental and quasi-experimental designs for estimating program impacts, coupled with a large-scale, mixed-methods study of program implementation under the Investing in Innovation (i3) scale-up. The primary goals of the evaluation are to: (1) Provide experimental evidence of the short- and long-term impacts of Reading Recovery on student learning in schools that are part of the i3 scale-up; and (2) Assess the implementation of Reading Recovery under the i3 grant, including fidelity to the program model and progress toward the scale-up goals. The impact evaluation includes a multi-site randomized controlled trial (RCT) for estimating immediate impacts, a regression discontinuity study (RD) for estimating longterm impacts, and an implementation study for assessing fidelity of implementation and exploring program implementation in depth. The RCT includes nearly 7,000 randomized students in more than 1,200 schools over four years. The RD study measures Reading Recovery's impacts at the end of first grade and in third grade, and replicates the RCT's immediate post-treatment findings in a separate sample of students. The implementation study involves a combination of qualitative and quantitative research executed on a large scale over the same four-year timeframe. The evaluation's key findings pertain to the following topics: (1) Scale-Up Processes, Challenges, and Outcomes; (2) Immediate Impacts of Reading Recovery; (3) Sustained Impacts of Reading Recovery; and (4) Implementation Fidelity. The authors' analysis revealed strong fidelity to the program model in all of these areas and all years of the scale-up. This suggests that the intervention was delivered as designed to the students in the scale-up, and that teachers delivering Reading Recovery lessons were properly trained. In total, the results of the fidelity analysis support the validity of their impact findings. Three appendices are included. [To view the brief for this report, "Evidence for Early Literacy Intervention: The Impacts of Reading Recovery," see ED586802.]

As part of the 2010 economic stimulus, a $55 million "Investing in Innovation" (i3) grant from the US Department of Education was awarded to scale up Reading Recovery across the nation. This paper presents the final round of results from the large-scale, mixed methods randomized evaluation of the implementation and impacts of Reading Recovery under this i3 Scale-Up. Over 10,000 low performing first-grade students from more than 1,000 schools participated in this study from 2010-2015. The evaluation design for the i3 scale-up of Reading Recovery includes a rigorous experimental research design (i.e., a multi-site randomized controlled trial) that supports strong causal inferences about program impacts, coupled with mixed methods descriptions of program implementation and contextual factors. Short-term impacts on students' reading performance are estimated by comparing mid-year reading achievement of students randomly assigned to participate in reading recovery at the beginning of first grade to students randomly assigned to the control condition. The consistently large positive impacts of Reading Recovery under the i3 scale-up suggest that this relatively large investment has led to substantial improvements in the reading performance of many thousands of students across the nation. In addition, results from the mixed-methods implementation evaluation revealed numerous programmatic and contextual factors that can support or hinder implementation of Reading Recovery by individual schools and may explain variation in impacts across sites.

Reading Recovery is a short-term early intervention designed to help the lowest-achieving readers in first grade reach average levels of classroom performance in literacy. Students identified to receive Reading Recovery meet individually with a specially trained Reading Recovery teacher every school day for 30-minute lessons over a period of 12 to 20 weeks. The purpose of these lessons is to support rapid acceleration of each child's literacy learning. In 2010, The Ohio State University received a Scaling Up What Works grant from the U.S. Department of Education Investing in Innovation (i3) Fund to expand the use of Reading Recovery across the country. The award was intended to fund the training of 3,675 new Reading Recovery teachers in U.S. schools, thereby expanding service to an additional 88,200 students. The Consortium for Policy Research in Education (CPRE) was contracted to conduct an independent evaluation of the i3 scale-up of Reading Recovery over the course of five years. The evaluation includes parallel rigorous experimental and quasi-experimental designs for estimating program impacts, coupled with a large-scale mixed-methods study of program implementation. This report presents the findings of the second year of the evaluation. The primary goals of this evaluation are: (1) to provide experimental evidence of the impacts of Reading Recovery on student learning under this scale-up effort; (2) to assess the success of the scale-up in meeting the i3 grant's expansion goals; and (3) to document the implementation of the scale-up and fidelity to program standards. This document is the second in a series of three reports based on the external evaluation of the Reading Recovery i3 Scale-Up. The report represents results from the impact and implementation studies conducted over the 2012-2013 school year--the third year of the scale-up effort and the second full year of the evaluation. [This document is a reissue of the December 2014 CPRE Research Report RR-79 under the same title. For the Year One Report, see ED547669.]

This study addresses the question: Do the kinds and amounts of pre-service education and preparation that beginning teachers receive before they start teaching have any impact on whether they leave teaching? We examine a wide range of measures of teachers' subject-matter education and pedagogical preparation. We also compare different fields of teaching, with a particular focus on mathematics and science. Our data source is the National Center for Education Statistics' nationally representative 2003-04 Schools and Staffing Survey and its supplement, the 2004-05 Teacher Follow-up Survey. Our analyses show that beginning teachers widely varied in the pre-service education and preparation they received. In general, mathematics teachers and, especially, science teachers tended to have more subject-matter content education and more graduate-level education, and to have less pedagogical and methodological preparation than other teachers. Our analyses also show that, after controlling for the background characteristics of teachers and their schools, some aspects of the education and preparation that beginning teachers received were significantly associated with their attrition, while others were not. Specifically, the type of college, degree, entry route or certificate mattered little. What did matter was the substance and content of new teachers' pedagogical preparation. Those with more training in teaching methods and pedagogy--especially practice teaching, observation of other classroom teaching and feedback on their own teaching--were far less likely to leave teaching after their first year on the job.

With the goal of increasing students' academic readiness for college, high schools in the United States are increasingly offering "credit-based transition programs," including International Baccalaureate (IB), Advanced Placement (AP), and dual enrollment. Existing research points to the promise of IB, AP, and other credit-based transition programs for improving students' academic readiness for college; however, conclusions about program effects are often limited by potential issues of selection bias. Despite strong statistical controls and assumptions to address selection, observational research may not be able to determine whether differences in outcomes are caused by program participation or are simply an artifact of the unmeasured characteristics of schools, students, and families that correlate with the decision to participate in these optional programs. To address this knowledge gap and inform future studies of the impacts of IB and other credit-based transition programs, this paper makes three contributions: (1) A review of existing literature is used to produce an empirically-based conceptual model of selection into IB; (2) The conceptual model is used to identify the characteristics of students and schools that participate in the IB Diploma Programme using data from the National Center for Education Statistics and the Florida Education Data Warehouse. The conceptual model also allows researchers to identify key predictors for which there are no data available; and (3) A test of the ability of the available data to adjust for observed selection bias using propensity score methods, with the degree of bias reduction reported for each predictor is described. The data used in this study come from the Florida K-20 Education Data Warehouse (FL-EDW) and the U.S. Department of Education's Common Core of Data (CCD). The subset of data from FL-EDW has student-level records for 20,373 students who participated in an IB Diploma Programme and graduated between 2002 and 2007, and student-level records for 86,008 randomly sampled students who did not participate in an IB Diploma Programme and graduated over the same time period. This study reveals that, when looking at the statewide population in Florida, the selection bias associated with voluntary participation in IB is very large, and that mechanisms for dealing with selection bias using propensity scores may not be sufficient. Tables and figures are appended.

Interest in variation in program impacts--How big is it? What might explain it?--has inspired recent work on the analysis of data from multi-site experiments. One critical aspect of this problem involves the use of random or fixed effect estimates to visualize the distribution of impact estimates across a sample of sites. Unfortunately, unless the sample sizes within sites are enormous (e.g., >1,000), the shape of the distribution of estimates is dominated by the normal distribution as a result of imperfect reliability in site-specific estimates. The Central Limit Theorem explains why this occurs and why the problem is unavoidable. The purpose of this paper is to demonstrate the scope and severity of this problem, and to provide cautionary guidance regarding the interpretation of distributions of site-specific impact estimates in multi-site experiments. Furthermore, the paper explores methods for recovering the distribution of true site-specific impacts when that distribution is non-normal. For illustrative purposes, data from the multi-site randomized trial under the Reading Recovery Investing in Innovation (i3) Scale-Up is used in addition to generalized Monte Carlo simulation. Reading Recovery is an early reading intervention for first-graders who are reading substantially below grade level. Data figures are appended.

As a critical step in understanding the impacts of IB, the analyses presented in this report examined the selection mechanisms behind IB participation across Florida, the state with the second highest representation of IB programs in the nation. We use longitudinal student and school-level data from 1995 through 2009 from the Florida K-20 Education Data Warehouse (EDW) to characterize individual students' educational histories from elementary school through high school and into college. To address issues of selection bias, we use propensity score methods (Rosenbaum & Rubin, 1983) to adjust for preexisting differences between IB and non-IB students. These analyses are designed to address the following research questions: (1) What are the student- and school-level predictors of participating in the IB Diploma Programme in Florida?; (2) To what degree does propensity score stratification or matching reduce selection bias associated with key student and school-level factors?; (3) What are the estimated differences in key postsecondary access indicators (i.e., SAT and ACT scores) and enrollment statistics (e.g., college selectivity) with and without different types of propensity score adjustments? Results revealed that, when looking at the statewide population in Florida, the selection bias associated with voluntary participation in IB is very large, and that mechanisms for dealing with selection bias using propensity scores may not be sufficient.

Reading Recovery (RR) is a short-term early intervention designed to help the lowest-achieving readers in first grade reach average levels of classroom performance in literacy. Students identified to receive Reading Recovery meet individually with a specially trained Reading Recovery (RR) teacher every school day for 30-minute lessons over a period of 12 to 20 weeks. The purpose of these lessons is to support rapid acceleration of each child's literacy learning. In 2010, The Ohio State University received a Scaling Up What Works grant from the U.S. Department of Education's Investing in Innovation (i3) Fund to expand the use of Reading Recovery across the country. The award was intended to fund the scale-up of Reading Recovery by training 3,675 new RR Teachers in U.S. schools, thereby expanding capacity to allow service to an additional 88,200 students. The Consortium for Policy Research in Education (CPRE) was contracted to conduct an independent evaluation of the i3 scale up of Reading Recovery over the course of five years. The evaluation includes parallel rigorous experimental and quasi-experimental designs for estimating program impacts, coupled with a large-scale mixed-methods study of program implementation under the i3 scale-up. This report presents findings through the second year of the evaluation. The primary goals of this evaluation were: (1) to assess the success of the scale-up in meeting the i3 grant's expansion goals; (2) to document the implementation of scale-up and fidelity to program standards; and (3) to provide experimental evidence of the impacts of Reading Recovery on student learning under this scale-up effort. This document is the first in a series of three annual reports produced based on our external evaluation of the Reading Recovery i3 Scale-Up. This report presents early results from the experimental impact and implementation studies conducted over the 2010-11 and 2011-12 school years. An appendix includes: Statistical Model for Impacts of Reading Scores. [For "WWC Review of the Report 'Evaluation of the i3 Scale-up of Reading Recovery Year One Report, 2011-12.' What Works Clearinghouse Single Study Review," see ED547670.]

State assessments are increasingly used as outcome measures for education evaluations. The scaling of state assessments produces variability in measurement error, with the conditional standard error of measurement increasing as average student ability moves toward the tails of the achievement distribution. This report examines the variability in pretest-posttest correlation coefficients of state assessment data for samples of low-performing, average-performing, and proficient students to illustrate how sample characteristics (including the measurement error of observed scores) affect pretest-posttest correlation coefficients. As an application, this report highlights how statistical power can be attenuated when correlation coefficients vary according to sample characteristics. Population achievement data from four states and two large districts in both English/Language Arts and Mathematics for three recent years are examined. The results confirm that pretest-posttest correlation coefficients are smaller for samples of low performers, reducing statistical power for impact studies. We also find substantial variation across state assessments. These findings suggest that it may be useful to assess the pretest-posttest correlation coefficients of state assessments for an intervention's target population during the planning phase of a study. Population descriptive statistics for ELA and Mathematics assessments for all eligible students in available years are appended. (Contains 26 tables, 8 figures and 23 footnotes.)

This study examines and compares the recruitment and retention of minority and White elementary and secondary teachers and attempts to empirically ground the debate over minority teacher shortages. The data we analyze are from the National Center for Education Statistics' nationally representative Schools and Staffing Survey and its longitudinal supplement, the Teacher Follow-up Survey. Our data analyses show that a gap continues to persist between the percentage of minority students and the percentage of minority teachers in the U.S. school system. But this gap is not due to a failure to recruit new minority teachers. Over the past two decades, the number of minority teachers has almost doubled, outpacing growth in both the number of White teachers and the number of minority students. Minority teachers are also overwhelmingly employed in public schools serving high-poverty, high-minority and urban communities. Hence, the data suggest that widespread efforts over the past several decades to recruit more minority teachers and employ them in hard-to-staff and disadvantaged schools have been very successful. This increase in the proportion of teachers who are minority is remarkable because the data also show that over the past two decades, turnover rates among minority teachers have been significantly higher than among White teachers. Moreover, though schools' demographic characteristics appear to be highly important to minority teachers' initial employment decisions, this does not appear to be the case for their later decisions to stay or depart. Neither a school's poverty-level student enrollment, a school's minority student enrollment, a school's proportion of minority teachers, nor whether the school was in an urban or suburban community was consistently or significantly related to the likelihood that minority teachers would stay or depart, after controlling for other background factors. In contrast, organizational conditions in schools were strongly related to minority teacher departures. Indeed, once organizational conditions are held constant, there was no significant difference in the rates of minority and White teacher turnover. The schools in which minority teachers have disproportionately been employed have had, on average, less positive organizational conditions than the schools where White teachers are more likely to work, resulting in disproportionate losses of minority teachers. The organizational conditions most strongly related to minority teacher turnover were the level of collective faculty decision-making influence and the degree of individual classroom autonomy held by teachers; these factors were more significant than were salary, professional development or classroom resources. Schools allowing more autonomy for teachers in regard to classroom issues and schools with higher levels of faculty input into school-wide decisions had far lower levels of turnover. (Contains 6 figures, 10 tables and 7 endnotes.) [Funding for this paper was provided by the Center for Educational Research in the Interest of Underserved Students, University of California, Santa Cruz and the Sally Hewlett and the Flora Family Foundation.]

This study examines the magnitude, destinations, and determinants of the departures of mathematics and science teachers from public schools. The data are from the National Center for Education Statistics' nationally representative Schools and Staffing Survey and its longitudinal supplement, the Teacher Follow-up Survey. Our analyses show that rates of mathematics and science teacher turnover, both those moving between schools and those leaving teaching altogether, have increased over the past two decades, but have not been consistently different than those of non-mathematics/science teachers. Mathematics and science teachers who left teaching were also no more likely than other teachers to take non-education jobs, such as in technological fields, or to be working for private business or industry. The data also show that, like other teachers, there are large school-to-school differences in mathematics and science turnover. High poverty, high minority, and urban public schools have among the highest mathematics and science turnover levels. In the case of cross-school migration, the data show there is an annual asymmetric reshuffling of a significant portion of the mathematics and science teaching force from poor to not poor schools, from high-minority to low-minority schools, and from urban to suburban schools. However, our multivariate analyses showed that a number of key organizational characteristics and conditions of schools accounted for these school differences in turnover. The strongest factors for mathematics teachers were the degree of individual classroom autonomy held by teachers, the provision of useful professional development, and the degree of student discipline problems. For science teachers, the strongest factors were the maximum potential salary offered by school districts, the degree of student discipline problems in schools, and useful professional development. (Contains 3 figures, 8 tables and 8 endnotes.)

The objective of this study is to use nationally representative data to examine the rates, destinations and sources of math and science teacher turnover. There are four sets of research questions the authors address concerning the departure of mathematics and science teachers from their schools: (1) At what rates do mathematics/science teachers move from or leave their teaching jobs? How do their turnover rates compare to those of other teachers? Have their turnover rates changed over time? What is the magnitude of both math science teacher mobility and attrition?; (2) Which types of schools have higher levels of mathematics/science teacher turnover?; (3) What are the destinations of mathematics/science teachers who move from or leave their teaching jobs? What proportions of those departing move to other schools, quit to raise families, go to graduate school, go into non-teaching occupations within education (e.g., school administration, higher education, etc) or go into non-educational occupations?; and (4) Which particular factors, aspects, and conditions of schools and of teachers' jobs are most tied to the turnover of mathematics and science teachers? This study utilizes nationally representative data on teachers, and therefore the setting is the whole United States. Findings suggest that schools are not simply victims of inexorable demographic trends and that there is a significant role for the management of schools in both the genesis of, and solution to, school staffing problems. The data suggest that improvements in these conditions of the teaching job, such as increased teacher salaries, and enhanced faculty input into school decision making, would all contribute to lower rates of turnover, in turn, diminish school staffing problems, undermine the so-called teacher shortage and, hence, ultimately, aid the performance of schools.

The purpose of this study is to provide empirical benchmarks of the conditional reliabilities of state tests for samples of the student population defined by ability level. Given that many educational interventions are targeted for samples of low performing students, schools, or districts, the primary goal of this research is to determine how severe the attenuation in conditional reliability is, depending on the test, subject matter, and average ability level of the students in the research sample. This objective is addressed, along with implications for future research, in research questions 1 and 2 below: (1) What are the conditional reliabilities (defined as the median and 95% CI for pretest-posttest correlations) of given state tests for subsets of the population defined by achievement level?; and (2) What are the implications for power analysis when state assessments are the outcomes of interest for samples of students taken from the tails of the achievement distribution? The results of research question 1 will be used to explore the implications of conditional reliability in the prospective power analysis for a study using state assessments as outcomes. This study demonstrates that the reliability of these assessments may be compromised when low and/or high ability segments of the student population are to be sampled for the intervention. The simulation results indicate that if a study were to be performed on a sample of low performing students (students 2 standard deviations below the mean), the use of a state test would be far less appropriate than if the intervention was focused on students whose ability levels were closer to the average ability level of the population as a whole. These findings have important implications for prospective power analysis and the precision of impact estimates for randomized trial designs. (Contains 1 table and 3 figures.)

Securing data on students' academic achievement is typically one of the most important and costly aspects of conducting education experiments. As state assessment programs have become practically universal and more uniform in terms of grades and subjects tested, the relative appeal of using state tests as a source of study outcome measures has grown. However, the variation in state assessments--in both content and proficiency standards--complicates decisions about whether a particular state test is suitable for research purposes and poses difficulties when planning to combine results across multiple states or grades. This discussion paper aims to help researchers evaluate and make decisions about whether and how to use state test data in education experiments. It outlines the issues that researchers should consider, including how to evaluate the validity and reliability of state tests relative to study purposes; factors influencing the feasibility of collecting state test data; how to analyze state test scores; and whether to combine results based on different tests. It also highlights best practices to help inform ongoing and future experimental studies. Many of the issues discussed are also relevant for nonexperimental studies. Appendices include: (1) State Testing Programs Under NCLB; (2) How NCEE-Funded Evaluations Use State Test Data. (Contains 35 footnotes and 4 tables.)

This study investigates the impact of America's Choice on student writing performance in Georgia. The analyses in this study focus on the change that occurred during the first year of implementation, the 2001-2002 school year. Two research questions guided these analyses: (1) What effect did Georgia's Choice have on the writing scores from the state assessment? and (2) To what extent is the effect of Georgia's Choice related to implementation of specific components of the design? The data sources for this study were individual-level student test scores on the Georgia writing assessments, school context measures (e.g., poverty rate, ethnic composition, etc.), and implementation data collected via a survey of cluster leaders, school principals, and the school leadership teams. This report is organized into six sections. In the next section, some background on Georgia's Choice and the rollout of the design during the first year is presented. Then it describes the data sources and sample. Next, it describes the methods, and then it presents the results of the analyses. Finally, it offers some interpretations of the results and explores the implications of these results for Georgia schools, America's Choice, and for comprehensive school reform in general. Appended are: (1) Scales of Implementation; (2) Results from Statistical Models of Impact; and (3) Results from Statistical Models Linking Implementation Measures and Impact. (Contains 6 tables and 3 footnotes.)

The Third International Mathematics and Science Study (TIMSS) contains no direct measures of student ability and motivation. Researchers created a new variable, Student Task Persistence (STP), that is an index of student engagement in providing answers to TIMSS questions on the background questionnaire. The purpose of this research was to investigate the possibility that cross-national differences in mathematics and science achievement can be partly explained by STP. Data came from the TIMSS Student Questionnaire and the TIMSS achievement tests that were administered to national probability samples at grades 3, 4, 7, 8, and the final year of secondary school. Multilevel analyses were undertaken to study the relationships between STP and achievement at student, classroom, and national levels. Evidence suggests that the STP variable is important for two reasons. First, it represents the only available indicator of student ability and motivation to perform the tasks required by TIMSS Second, it is one of the strongest predictors of national differences in mathematics and science achievement. At the eighth grade level, at least, national mean STP scores accounted for well over half of the variation in national mean mathematics and science achievement scores, thereby leaving less than half the national-level variation in achievement to be accounted for by other factors such as student knowledge of mathematics and science content. Two appendixes contain survey questions to be asked after the Student Background Questionnaire and the achievement tests. (Contains 5 tables and 16 references.) (SLD)

This report provides feedback on the implementation of America's Choice school design in a Florida school district. It seeks evidence of the impacts of the design, focusing on the literacy workshops. The evaluation asks three central questions: Is America's Choice being carried out in the manner envisioned? Are instructional practices of teachers changing in ways that would improve student learning? To what degree can improvements in student achievement be attributed to the design? To answer these questions, researchers compared the performance of students in America's Choice to that of students in other schools in the Duval County school district in northeastern Florida. The data show that students in the America's Choice schools regularly outgained students in other district schools in writing and, to a lesser extent, reading and mathematics. Detectable patterns of effect were strongest in writing; students in America's Choice schools consistently demonstrated higher learning gains than students in other schools in the district. The results were most dramatic in fourth grade. Overall, across grades and subjects, the America's Choice design had an equal effect on white and minority students. In 22 percent of the grades and subjects examined, America's Choice was associated with a statistically significant reduction in the gap in performance between white and minority students by increasing the performance of minority students at a higher rate than white subjects. Appended are: (1) Writing Results by Year, Grade, and Cohort; (2) Reading Results by Year and Grade for Cohort II; (3) Reading and Mathematics Results by Year and Grade for Cohort III; (4) Predicted Writing Performance Gains for White and Minority Students in America's Choice and Other District Schools; (5) Predicted Reading Performance Gains for White and Minority Students in America's Choice and Other District Schools; and (6) Predicted Mathematics Performance Gains for White and Minority Students in America's Choice and Other District Schools. (RJM)