Search

Implementation analyses conducted as part of impact studies can help educators know whether a tested intervention is likely to be a good fit for their own settings. This guide can help researchers design and conduct these kinds of analyses. The guide provides steps and recommendations about ways to specify implementation research questions, assess whether and how the planned intervention is implemented, document the context in which the intervention is implemented, and measure the difference between the intervention and what members of the control group receive. It presents strategies for analysis and reporting about these topics, and also for linking implementation and impact findings. The guide offers key definitions, examples, templates, and links to resources.

BASIE is a framework for interpreting impact estimates from evaluations. It is an alternative to null hypothesis significance testing. This guide walks researchers through the key steps of applying BASIE, including selecting prior evidence, reporting impact estimates, interpreting impact estimates, and conducting sensitivity analyses. The guide also provides conceptual and technical details for evaluation methodologists.

Open science envisions that researchers will make their study data available to other investigators to facilitate research transparency and accelerate the development of knowledge. This guide describes key issues that education researchers should consider when deciding which study data to share, how to organize the data, what documentation to include, and where to share their final dataset. The guide also provides strategies for addressing related challenges and includes links to other resources, a checklist aligned to each section, and appendices that contain templates and samples.

This guide will help researchers design and implement impact studies in education so that the findings are more generalizable to the study's target population. Guidance is provided on key steps that researchers can take, including defining the target population, selecting a sample of schools--and replacement schools, when needed--managing school recruitment, assessing, and adjusting for differences between the sample and target population, and reporting information on the generalizability of the study findings.

In 2017, the Office of Planning, Research, and Evaluation (OPRE) in the Administration for Children and Families (ACF) funded Mathematica to conduct the Infant and Toddler Teacher and Caregiver Competencies (ITTCC) project. The project aims to examine existing efforts across states, institutions of higher education, professional organizations, and early care and education (ECE) programs related to competencies for infant and toddler (I/T) teachers and caregivers who work in group settings and build a conceptual foundation to inform future measurement, research, and evaluation. This compendium describes the results of a scan of competency-aligned measures. The purpose of this compendium is to offer the early childhood education field a sense of existing options for assessing competencies for I/T teachers and caregivers who work in group care settings (centers and family child care homes). The measures included in this compendium were identified through prior activities undertaken for the ITTCC project, including the scan of existing competency frameworks and the literature review, and based on additional expert input. For each measure, the authors developed a profile that includes key information about the focus of the measure and how it has been or can be used. Although these profiles are not exhaustive, they include important information about the specific competencies the measures address and how they have been used to assess the competencies of I/T teachers and caregivers.

Head Start is a national program that promotes school readiness by enhancing the social and cognitive development of children through the provision of educational, health, nutritional, social, and other services to enrolled children and families. The Head Start program provides comprehensive child development services to economically disadvantaged children and families through grants to local public and private nonprofit and for-profit agencies. This report includes key information on the Head Start Family and Child Experiences Survey 2014-2018 (FACES 2014) study design; in addition, a set of data tables presents descriptive statistics for the characteristics of programs, centers, classrooms, and teachers serving Head Start children and families in spring 2017. The purpose of this report is twofold: (1) to provide information about the FACES study, including the background, design, methodology, measures, and analytic methods; and (2) to report detailed descriptive statistics and related standard errors in a series of tables on the programs, their staff, and classrooms. The data provide descriptive information from classroom observations and staff surveys about Head Start's efforts to help children and families meet their goals, and local efforts to meet the Head Start Program Performance Standards. [For "A Portrait of Head Start Classrooms and Programs in Spring 2015: FACES 2014-2015 Data Tables and Study Design. OPRE Report 2017-101," see ED592749.]

Underlying all What Works Clearinghouse (WWC) products are WWC Study Review Guides, which are intended for use by WWC certified reviewers to assess studies against the WWC evidence standards. As part of an ongoing effort to increase transparency, promote collaboration, and encourage widespread use of the WWC standards, the Institute of Education Sciences provides external users with access to a web-based WWC Study Review Guide for conducting reviews of group design studies, including randomized controlled trials and quasi-experimental designs. Reviewers use the Study Review Guide to document the characteristics of studies, including features that pertain to a study's eligibility under a WWC protocol. The Study Review Guide assists the reviewer in assessing the study design and implementation against the WWC standards, and coding the study findings in a systematic manner consistent with WWC reporting guidelines. The WWC provides a separate Excel-based tool for conducting reviews of single-case design studies. This document guides users through the public-use version of the WWC Study Review Guide, available at https://ies.ed.gov/ncee/wwc/wwcsrgpublic. Members of the public are invited to use the public-use version of the SRG to understand how WWC reviewers document the characteristics of a study, determine its eligibility, and assess its design and implementation against the WWC standards. [Cover title varies: "WWC Web-Based Study Review Guide User Guide."]

This report includes key information on the Head Start Family and Child Experiences Survey 2014-2018 (FACES 2014) study design and a set of data tables presents descriptive statistics for the characteristics of classrooms, teachers, centers, and programs serving Head Start children and families in spring 2015. Data are drawn from the spring 2015 round FACES 2014. FACES which was first launched in 1997 as a periodic, longitudinal study of program performance. The study is conducted by Mathematica Policy Research and its partners--Educational Testing Service and Juárez and Associates--under contract to the Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. The purpose of this report is two-fold: (1) to provide information about the FACES study, including the background, design, methodology, measures, and analytic methods, and (2) to report detailed descriptive statistics and related standard errors in a series of tables on the programs, their staff, and classrooms. The data provide descriptive information from classroom observations and staff surveys about Head Start's efforts to help children and families meet their goals and local efforts to meet the Head Start Program Performance Standards. The data tables provide descriptive information on Head Start classrooms, teachers, centers, and programs. The FACES sample provides information at the national level about Head Start programs, centers, classrooms, and the children and families they serve. We selected a sample of Head Start programs from the 2012-2013 Head Start Program Information Report, with two centers per program and two classrooms per center selected for participation. One-hundred seventy-six programs, 346 centers, and 667 classrooms participated in the study in spring 2015. The statistics found in these tables are estimates of key characteristics of Head Start teachers, classrooms, centers, and programs in spring 2015. Teacher data on teacher characteristics are weighted to represent all teachers in Head Start. Teacher data that describe Head Start classrooms and classroom observation data are weighted to represent all Head Start classrooms. Director survey data are weighted to represent all Head Start programs or centers.

Children from low-income families are more likely than those from higher income families to have poor social, emotional, cognitive, behavioral, and health outcomes. One approach that has helped parents and their young children is home visiting, which provides information, resources, and support to expectant parents and families with young children. This brief summarizes evidence from existing studies on the impact of early childhood home visiting on children 5 and older for four national models of home visiting. The primary research questions include:(1) What are the effects of home visiting programs for families as children get older?; (2) How do the monetary benefits of home visiting compare with their costs?; and (3) How do the monetary benefits of home visiting compare with their costs?

The goal of this practice guide is to offer educators specific, evidence-based recommendations that address the challenges of preventing dropout in secondary schools. This guide synthesizes the best publicly available research and shares practices that are supported by evidence. It is intended to be practical and easy for teachers and school leaders to use. The guide includes many examples in each recommendation to demonstrate the concepts discussed. Throughout the guide, examples, definitions, and other concepts supported by evidence are indicated by endnotes within the example title or content. For examples that are supported by studies that meet What Works Clearinghouse design standards, the citation in the endnote is in bold text. Examples without specific citations were developed in conjunction with the expert panel based on their experience, expertise, and knowledge of the related literature. This guide is targeted to school and district administrators, as well as members of student-support teams including school counselors, social workers, psychologists, and teachers. It provides recommendations that can be implemented in conjunction with existing academic curricula and student-support services. This guide contains appendices.

It is widely recognized that the 12 percent of all youth in American public schools who have disabilities comprise a set of students with distinct capacities and needs. Federal legislation, including the most recent updates to the Individuals with Disabilities Education Act (IDEA) in 2004, identifies different disability groups and mandates that students in each group have access to a free and appropriate public education. How youths' characteristics, experiences, and challenges vary by disability group remains of interest, particularly given the changing educational, social, and economic landscape that might affect youth with different disabilities in different ways. The National Longitudinal Transition Study (NLTS) 2012 provides updated information on youth with disabilities in light of these changes, to inform efforts to address their needs. Sponsored by the U.S. Department of Education under a congressional mandate to study IDEA 2004 and the students it serves, the NLTS 2012 describes the backgrounds of secondary school youth and their functional abilities, activities in school and with friends, academic supports received from schools and parents, and preparation for life after high school. Through surveys in 2012 and 2013, the study collected data on a nationally representative set of nearly 13,000 students-mostly those with an individualized education program (IEP) and expected to receive special education services. The study also includes students without an IEP, who either have no identified disability or who have an impairment that does not qualify them for special education but allows them to receive accommodations through a 504 plan under the Rehabilitation Act, another federal law pertaining to the rights and needs of youth with disabilities. This second volume of findings from the NLTS 2012 focuses on youth with an IEP only and the similarities or dissimilarities across 12 disability groups defined by IDEA 2004. The assessment of diversity among the disability groups in the decade following IDEA 2004 suggests several key points: (1) Youth with intellectual disability and emotional disturbance are the most socioeconomically disadvantaged groups and the most likely to attend lower-performing schools; (2) Difficulties with health, communication, and functioning independently are most prevalent among youth with autism, intellectual disability, multiple disabilities, and orthopedic impairments; (3) The groups that most commonly face health and functional challenges are also less engaged with friends and in school activities, but youth with emotional disturbance are most likely to get into trouble; (4) Youth with autism, intellectual disability, and multiple disabilities are most likely to receive academic modifications but least likely to receive some other forms of academic support; and (5) The same three groups--youth with autism, intellectual disability, and multiple disabilities--are least likely to take steps to prepare for college and employment. These findings highlight some differences in the challenges that youth with an IEP faced in the decade after IDEA 2004, depending on their disability. Although the characteristics and experiences described capture only a subset of those discussed in this volume, prior research suggests that they could be important indicators of students' later outcomes. The following are appended: (1) Technical notes and methodology for volume 2: Comparisons across disability groups; (2) Detailed tables for chapter 2 of volume 2: Comparisons across disability groups; (3) Detailed tables for chapter 3 of volume 2: Comparisons across disability groups; (4) Detailed tables for chapter 4 of volume 2: Comparisons across disability groups; (5) Detailed tables for chapter 5 of volume 2: Comparisons across disability groups; and (6) Detailed tables for chapter 6 of volume 2: Comparisons across disability groups. [For the Executive Summary, see ED573353; For Volume 1, see ED573341.]

In response to the recession that began in 2007, the U.S. Congress passed, and President Barack Obama signed into law, the American Recovery and Reinvestment Act of 2009 (Pub. Law 111-5). At an estimated cost of $831 billion, this economic stimulus package sought to save and create jobs, provide temporary relief to those adversely affected by the recession, and invest in education, health, infrastructure, and renewable energy. States and school districts received $100 billion to secure teachers' jobs and promote innovation in schools. This funding included $3 billion for School Improvement Grants (SIG), one of the Obama administration's signature programs and one of the largest federal government investments in an education grant program. The SIG program awarded grants to states that agreed to implement one of four school intervention models--transformation, turnaround, restart, or closure-in their lowest-performing schools. Each of the models prescribed specific practices designed to improve student outcomes, including outcomes for high-need students such as English language learners (ELLs) (U.S. Department of Education 2010a). Given the importance of the SIG program and sizable investment in it, the Institute of Education Sciences (IES) commissioned this evaluation to focus on four primary questions: (1) Did schools implementing a SIG-funded model use the improvement practices promoted by SIG, and how did that compare to use of those practices by schools not implementing a SIG-funded model?; (2) Did use of SIG-promoted practices include a focus on ELLs, and did that focus on ELLs differ between schools implementing a SIG-funded model and schools not implementing one?; (3) Did receipt of SIG funding to implement a school intervention model have an impact on outcomes for low-performing schools?; and (4) Was the type of school intervention model implemented related to improvement in outcomes for low-performing schools? This is the final report for this evaluation of SIG. This final report builds on the earlier briefs and report by including an additional year of data (spring 2013) and by examining whether receipt of SIG funding had an impact on student outcomes. The findings in this report suggest that the SIG program did not have an impact on the use of practices promoted by the program or on student outcomes (including math or reading test scores, high school graduation, or college enrollment), at least for schools near the SIG eligibility cutoff. The following are appended: (1) RDD Impact Analysis; (2) Correlational Analysis; (3) Additional Figures Based on School Surveys; (4) District Reported Practices Aligned with the SIG Application Criteria; (5) Detailed Findings from Interviews and Surveys; (6) Survey Questions Aligned with SIG Practices; and (7) Additional Information about English Language Learner-Focused Implementation Analysis. [For the Executive Summary, "School Improvement Grants: Implementation and Effectiveness: Executive Summary. NCEE 2017-4012," see ED572213.]

The goal of this practice guide is to offer educators specific, evidence-based recommendations that address the challenges of teaching students in grades 6-12 to write effectively. This guide synthesizes the best publicly available research and shares practices that are supported by evidence. It is intended to be practical and easy for teachers to use. The guide includes many examples in each recommendation to demonstrate the concepts discussed. Throughout the guide, examples, definitions, and other concepts supported by evidence are indicated by endnotes within the example title or content. For examples that are supported by studies that meet WWC design standards, the citation in the endnote is bolded. Examples without specific citations were developed in conjunction with the expert panel based on their experience, expertise, and knowledge of the related literature. This guide provides secondary teachers in all disciplines and administrators with instructional recommendations that can be implemented in conjunction with existing standards or curricula. Teachers can use the guide when planning instruction to support the development of writing skills among students in grades 6-12 in diverse contexts. The following are appended: (1) Postscript from the Institute of Education Sciences; (2) About the Authors; (3) Disclosure of Potential Conflicts of Interest; and (4) Rationale for Evidence Ratings.

The goal of this practice guide is to offer educators specific, evidence-based recommendations for teaching foundational reading skills to students in kindergarten through 3rd grade. This guide is a companion to the existing practice guide, "Improving Reading Comprehension in Kindergarten Through 3rd Grade", and as a set, these guides offer recommendations for preparing students to be successful readers. Both guides recommend some practices that can and should be implemented beginning in kindergarten, and both guides also suggest some instructional practices that can be implemented after students have mastered early reading skills. This guide synthesizes the best available research on foundational reading skills and shares practices that are supported by evidence. It is intended to be practical and easy for teachers to use. The guide includes many examples in each recommendation to demonstrate the concepts discussed. This guide provides teachers, reading coaches, principals, and other educators with instructional recommendations that can be implemented in conjunction with existing standards or curricula and does not recommend a particular curriculum. Teachers can use the guide when planning instruction to support the development of foundational reading skills among students in grades K-3 and in diverse contexts. Professional-development providers, program developers, and researchers can also use this guide. Professional-development providers can use the guide to implement evidence-based instruction and align instruction with state standards or to prompt teacher discussion in professional learning communities. Program developers can use the guide to create more effective early-reading curricula and interventions. Finally, researchers may find opportunities to test the effectiveness of various approaches to foundational reading education and explore gaps or variations in the reading-instruction literature. The following are appended: (1) Postscript from the Institute of Education Sciences; (2) About the Authors; (3) Disclosure of Potential Conflicts of Interest; and (4) Rationale for Evidence Ratings. A glossary is included. [For the companion guide, "Improving Reading Comprehension in Kindergarten through 3rd Grade: IES Practice Guide. NCEE 2010-4038," see ED512029.]

This report describes in detail how researchers, policymakers, and program administrators can recognize opportunities for experiments and carry them out. Specifically, the report focuses on opportunistic experiments, defined as randomized controlled trials (RCTs) that study the effects of initiatives, program changes, or policy actions that agencies or programs plan or intend to implement -- as opposed to studying an intervention or policy action that is developed and implemented specifically for a research study. [This report was adapted from "Recognizing and Conducting Opportunistic Experiments in Education: A Guide for Policymakers and Researchers. REL 2014-037" (ED544801).]

Online charter schools--also known as virtual charters or cyber charters--are publicly funded schools of choice that eschew physical school buildings and use technology to deliver education to students in their own homes. These schools typically provide students with computers, software, and network-based resources, while also providing access to teachers via email, telephone, web, and/or teleconference. Online charter schools deliver instruction using a radically different approach than conventional public schools. Nonetheless, critics of online charter schools worry that they might not be effective in promoting student learning. This report and its companion volumes describe the findings of the most ambitious and comprehensive study of online charter schools to date, conducted jointly by Mathematica Policy Research, the Center for Research on Education Outcomes (CREDO) at Stanford University, and the Center on Reinventing Public Education (CRPE) at the University of Washington. This volume begins with a snapshot of online charter schools operating across the country, describing their numbers, the states in which they operate, and the students they serve. The report then describes the instructional programs of online charter schools; methods used to engage students and parents, along with expectations of parental involvement; the teachers and principals of online charter schools; and their management and governance. Key findings include: (1) Student-driven, independent study is the dominant mode of learning in online charter schools, with 33 percent of online charter schools offering only self-paced instruction; (2) Online charter schools typically provide students with less live teacher contact time in a week than students in conventional schools have in a day; (3) Maintaining student engagement in this environment of limited student-teacher interaction is considered the greatest challenge by far, identified by online charter school principals nearly three times as often as any other challenge; (4) Online charter schools place significant expectations on parents, perhaps to compensate for limited student-teacher interaction, with 43, 56, and 78 percent of online charters at the high school, middle, and elementary grade levels, respectively, expecting parents to actively participate in student instruction; and (5) These findings suggest reason for concern about whether the online charter school sector is likely to be effective in promoting the achievement of its students. Appended are: (1) Survey Methods; and (2) Supplemental Table.

Mastering algebra is important for future math and postsecondary success. Educators will find practical recommendations for how to improve algebra instruction in the What Works Clearinghouse (WWC) practice guide, "Teaching Strategies for Improving Algebra Knowledge in Middle and High School Students". The methods and examples included in the guide focus on helping students analyze solved problems, recognize structure, and utilize alternative approaches to solving algebra problems. Each recommendation includes the level of supporting research evidence behind it, examples to use in class, and solutions to potential implementation roadblocks. Teachers can implement these strategies in conjunction with existing standards or curricula. In addition, these strategies can be utilized for all students learning algebra in grades 6-12 and in diverse contexts, including during both formative and summative assessment. Administrators and professional development providers can use the guide to implement evidence-based instruction and align instruction with state standards or to prompt teacher discussion in professional learning communities. Appended are: (1) Postscript from the Institute of Education Sciences; (2) About the Authors; (3) Disclosure of Potential Conflicts of Interest; and (4) Rationale for Evidence Ratings. A Glossary is included.

Technology continues to radically change how we create and consume information. Today, news, reports, and other material are often delivered quickly through pictures, colors, or other eye-catching visual elements. Words still matter, but they may be tweeted, viewed on a smartphone, or placed in a call-out box in a report. The design of these items can greatly affect whether your reader notices, reads, or understands the words that you write. This guide offers a basic overview on how researchers can effectively use design to create engaging and visually appealing Regional Educational Laboratory (REL) products. It will cover some key concepts behind good design and discuss how to use basic elements like photographs, images, color, tables, figures, and type to create useful publications and digital products. The guide also touches on how researchers can use data visualization to make complex concepts accessible.

PowerPoint presentations, Excel documents, Web pages, and videos posted on federal websites must meet the requirements of the 1998 amendment to Section 508 of the Rehabilitation Act. Doing so ensures that these materials are accessible to a wide range of people with disabilities, including visual, auditory, sensory, and motor impairments. These individuals use assistive technology to help them understand electronic information--for example, screen readers and text-to-speech software interpret words on the page and translate them to a computerized voice that reads the information. Accessible products work hand in hand with assistive technology. However, accessibility requirements are broad and lack step-by-step recommendations to walk the reader through the process. There is no "magic bullet" software that creates compliant files at the press of a button, so accessibility usually involves setting up files from the get-go in a structured and systematic way by using built-in features of the native software application. This is especially important with large, multi-author files, which may have complex production requirements and deadlines. For guidelines on setting up accessible templates, go to http://www.hhs.gov/web/508/accessiblefiles/index.html. The goal of this guide is to offer tips and tools for those who have a basic understanding of 508 compliance to help ease the process and reduce headaches and delays.

The Supplemental Nutrition Assistance Program (SNAP) serves as the foundation of America's national nutrition safety net. It is the nation's first line of defense against food insecurity and offers a powerful tool to improve nutrition among low-income individuals. SNAP is the largest of the 15 domestic food and nutrition assistance programs administered by the U.S. Department of Agriculture's (USDA's) Food and Nutrition Service (FNS). This report describes the characteristics of SNAP households and participants nationwide in fiscal year 2012 (October 2011 through September 2012). It also presents an overview of SNAP eligibility requirements and benefit levels in fiscal year 2012. The appendices provide detailed tabulations of household and participant characteristics for the nation and by State, as well as a brief description of the sample design and the sampling error associated with the estimates presented in the report.

Posting a report on the web or sending out a press release used to be all that was needed to help get the word out about one's research findings. But times have changed. Today, effective dissemination depends on getting the right message to the right people at the right time in the right form. To break through the clutter, researchers need the right elements: a compelling story, engaging visuals, and a set of electronic tools, including email, video, and social media. A successful newsflash--an email blast announcing new research findings--incorporates all of these elements. The succinct newsflash format is also perfect for smaller devices like tablets and smart phones, which are becoming increasingly important for delivering information. This guide provides tips for researchers on how to quickly and effectively inform decision makers, stakeholders, and practitioners through the use of newsflashes.

The Knowledge Is Power Program (KIPP) is a rapidly expanding network of public charter schools whose mission is to improve the education of low-income children. As of the 2012-2013 school year, 125 KIPP schools are in operation in 20 different states and the District of Columbia (DC). Ultimately, KIPP's goal is to prepare students to enroll and succeed in college. Prior research has suggested that KIPP schools have positive impacts on student achievement, but most of the studies have included only a few KIPP schools or have had methodological limitations. This is the second report of a national evaluation of KIPP middle schools being conducted by Mathematica Policy Research. The evaluation uses experimental and quasi-experimental methods to produce rigorous and comprehensive evidence on the effects of KIPP middle schools across the country. The study's first report, released in 2010, described strong positive achievement impacts in math and reading for the 22 KIPP middle schools for which data were available at the time. For this phase of the study, the authors nearly doubled the size of the sample, to 43 KIPP middle schools, including all KIPP middle schools that were open at the start of the study in 2010 for which they were able to acquire relevant data from local districts or states. The average impact of KIPP on student achievement is positive, statistically significant, and educationally substantial. KIPP impact estimates are consistently positive across the four academic subjects examined, in each of the first four years after enrollment in a KIPP school, and for all measurable student subgroups. A large majority of the individual KIPP schools in the study show positive impacts on student achievement as measured by scores on state-mandated assessments. KIPP produces similar positive impacts on the norm-referenced test, which includes items assessing higher-order thinking. Estimated impacts on measures of student attitudes and behavior are less frequently positive, but the authors found evidence that KIPP leads students to spend significantly more time on homework, and that KIPP increases levels of student and parent satisfaction with school. On the negative side, the findings suggest that enrollment in a KIPP school leads to an increase in the likelihood that students report engaging in undesirable behavior such as lying to or arguing with parents. These findings are described in this report. The following appendixes are included: (1) Sample selection and baseline characteristics; (2) Constructing survey outcomes; (3) Schools attended by lottery winners and lottery non-winners; (4) Analytic methods for the matched comparison group analysis; (5) Analytic methods for lottery-based analysis; and (6) Validation of matching methods using lottery-based impact estimates. (Contains 46 tables, 78 footnotes, and 16 figures.) [For "What Works Clearinghouse Quick Review: 'KIPP Middle Schools: Impacts on Achievement and Other Outcomes, Final Report,'" see ED540896.]

Recess periods often lack the structure needed to support physical activity and positive social development (Robert Wood Johnson Foundation 2010). The Playworks program places full-time coaches in low-income schools to provide opportunities for organized play during recess and throughout the school day. Playworks activities are designed to engage students in physical activity, foster social skills related to cooperation and conflict resolution, improve students' ability to focus on class work, decrease behavioral problems and improve school climate. The Robert Wood Johnson Foundation (RWJF) contracted with Mathematica Policy Research and its subcontractor, the John W. Gardner Center for Youth and Their Communities (JGC) at Stanford University, to conduct a rigorous evaluation of Playworks. Twenty-five schools interested in implementing Playworks were randomly assigned to a treatment group that received Playworks in the 2010-2011 school year or to a control group that was not eligible to implement Playworks until the following year. The authors collected data from students, teachers and school staff in spring 2011 to document the implementation of Playworks and assess the impact of the program on key outcomes in six domains: (1) school climate, (2) conflict resolution and aggression, (3) learning and academic performance, (4) recess experience, (5) youth development and (6) student behavior. Ultimately, four additional schools will be added to the study and further analyses will be released. A description of study design and data sources. (Contains 4 exhibits, 15 tables and 2 endnotes.)

The Commonwealth of Pennsylvania plans to develop a new statewide evaluation system for teachers and principals in its public schools by school year 2013-2014. To inform the development of this evaluation system, the Team Pennsylvania Foundation (Team PA) undertook the first phase of the Pennsylvania Teacher and Principal Evaluation Pilot--henceforth referred to as Phase 1--2010 and 2011 in collaboration with a broad stakeholder group that included leaders from the Pennsylvania Department of Education (PDE), the Pennsylvania State Education Association (PSEA), school districts, and the business community. The purpose of Phase 1 was to develop and implement a pilot set of performance measures to obtain lessons for improving the use of classroom observations and student data in evaluating teacher and principal performance. None of the results from Phase 1 had a bearing on actual evaluations or personnel decisions for any teacher or principal. This report presents findings for the second track of Phase 1. In this track, Mathematica Policy Research used student-level data to develop value-added models (VAMs) for estimating teacher and principal effectiveness. VAMs estimate the effects of educators on student achievement growth. VAMs belong to the class of models that are generally referred to as student growth models, but a VAM estimate is not a measure of student growth; rather, it is an estimate of an educator's or a school's "contribution" to student growth. VAMs can be appropriate for use in teacher or principal evaluations because they produce information about educator effectiveness. Other indicators like student proficiency rates and descriptive measures of student growth might be appropriate as targets for school accountability purposes, but they should not be viewed as indicating what a teacher or school has contributed to student learning. Appended are: (1) Technical Specifications of the VAMS; (2) Data Sources and Sample Characteristics; and (3) Technical Results from Value-Added Analyses. (Contains 34 tables, 9 figures and 52 footnotes.) [For the "Value-Added Estimates for Phase 1 of the Pennsylvania Teacher and Principal Evaluation Pilot. Executive Summary", see ED531803. This report was supported by the Team Pennsylvania Foundation.]

In 2007, using funds from the federal Teacher Incentive Fund (TIF) and private foundations, the Chicago Public Schools (CPS) began piloting its version of a schoolwide reform model called the Teacher Advancement Program (TAP). Under the TAP model, teachers can earn extra pay and take on increased responsibilities through promotion (to mentor teacher or master teacher), and they become eligible for annual performance bonuses based on a combination of their contribution to student achievement (known as "value added") and observed performance in the classroom. The model calls for weekly meetings of teachers and mentors ("cluster groups"), and regular classroom observations by a school leadership team to help teachers meet their performance goals. The idea behind TAP is that giving teachers performance incentives, along with tools to track their performance and improve instruction, will help schools attract and retain talented teachers and help all teachers raise student achievement. This report is the last in a series of reports providing evidence on the impacts of CPS' version of TAP, called "Chicago TAP." It presents findings from the four-year implementation period, with special emphasis on the 2009-2010 and 2010-2011 school years, the third and fourth years of the program's rollout in Chicago. Earlier reports (Glazerman et al. 2009; Glazerman and Seifullah 2010) provide detailed data on each of the first two years of the program, respectively. CPS implemented Chicago TAP as a pilot program intended for 40 high-need schools. The program began in 10 schools in the first year (cohort 1) with a rollout plan to add 10 more Chicago TAP schools (cohorts 2, 3, and 4) in each year of the TIF grant's four-year implementation period. The authors address three research questions regarding Chicago TAP: (1) How was the program implemented?; (2) What impact did the program have on student achievement?; and (3) What impact did the program have on teacher retention within schools? To assess the first year under Chicago TAP for schools that began the program in fall 2009 (cohort 3), the authors looked at how teacher development and compensation practices in Chicago TAP schools differ from practices normally implemented in CPS schools. The authors found that teachers in Chicago TAP schools reported receiving significantly more mentoring support than teachers in similar non-TAP (control) schools. This finding reflects the fact that under the Chicago TAP model, teachers are guided by mentor teachers, and cluster groups meet weekly. They also found that veteran teachers in Chicago TAP schools were more likely than their control group counterparts to provide mentoring support to their colleagues; this finding is consistent with the fact that under Chicago TAP, teachers have the opportunity to assume leadership roles and responsibilities as Chicago TAP mentor or lead teachers. Teachers in Chicago TAP schools (veteran and novice) were aware of their eligibility for performance-based compensation. The authors found that the amount of compensation they expected approached the amount that was eventually paid out; that is, the average expectation was about $900, and the actual amount paid out in bonuses to this group was an average of about $1,100 per teacher. They generally did not find evidence of an impact of Chicago TAP on teacher attitudes or school climate. While the introduction of Chicago TAP led to real changes inside the schools, the program did not consistently raise student achievement as measured by growth in Illinois Standards Achievement Test (ISAT) scores. The authors found evidence of both positive and negative test score impacts in selected subjects, years, and cohorts of schools, but overall there was no detectable impact on math, reading, or science achievement that was robust to different methods of estimation. For example, impacts on science scores overall (across years and cohorts) were positive, but not statistically significant unless they used one particular matching method that excluded some Chicago TAP schools from the analysis. The authors did find evidence suggesting that Chicago TAP increased schools' retention of teachers, although the impacts were not uniform or universal across years, cohorts, and subgroups of teachers. They found that teachers who were working in Chicago TAP schools in 2007 returned in each of the following three years at higher rates than teachers in comparable non-TAP schools. For example, the authors found that 67 percent of classroom teachers in cohort 1 schools in fall 2007 returned to their same school in fall 2010 compared to about 56 percent of teachers in non-TAP schools, an impact of nearly 12 percentage points. In other words, teachers in Chicago TAP schools in fall 2007 were about 20% more likely than teachers in comparison schools to be in those same schools three years later. When the authors looked at teachers who were working in schools that started Chicago TAP in later years, some of the impact estimates were not statistically significant. The authors also found some evidence of impacts on retention for subgroups of teachers, such as those with less experience, but the pattern of findings was not consistent. When they considered retention of teachers in the district, the authors did not find consistent evidence of a measurable impact. Given that Chicago TAP is a school-specific program, their main focus was on school-level retention, as opposed to retention in the district. Appended are: (1) Propensity Score Matching; and (2) Supplemental Tables. (Contains 32 tables, 6 figures and 27 footnotes.) [For related reports, see "An Evaluation of the Teacher Advancement Program (TAP) in Chicago: Year One Impact Report. Final Report" (ED507502) and "An Evaluation of the Teacher Advancement Program (TAP) in Chicago: Year Two Impact Report" (ED510712).]

At the request of Pittsburgh Public Schools (PPS) and the Pittsburgh Federation of Teachers (PFT), Mathematica has developed value-added models (VAMs) that aim to estimate the contributions of individual teachers, teams of teachers, and schools to the achievement growth of their students. The authors' work in estimating value-added in Pittsburgh supports the larger, joint efforts of PPS and the PFT to "empower effective teachers" through evaluation, professional development, and compensation. Pittsburgh's VAMs use not only state assessments but also course-specific assessments, student attendance, and course completion rates, thereby aiming to produce estimates of the contributions of teachers and schools that are fair, valid, reliable, and robust. The findings in this report suggest that the VAM estimates provide meaningful information about teacher and school performance in Pittsburgh. The VAM results for individual schools and teachers have been reported to them privately. The first three chapters of this report describe the student outcomes that are used in Pittsburgh's VAMs (Chapter II); enumerate the information on students that is used to predict their performance and account for factors outside the control of the teacher or school (Chapter II); discuss the technical details of the VAMs (Chapter III); and explain some of the limitations of the VAMs (Chapter IV). The last four chapters of the report explain how VAMs for each student outcome are combined to create a series of composite measures for each school (Chapter V); describe the process for locating the performance of Pittsburgh's schools in the statewide distribution of value-added (Chapter VI); present summary statistics related to VAM results for Pittsburgh schools and teachers (Chapter VII); and discusses the application of VAMs for use in two programs designed to recognize and reward outstanding performance: the Promise-Readiness Corps and Students and Teachers Achieving Results (STAR) (Chapter VIII). (Contains 15 tables and 6 figures.)

Charter schools--public schools of choice that are operated autonomously, outside the direct control of local school districts--have become more prevalent over the past two decades. There is no consensus about whether, on average, charter schools are doing better or worse than conventional public schools at promoting the achievement of their students. Nonetheless, one research finding is clear: Effects vary widely among different charter schools. Many educators, policymakers, and funders are interested in ways to identify and replicate successful charter schools and help other public schools adopt effective charter school practices. Charter-school management organizations (CMOs), which establish and operate multiple charter schools, represent one prominent attempt to bring high performance to scale. The National Study of CMO Effectiveness aims to fill the gap in systematic evidence about CMOs, providing the first rigorous nationwide examination of CMOs' effects on students' achievement and attainment. The study includes an examination of the relationships between the practices of individual CMOs and their effects on student achievement, with the aim of providing useful guidance to the field. This updated edition of the report provides key findings from the study on CMO practices, impacts, and the relationships between them. A forthcoming report will explore promising practices in greater depth. This study uses multiple data sources to describe CMOs, assess their impacts on students, and identify practices associated with positive impacts in order to address the following research questions: (1) How quickly are CMOs growing? Which students and areas do they serve and what resources do they use? What are the practices and structures of CMOs? What state policies and other factors appear to influence the location and growth of CMOs?; (2) What are the impacts of CMOs on student outcomes and to what extent do these impacts vary across CMOs?; and (3) Which CMO practices and structures are positively associated with impacts? To examine eligible CMOs and address the research questions, the authors conducted a survey of CMO central office staff, surveys of CMO principals and principals in nearby conventional public schools, a survey of CMO teachers, and site visits to 10 CMOs and 20 schools. In addition, they collected and analyzed school records with data on student characteristics and outcomes (including test scores), and they examined CMO financial records and business plans. Findings include: (1) Comprehensive behavior policies are positively associated with student impacts; (2) Intensive teacher coaching is positively associated with student impacts; (3) CMOs using TFA and teaching fellow teachers have higher impacts, but other staffing decisions are not associated with impacts; (4) CMOs categorized as "data-driven" and "time on task" have larger impacts, on average, than two other categories of CMOs; and (5) Tightness of CMO management is weakly associated with impacts. As is often the case in studies of this kind, some of the interesting findings raise other important questions. The following questions are discussed in this report: (1) To what extent do CMOs produce positive effects on longer term student outcomes?; (2) What explains why some CMOs have negative impacts on test scores?; (3) Which promising strategies should CMOs implement and how should they implement them?; (5) To what extent do CMOs add value compared to independent charter schools?; (6) Are new CMOs using the same strategies and producing the same impacts as more established CMOs?; and (7) What other factors might contribute to CMO impacts? Appended are: (1) Construction and Analysis of Measures Used in Chapter III; (2) Experimental Impacts; (3) Validation of Impact Estimation Approach; (4) Methodology for Estimating CMO and School- Level Impacts on Achievement in Middle- Schools; (5) Baseline Equivalence; (6) Method for Dealing with Grade Repetition; (7) Methodology and Results for CMO Impacts on High School Achievement and Attainment; (8) Impacts on Middle School Test Scores by CMO, Year, and Subject; (9) Comparing CMO and Independent Charter Impacts; (10) Subgroup Impacts; (11) Multiple Comparison Adjustments for Impact Analyses; (12) Methods for Correlating Impacts and CMO Characteristics; and (13) Correlational Analysis Results. Individual chapters contain footnotes. (Contains 56 tables and 49 figures.) [This document was commissioned by NewSchools Venture Fund and written with assistance from Michael Barna, Emily Caffery, Hanley Chiang, John Deke, Melissa Dugger, Emma Ernst, Alena Davidoff-Gore, Eric Grau, Thomas Decker, Mason DeCamillis, Philip Gleason, Amanda Hakanson, Jane Nelson, Antoniya Owens, Julie Redline, Davin Reed, Chris Rodger, Margaret Sullivan, Christina Tuttle, Justin Vigeant, Tiffany Waits, and Clare Wolfendale. For an earlier edition of this report, "The National Study of Charter Management Organization (CMO) Effectiveness. Charter-School Management Organizations: Diverse Strategies and Diverse Student Impacts," see ED526951.]

Charter schools--public schools of choice that are operated autonomously, outside the direct control of local school districts--have become more prevalent over the past two decades. There is no consensus about whether, on average, charter schools are doing better or worse than conventional public schools at promoting the achievement of their students. Nonetheless, one research finding is clear: Effects vary widely among different charter schools. Many educators, policymakers, and funders are interested in ways to identify and replicate successful charter schools and help other public schools adopt effective charter school practices. Charter-school management organizations (CMOs), which establish and operate multiple charter schools, represent one prominent attempt to bring high performance to scale. Many CMOs were created in order to replicate educational approaches that appeared to be effective, particularly among disadvantaged students. Attracting substantial philanthropic support, CMO schools have grown rapidly from encompassing about 6 percent of all charter schools in 2000 to about 17 percent of a much larger number of charter schools by 2009 (Miron 2010). Some of these organizations have received laudatory attention through anecdotal reports of dramatic achievement results. The National Study of CMO Effectiveness aims to fill the gap in systematic evidence about CMOs, providing the first rigorous nationwide examination of CMO achievement effects. The study includes an examination of the relationships between the practices of individual CMOs and their effects on student achievement, with the aim of providing useful guidance to the field. Mathematica Policy Research and the Center on Reinventing Public Education (CRPE) are conducting the study with funding from the Bill & Melinda Gates Foundation and the Walton Family Foundation, and project management assistance from the NewSchools Venture Fund. This report provides key findings from the study on CMO practices, impacts, and the relationships between them. Additional reports will explore promising practices in greater depth and examine longer-term impacts of CMOs on high school graduation and college entry. Appended are: (1) Construction and Analysis of School Practice Measures Used in Chapter III; (2) Validation of The Quasi- Experimental Methods in Experimental Sites; (3) Propensity Score Matching Method; (4) Baseline Equivalence of Student Comparison and Treatment Groups; (5) Method for Dealing With Grade Repeaters; (6) Multiple Comparison Adjustments for Impact Analyses; (7) Impacts by Year and Subject; (8) Impact Estimates for independent Charter Schools; (9) Subgroup Impacts; (10) Methods for Correlating Impacts and Practices; and (11) Detailed Results from Correlation of Impacts and Practices. (Contains 11 tables and 46 figures.) [This report was written with assistance from Michael Barna, Emily Caffery, Hanley Chiang, John Deke, Melissa Dugger, Emma Ernst, Alena Davidoff-Gore, Eric Grau, Thomas Decker, Mason DeCamillis, Philip Gleason, Amanda Hakanson, Jane Nelson, Antoniya Owens, Julie Redline, Davin Reed, Chris Rodger, Margaret Sullivan, Christina Tuttle, Justin Vigeant, Tiffany Waits and Clare Wolfendale. For related report, "The National Study of Charter Management Organization (CMO) Effectiveness. Report on Interim Findings," see ED516865.]

The Knowledge Is Power Program (KIPP) is a network of charter schools designed to improve the educational opportunities available to low-income families. KIPP schools seek to boost their students' academic achievement and, ultimately, prepare them to enroll and succeed in college. This paper provides the most thorough analysis to date of the characteristics of students entering and departing KIPP schools, making use of longitudinal, student-level data from 22 KIPP middle schools and district schools in their surrounding communities. In Chapter II, the authors show that, on average, KIPP students are more likely to be black or Hispanic and have lower incomes than students in the surrounding school districts, but they are less likely to be English language learners or students with disabilities. Chapter III addresses attrition from KIPP schools. Students who transfer out of KIPP schools tend to have lower achievement than students who remain, but this is also true for nearby district schools: transferring students tend to be more disadvantaged than persisting students, regardless of what schools they attend. Chapter IV describes patterns of late arrival across the 22 KIPP middle schools in their sample. Supplemental tables are appended. (Contains 5 tables, 5 figures and 16 footnotes.)

The American Institutes for Research (AIR) and Mathematica Policy Research (MPR) developed this guide to help you consider evaluation issues likely to arise as you launch ARRA-funded initiatives and other educational reform activities. Many states are already involved in evaluation, so many of the ideas presented here may be familiar. The authors hope that this guide will provide additional information and straightforward strategies to help you integrate evaluation into your educational reform efforts. The American Recovery and Reinvestment Act (ARRA) is providing states like yours and districts important opportunities to strengthen efforts to improve educational outcomes for all children. This guide does not address all the ways in which you may be using ARRA and other funds to support educational reforms or all the approaches you could use to evaluate ARRA-funded and other educational initiatives in your state. Instead, the guide aims to help you think about opportunities for building evaluation into your ongoing efforts and define your evaluation priorities. Throughout the guide, the authors' intent is to provide easy-to-use tools and resources--including tips and checklists--that you can use as part of an evaluation of most of the educational reform strategies that you are likely to initiate. These tips suggest ways in which you can reduce the need for special data collection efforts and make it easier to combine information from various data sources by building on existing data systems. (Contains 13 figures, 3 tables and 5 footnotes.)

The Social Security Administration is funding a random assignment evaluation of six demonstration projects to improve employment and other outcomes for youth ages 14 to 25 who are either receiving disability benefits or are at high risk of receiving them in the future. This report reviews the Youth Transition Demonstration (YTD) projects, located in Buffalo, New York, including findings from a process analysis on implementation, as well as an analysis of one-year impacts on youth's use of services, employment and earnings, income and benefits, educational attainment, and attitudes and expectations. Appended are: (1) Additional Analyses and Technical Discussion; (2) Supporting Tables for Chapter III; and (3) The SSA Waivers for YTD. (Contains 60 tables, 10 figures, and 124 footnotes.) [This report was submitted to the Social Security Administration's Office of Program Development and Research.]

The Head Start Family and Child Experiences Survey (FACES), sponsored by the U.S. Department of Health and Human Services, Administration for Children and Families (ACF), was first launched in 1997 as a periodic longitudinal study of program performance. This report is the fourth in a series that uses data from the FACES 2006 cohort to describe the population of 3- and 4-year-olds who entered Head Start for the first time in fall 2006, their families, and their classrooms. Guided by the FACES conceptual framework (Figure 1), earlier reports documented the diversity in the Head Start population in terms of demographic and socioeconomic characteristics, the skills that children have when they first enter the program, and the gains in these skills over one or two years of program participation. The current report describes the group of children who first entered Head Start in fall 2006 either as a 3- or 4-year-old, completed one or two years in the program, and attended kindergarten the year after graduating from Head Start. As in the earlier reports, the authors profile the demographic characteristics of this group and describe their home and family life, drawing comparisons where appropriate to the characteristics of the population of children and families when they first entered Head Start or after completing one year in the program. New to this report is a description of the schools and kindergarten classrooms Head Start graduates attend. The authors describe broad characteristics of their schools such as size, student body composition, and school type. They describe children's kindergarten classrooms and teachers, including information on characteristics such as the length of the school day (full- versus half-day kindergarten), class size, child-to-staff ratio, and teachers' experience and degrees. They once again document children's gains in a broad set of skills from program entry to Head Start graduation and to the end of the kindergarten year, and investigate the associations between children's skills when entering and leaving Head Start, their skills at the end of Head Start, and their progress through the spring of their kindergarten year. The findings in the report are intended to answer five research questions: (1) What are the child/family demographics and home environment characteristics of children who complete Head Start and enroll in kindergarten? How involved are their parents in their schools and education?; (2) What are the characteristics of the schools and kindergarten programs children attend after completing Head Start? What are the characteristics of their kindergarten classrooms and teachers?; (3) What developmental gains do children make during Head Start and beyond? How do their skills compare to those of their peers; (4) Are children's school readiness skills at the end of Head Start related to developmental outcomes at the end of kindergarten? Are there cross-domain relationships between children's language, literacy, math, and social-emotional skills?; and (5) What child/family and Head Start characteristics relate to children's development at the end of Head Start and the gains they make from the time they enter Head Start through the spring of kindergarten? Does their growth in school readiness skills vary by their skills when first entering Head Start? The remainder of the report is organized into six sections. First, the authors provide background on the study methodology and sample. Second, they offer information on children's characteristics, family demographics, and home life, including language background, educational environment of the home, family routines, and socioeconomic risk status. They include information on parents' involvement with their children's elementary schools, the level of satisfaction with their children's schools, and parents' beliefs about how well Head Start prepared their children for kindergarten. Third, they describe the schools Head Start children attend for kindergarten, their kindergarten classrooms, and their teachers. They include information on the background of the children in their classrooms as well as educational experiences in the classroom. Fourth, they chronicle children's developmental progress from the time they completed Head Start through the end of kindergarten, considering whether these outcomes vary by gender, race/ethnicity, or risk status. Fifth, they explore the associations between children's school readiness skills as they complete Head Start and their developmental outcomes at the end of kindergarten. Sixth, they investigate associations of child/family and Head Start characteristics with children's development at the end of Head Start and their developmental progress from Head Start entry to the end of kindergarten. They also explore the relationship of children's relative skills at program entry (that is, low, average, or high ability) to their development progress during this time period. (Contains 2 tables, 26 figures and 70 endnotes.) [For related report, "The Data Tables for FACES 2006: Head Start Children Go to Kindergarten. ACF-OPRE Report", see ED517212.]

The Head Start Family and Child Experiences Survey (FACES), sponsored by the U.S. Department of Health and Human Services, Administration for Children and Families (ACF), was first launched in 1997 as a periodic longitudinal study of program performance. Successive nationally representative samples of Head Start children and their families, classrooms, and programs provide descriptive information on the population served; staff qualifications, credentials, beliefs, and opinions; classroom practices and quality measures; and child and family outcomes. FACES includes a battery of direct child assessments across multiple domains. It also includes interviews with the child's parents, teachers, and program managers, as well as direct observations of classroom quality. This set of tables is designed to accompany a research brief that describes the group of children who first entered Head Start in fall 2006 either as a 3- or 4-year-old, completed one or two years in the program, and attended kindergarten the year after graduating from Head Start. Head Start Children Go to Kindergarten profiles the demographic characteristics of this group and describes their home and family life. It includes a description of the schools and kindergarten classrooms Head Start graduates attend. The report documents children's gains in a broad set of skills from program entry to Head Start graduation and to the end of the kindergarten year, and investigates the associations between children's skills when entering and leaving Head Start, their skills at the end of Head Start, and their progress through the spring of their kindergarten year (West et al. 2010b). Following an introduction to the study methodology and sample, the tables in the first section provide information on the children's characteristics, family demographics, and home life, including language background, educational environment of the home, family routines, and socioeconomic risk status. They include information on parents' involvement with their children's elementary schools, the level of satisfaction with their children's schools, and parents' beliefs about how well Head Start prepared their children for kindergarten. In the second set of tables, the authors provide information about the schools Head Start children attend for kindergarten, their kindergarten classrooms, and their teachers. The authors include information on the background of the children in their classrooms as well as educational experiences in the classroom. The third set of tables chronicles children's developmental progress from the time they completed Head Start through the end of kindergarten. In the final two sections, the authors explore (1) the associations between children's school readiness skills as they complete Head Start and their developmental outcomes at the end of kindergarten and (2) the associations of child/family and Head Start characteristics with children's development at the end of Head Start and their developmental progress from Head Start entry to the end of kindergarten. They also explore the relationship of children's relative skills at program entry (that is, low, average, or high ability) to their development progress during this time period. (Contains 109 tables and 18 notes.) [For related report, "Head Start Children Go to Kindergarten. ACF-OPRE Report", see ED517211.]

The Head Start Family and Child Experiences Survey (FACES) was first launched in 1997 as a periodic longitudinal study of program performance. Successive nationally representative samples of Head Start children, their families, classrooms, and programs provide descriptive information on the population served; staff qualifications, credentials, beliefs, and opinions; classroom practices and quality measures; and child and family outcomes. FACES includes a battery of direct child assessments across multiple domains. It also comprises interviews with the child's parents, teachers, and program managers, as well as direct observations of classroom quality. FACES is a tool for measuring Head Start program performance at the national level. This recurring data collection provides the means to assess program performance both currently and over time. This set of tables is designed to accompany a research brief which profiles the second year in the program for 3-year-old Head Start children and families who were newly enrolled in fall 2006 (ACF 2010b) and are still attending in spring 2008. FACES selects two groups of first-time enrollees--those entering at age 4 and those entering at age 3--who are expected to attend Head Start for one or two years, respectively, prior to kindergarten entry. The 3-year-old group is of particular interest for several reasons: (1) as the Head Start Program Information Report (PIR) shows, 3-year-olds occupy a growing share of the total population served by Head Start, increasing from 24 percent in 1980 to 40 percent in 2007 (ACF 2010a); (2) they may differ in important characteristics from children who enter at age 4 in terms of developmental level and exposure to prior care experiences; and (3) they have the potential to continue in Head Start for two program years or to leave for another prekindergarten experience. Following an introduction to the study methodology and sample, the tables in the first section provide information on the children's characteristics, family demographics, and home life, including language background, educational environment of the home, family routines, and socioeconomic risk status. The authors also include information on parent involvement in Head Start and their level of satisfaction with their own and their children's Head Start experiences. In the next set of tables, the authors provide information on children's developmental progress over two years of Head Start, including whether these outcomes vary by gender, race/ethnicity, or risk status. It is important to note that changes in children's skills and development during their program experience reflect a range of influences in their lives, including child-level characteristics, such as maturation and health status, as well as community, program, classroom, peer, and family influences. (Contains 55 tables and 15 notes.) [For related report, "A Second Year in Head Start: Characteristics and Outcomes of Children Who Entered the Program at Age Three. ACF-OPRE Report", see ED517208.]

The Head Start Family and Child Experiences Survey (FACES) was first launched in 1997 as a periodic longitudinal study of program performance. Successive nationally representative samples of Head Start children, their families, classrooms, and programs provide descriptive information on the population served; staff qualifications, credentials, beliefs and opinions; classroom practices and quality measures; and child and family outcomes. FACES includes a battery of direct child assessments across multiple domains. It also comprises interviews with the child's parents, teachers and program managers, as well as direct observations of classroom quality. This set of tables is designed to accompany a research brief which profiles the 3- and 4-year-old Head Start children and families who were newly enrolled in the program in fall 2006 and are still attending in spring 2007 (see Aikens et al. 2010). Following an introduction to the study methodology and sample, the tables in the first section provide information on the children's characteristics, family demographics, and home life, including language background, educational environment of the home, family routines, and socioeconomic risk status in spring 2007. These tables also include information on parent involvement in Head Start and level of satisfaction with their own and their children's Head Start experiences. The next sections, on cognitive and social-emotional/health outcomes in spring 2007, chronicle children's developmental progress over the Head Start year. They examine whether these outcomes vary by age, gender, race/ethnicity, or risk status. The following section presents the characteristics of their teachers and classrooms, including measures of observed quality in spring 2007. Subsequent sections provide information on fall-spring change in family environment, child cognitive, social-emotional, and health outcomes. The next section examines the relationships among child, family, and classroom factors and children's outcomes; the methods used for those analyses appear in advance of the tables in that section. The final section provides tables of standard deviations and standard errors. (Contains 183 tables and 12 footnotes.) [For related report, "A Year in Head Start: Children, Families and Programs. ACF-ORPRE Report", see ED517213.]

This report summarizes research findings and implementation practices for teacher and principal value-added models (VAMs), as a first step in the Team Pennsylvania Foundation's (Team PA) pilot project to inform the development of a full, statewide model evaluation system. We have selected 21 studies that represent key issues and findings in the literature and examined varying degrees of value-added implementation in seven school districts or states. The authors present information aimed at VAM development: typical data elements, important modeling considerations, features specific to teachers or principals, and broad implementation features. Three appendixes are included: (1) Article Summaries; (2) VAM Implementation in School Districts and States; and (3) Technical Description of Value-Added Models. (Contains 28 tables and 17 footnotes.)

In this set of four supplemental tables, the authors compare the baseline test scores of the treatment and matched control group samples observed in each year after KIPP entry (outcome years 1 to 4). As discussed in Chapter III, the authors used an iterative propensity score estimation procedure to calculate each student's probability of entering KIPP in fifth or sixth grade, and then applied nearest-neighbor matching (without replacement) to identify a matched comparison group of students similar to KIPP students in terms of observed demographic characteristics and prior achievement at baseline. In that chapter, the authors also showed that the mean baseline math and reading scores of KIPP students in each of the 22 schools are not significantly different from their matched comparison students at the five percent level (see Table III.1). (Contains 1 footnote and 4 tables.) [This report was submitted to KIPP Foundation. For the accompanying document, see ED511107.]

As part of Phase 3 of the Universal Preschool Child Outcomes Study (UPCOS-3), Mathematica Policy Research worked with the First 5 LA Children and Families Commission and Los Angeles Universal Preschool (LAUP) to conduct a descriptive study of the characteristics of classrooms in LAUP programs during winter 2010. This study has a particular focus on the specific instructional practices used to support English language learners (ELLs) in LAUP's center-based and family child care programs. The study includes two key components: (1) classroom observations using measures determined as appropriate for settings with children who have diverse language backgrounds and (2) a survey of teachers regarding the instructional practices they use with children who are ELLs. The author reviews the procedures underlying the selection of classroom observation measures, the development of the instructional practices questionnaire, and the sampling procedures used for each type of data collection. They discuss the results from data collection using the classroom observation measures (Chapters II through IV) and instructional practices questionnaires (Chapter V) and they examine the relationships among these measures and summarize across them. (Contains 171 tables, 13 figures and 27 footnotes.)

In 2006, the Bill & Melinda Gates Foundation launched the Early Learning Initiative (ELI) to improve the school readiness of Washington State's children through three main strategies: (1) development of high-quality, community-wide early learning initiatives in two communities; (2) enhancement of statewide systems that support early learning; and (3) support for implementation of promising practices. The foundation joined with other private funders and state officials to form Thrive by Five (Thrive) Washington to energize development and support of high-quality early learning opportunities for all children in the state. In tandem with the formation of Thrive, the Gates Foundation sought two communities with a high level of need for early learning services and the capacity to develop and implement high-quality, community-wide early learning initiatives. The Gates Foundation selected East Yakima, a neighborhood in the central Washington community of Yakima, and White Center, an unincorporated area just outside Seattle. Thrive has worked with an intermediary agency in each community to develop and implement the initiative. Educational Service District 105, through its Ready by Five project, serves as the intermediary in East Yakima. In White Center, Puget Sound Educational Service District (PSESD) serves as the intermediary for the White Center Early Learning Initiative (WCELI). Thrive, the two communities, and other stakeholders worked closely with the Washington State Department of Early Learning (DEL) to develop Seeds to Success, a child care quality rating and improvement system (QRIS). In spring 2009, Thrive, Ready by Five, and Child Care Resources (PSESD's partner in White Center) began a six-month field test of a streamlined version of the DEL QRIS, referred to as the Seeds to Success Modified Field Test (Seeds). This report provides a detailed description of Seeds. Chapter II provides an overview of the impact and implementation studies that comprised the evaluation; specifically, the authors describe the study designs, data collection processes, and analysis methods. In Chapter III, they present findings from the impact evaluation. In Chapter IV, they discuss findings from the implementation study. Chapter V presents lessons learned about implementing the Seeds model. Appendix A contains additional details about the evaluation methodology. Appendices include: (1) Methodological and Technical Details; and (2) Distribution of Mean ERS and Arnett CIS Scores. (Contains 3 boxes, 36 tables, 10 figures and 41 footnotes.) [This report was submitted to the Bill & Melinda Gates Foundation.]

The Knowledge Is Power Program (KIPP) is a bold effort to create a network of charter schools designed to transform and improve the educational opportunities available to low/income families. KIPP schools seek to actively engage students and parents in the educational process, expand the time and effort students devote to their studies, reinforce students' social competencies and positive behaviors, and dramatically improve their academic achievement. Ultimately, the goal of KIPP is to prepare students to enroll and succeed in college. The KIPP Foundation is guiding this effort by selecting and training school leaders, promoting the program model, and supporting the KIPP network schools. This report presents preliminary findings from a matched, longitudinal analysis designed to estimate KIPP's effect on student achievement. The author's preliminary work estimates effects in 22 KIPP middle schools--making this the first report that applies a rigorous (nonexperimental) methodological approach across a nationwide sample of KIPP schools. They selected schools for which they were able to collect longitudinal, student/level data, and that were established by the 2005/06 school year or earlier to ensure that a minimum of two entering cohorts of students per school would be observed for multiple years. They find that students entering these 22 KIPP schools typically had prior achievement levels that were lower than average achievement in their local school districts. For the vast majority of KIPP schools studied, impacts on students' state assessment scores in mathematics and reading are positive, statistically significant, and educationally substantial. Estimated impacts are frequently large enough to substantially reduce race/ and income/based achievement gaps within three years of entering KIPP. They describe these findings in more detail in this report. Appendices include: (1) Administrative Data; (2) Supplemental Tables for Chapter II; (3) Analytic Methods; (4) Alternative Specifications; and (5) Subgroup Analyses. (Contains 34 tables, 21 figures and 28 footnotes.) [This paper was submitted to KIPP Foundation. For the accompanying report, see ED511108.]

The National Study of CMO (charter management organizations) Effectiveness is a national, longitudinal research effort designed to measure how nonprofit charter school management organizations (CMOs) affect student achievement, and to examine the internal structures, practices, and policy contexts that may influence these outcomes. The study began in May 2008 and will conclude in summer 2011. This is an interim report on the work; the final report will be issued in 2011. Outcomes of this interim report will inform the final report's analysis of CMOs' academic effectiveness, which uses a combination of experimental and quasi-experimental data. This interim report reveals that CMOs are still a young and regionally concentrated phenomenon. CMO schools serve a primarily low-income and minority population, similar to that of the big city school districts in which most operate. CMOs differ on the methods they use to create new schools and to support schools once they are up and running. CMOs may differ from traditional school districts in important ways, both in terms of central office and school organization. Most CMO business plans acknowledged early reliance on foundation funding, but projected break-even points when fees from affiliated schools would cover the cost of central offices and services to schools. Interviews with heads of CMOs indicate that many are struggling to find a way to operate at scale on fees obtainable from charter schools. Tough CMOs were created in part to compensate for the fact that charter schools receive less funding than district-operated public schools, some CMO heads suggest the "scaling up" problem cannot be solved without more equitable public funding or access to district-owned facilities. Appendices include: (1) List of All Identified CMOs as of 2007; (2) Construction of the Survey Indices; (3) CEO Survey; and (4) Case Study Protocol. (Contains 35 exhibits and 83 footnotes.)

In 2007, the Chicago Public Schools (CPS) began implementing a schoolwide reform called the Teacher Advancement Program (TAP) using funds from the federal Teacher Incentive Fund (TIF) and private foundations. Under the TAP model, teachers can earn extra pay and responsibilities through promotion to mentor or master teacher as well as annual performance bonuses based on a combination of their value added to student achievement and observed performance in the classroom. The idea behind the program is that performance incentives, combined with tools for teachers to track performance and improve instruction, should help schools attract and retain talented teachers and help all teachers produce greater student achievement. This report provides evidence on the impacts of TAP during the 2008-2009 school year, the second year of the program's rollout in CPS. Appended are: (1) Propensity Score Matching; and (2) Longitudinal Analysis of Test Score Data. (Contains 18 tables, 7 figures and 14 footnotes.) [For the Year One Impact Report, see ED507502.]

Roads to Success (RTS) is a school and career planning program designed to be implemented for 45 minutes per week in grades 7 through 12. Researchers at Mathematica Policy Research used a random assignment design to estimate the impacts of receiving RTS in grades 7 and 8. More than half of the students in these schools were eligible for free or reduced-price lunches and the schools had few minority students. Using student survey data collected from more than 1,400 students, we found no evidence of statistically significant impacts of the RTS program on motivation to go to school to learn job skills or on learning and study habits at the end of grade 8. We did find a statistically significant impact at the 0.10 level suggesting that RTS reduced a composite measure of negative behaviors. However, exploratory analyses of subcomponents of this measure suggested mixed results for student behaviors. We conducted exploratory analyses of additional outcomes and found positive impacts of RTS on talking to school staff about career and school plans, confidence in knowing how to find out about what types of jobs are best, and confidence in knowing what is required to succeed in different careers. However, students in both the treatment and control group reported talking to their parents more than to school staff about most issues regarding school and career planning; more than 90 percent reported agreeing or strongly agreeing with the statement that they "will be able to overcome barriers that stand in the way of achieving my career goals." Also, fewer than 10 percent of the treatment and control group students reported career aspirations or expectations that seemed too high; fewer than one percent reported career aspirations that seemed too low. Appended are: (1) Follow-up Student Survey Instrument; (2) Coding Scheme for Reasonably Ambitious and Realistic Career Expectations; (3) Future Outcome Variables; (4) Full Regression Results for Confirmatory Outcomes; and (5) Subgroup Results. (Contains 1 figure, 21 tables and 25 footnotes.)

On February 17, 2009, President Barack Obama signed the American Recovery and Reinvestment Act into law. Passed in response to the 2008 recession, the Act's purpose was to create jobs, pump money into the economy, and encourage spending. Through the Act, states received $1.2 billion in funding for the workforce investment system to provide employment and training activities targeted to disadvantaged youth. Congress and the U.S. Department of Labor (DOL) encouraged states and local workforce investment areas charged with implementing these youth activities to use the funds to create employment opportunities for these youth in the summer of 2009. To gain insights into these summer initiatives, DOL's Employment and Training Administration contracted with Mathematica Policy Research to conduct an implementation evaluation of the summer youth employment activities funded by the Recovery Act. As part of the evaluation, Mathematica analyzed (1) monthly performance data submitted to ETA by the states, and (2) qualitative data collected through in-depth site visits to 20 local areas. This report describes the national context for implementation, provides an in-depth description of the experience of selected local areas, and presents lessons on implementation practices that may inform future summer youth employment efforts. Appended are: (1) Detailed List of Research Questions; (2) Characteristics of Youth Focus Group Participants; (3) Recovery Act Allocations and Draw Downs by State; and (4) Key Performance Statistics by State. (Contains 25 tables, 7 figures and 23 footnotes.) [This report was prepared for the U.S. Department of Labor, Employment and Training Administration, Office of Policy Development and Research.]

This final report updates the report "Upward Bound Math-Science: Program Description and Interim Impact Estimates" published in 2007 (Olsen et al. 2007). The 2007 interim report contained descriptive findings from a survey of Upward Bound Math-Science (UBMS) grantees from the late 1990s at the time of the study's initiation and impact estimates through the period four to six years after expected high school graduation of the study sample. The current report presents impact estimates for the period seven to nine years after scheduled high school graduation. Given the academic services provided by UBMS, it is natural to ask whether participating in UBMS affects the educational outcomes of the students who participate. From their impact analysis, the authors found that the UBMS program achieved the following: (1) Increased enrollment at four-year institutions; (2) Shifted enrollment from two-year to four-year institutions for Upward Bound participants; (3) Increased enrollment at more selective institutions; (4) Increased math and science course taking; (5) Increased postsecondary degree completion overall and at four-year institutions; and (6) Increased likelihood of earning a degree in a social science field of study. Appended are: (1) Data Collection and Outcome Measures; (2) Program Impacts by Subgroups; and (3) Sample Sizes and Standard Errors. (Contains 47 exhibits and 8 footnotes.) [For related report, "Upward Bound Math-Science: Program Description and Interim Impact Estimates," see ED497631.]

We used data from the School Nutrition Dietary Assessment III Study to examine the dietary patterns of school meal program participants and nonparticipants and the relationship between school meal participation and children's BMI and risk of overweight or obesity. School Breakfast Program (SBP) participants consumed more low nutrient energy dense (LNED) baked goods and more calories at breakfast than non participants. National School Lunch Program (NSLP) participants had lower intake of sugar-sweetened beverages and a lower percentage of calories from LNED foods and beverages than nonparticipants. Overall, NSLP participation was not significantly related to students BMI, though participants were less likely to be overweight or obese than non-participants among black students but more likely to be so among "other race" students. SBP participants had significantly lower BMI than non-participants, possibly because SBP participants are more likely to eat breakfast and eat more at breakfast, spreading calorie intake more evenly over the course of the day. A section on Dietary Patterns of School Meal Participants and Nonparticipants: Supplemental Tables is appended. (Contains 40 tables, 4 figures and 41 footnotes.)

In February 2007, First 5 LA contracted with Mathematica Policy Research, Inc. (MPR) and its subcontractors, Juarez and Associates, American Institutes for Research (AIR), and Berkeley Evaluation and Assessment Research (BEAR) Center, to conduct the First 5 LA/LAUP Universal Preschool Child Outcomes Study (UPCOS). The authors conducted the study in two phases. The spring 2007 pilot phase (Phase 1) examined the feasibility, reliability, and validity of selected child development measures in the large, culturally and linguistically diverse population of children served by LAUP programs. In early spring 2007, the authors selected a purposive sample of 418 4-year-olds enrolled in 14 LAUP programs. The children were predominantly Latino (60 percent), and 28 percent spoke primarily Spanish at home. Data collection included direct child assessments and a teacher interview encompassing ratings of children's behavior and development. The second phase of the study (fall 2007 and spring 2008) addressed the quality and overall implementation of LAUP programs, children's growth from fall to spring, and the relationships between family characteristics and children's development over time. After describing the characteristics of the representative sample of children and families, the authors report their findings related to the three broad questions this study addresses, which are described in more detail in Chapter II: (1) What is the overall level and range of quality in the implementation of LAUP/PoP center-based programs?; (2) How do children enrolled in LAUP/PoP center-based programs develop from fall to spring?; and (3) How are characteristics of children and families related to school readiness outcomes? In Chapter II, the authors describe the study methods used, including how they sampled programs and families; what child, parent, teacher, and classroom measures they used; what data collection procedures they followed; and what research questions the study was designed to address. In Chapter III, they present the results, beginning with descriptive data on programs, teachers, and classrooms, followed by descriptions of children and families. They then report what they learned about parents' involvement and satisfaction with the program. They conclude the results chapter with findings about the relationships between child and family characteristics and the children's school readiness outcomes. Chapter IV summarizes the findings and presents their assessment of their implications for programs and recommendations for future study. In the appendices, they provide details on the measures used, the routing procedure used for determining the language of assessment, what they learned about a new experimental observational measure of teacher-child interactions, and their report on the PoP programs. Appended are: (1) Measures Used in Phase 2 of the Universal Preschool Child Outcomes Study; (2) Procedures for Routing Children into the Most Appropriate Language for Assessment; (3) Tables of Results Presented in Chapter III; (4) Descriptive Results from the Pilot Study of the Language Interaction Snapshot (LISn); (5) Descriptive Information on LAUP Programs Participating in the Power of Preschool Demonstration Program; and (6) Development of a Respect for Differences Scale. Individual appendices contain references. Individual chapters contain footnotes. (Contains 98 tables, 24 figures and 4 boxes.) [For executive summary, "The Children of LAUP: Executive Summary of the First 5 LA Universal Preschool Child Outcomes Study," see ED533142.]

The BRIGHT program was designed to improve the educational outcomes of children in Burkina Faso. It focused on girls in particular and was implemented in 132 rural villages throughout the 10 provinces of the country in which girls' enrollment rates were lowest. It consisted of constructing primary schools with three classrooms and implementing a set of complementary interventions. This report documents the main findings from the impact evaluation of the BRIGHT program. The evaluation design involved comparing children in the 132 BRIGHT villages (participant group) with children in 161 similar villages that had applied to participate in BRIGHT but were not chosen (comparison group). In general, the main conclusions are that BRIGHT had about a 20 percentage point positive impact on girls' primary school enrollment, and had positive impacts on Math and French test scores for both girls and boys. Although the magnitude of BRIGHT's estimated impacts is larger than that observed in typical education interventions in developing countries, a cost-effectiveness analysis would be needed to assess whether the effects are large on a per-dollar basis. In particular, it would be useful to know whether building a less expensive school of the sort typically built in Burkina Faso would have generated similar impacts. While this evaluation cannot answer this question definitively, the authors found suggestive evidence indicating that part of the impact of BRIGHT came from having built a school in villages in which no school would have been available, and part from having built a school with a better infrastructure and add-on components than the typical school that would have been available without BRIGHT. Appendices include: (1) Copy of Application Form; (2) Rules to Calculate Eligibility Score; (3) List of Communities with Eligibility Scores; (4) School and Household Questionnaires; (5) Request for Proposal for Data Collection; (6) Training Manual Used for Data Collection; (7) Statistical Analyses to Verify Appropriateness of Regression Discontinuity Design; (8) Statistical Analyses to Verify Robustness of Impact Estimates; and (9) Access vs. Quality. (Contains 1 box, 16 tables, 10 figures, and 9 footnotes.)

Although Missouri has had a Career Ladder program for teachers since 1987, very little research has been carried out to measure the program's effects and what has been studied has not been comprehensive. This paper examines the program's effect on student achievement across the state, using longitudinal data on district math and reading scores for 524 Missouri school districts over a nine-year period. Our primary specification compares achievement levels in participating districts with a matched group of non-participating districts. We also applied alternative specifications to identify the impact of the program, for example controlling for prior district scores and measuring variations in district participation over time to identify effects of the program within a given district. Across the range of specifications, the estimated effects of the Career Ladder program range from small positive effects to no effect in both math and reading. We conclude that if the Career Ladder has a positive impact on test scores, it is probably very small. (Contains 10 tables and 10 footnotes.)

One goal of Missouri's Career Ladder program (CL) is to help school districts that have difficulty retaining teachers, particularly those that are small and rural, by offering their teachers opportunities to earn extra pay for extra work and professional development. This report seeks to estimate the effect Career Ladder Program has had on teachers' career decisions, specifically their decisions to stay in a specific school district or to remain in the teaching field. This report presents evidence suggesting that a school district's participation in the Missouri Career Ladder Program would tend to increase retention in the district and the profession, especially for mid-career teachers. The results reported in this paper are especially useful when combined with other findings on this program. Previously reported analyses using 10 years of district-level test score data failed to find evidence for an impact on test scores (Booker and Glazerman 2009). However, focus groups and interviews conducted by Silman and colleagues (Silman and Glazerman) suggest that the program is popular with participating teachers and is run almost entirely by teachers themselves. Thus, when considering the benefits, there was no measurable test score effect, but a positive effect on retention and possibly a positive effect on teacher well-being generated by increasing their incomes by an amount that likely exceeds their hourly rate for the extra work completed. These findings have several implications for the teaching profession. First, they suggest that small bonuses can affect behavior. Second, the magnitudes of the effect estimates remind people that one should not expect "large" effects from small bonuses. Third, the findings suggest that a bonus program that is "not" tied to student performance but offers additional pay for more work for mid-career teachers can make teachers feel better off even if it does not improve their teaching in an easily measurable way. (Contains 7 tables, 2 figures and 3 footnotes.)