Search

For decades, researchers and educators alike have been caught in waves of reforms that sought to change the quality of teaching and learning at scale. Research-practice partnerships (RPPs) emerged as a promising strategy for generating new knowledge and building new capabilities for improving teaching and school systems through research. RPPs seeking to improve instructional practice at scale face dual learning imperatives in that partners must learn how to: (1) manage and confront the complexity involved in trying to change instruction at scale; and (2) develop a model of partnership that is equipped to the task. This paper presents a framework that is composed of four elements and a reflective tool for partnering for instructional improvement at scale. The following presents the structure of this report: (1) Description of the challenge of scaling instructional improvement; (2) Literature review of RPPs and organizations focusing on the challenges faced by RPPs; (3) Introduction of the framework, which is composed of commitments, structures, expertise, and social learning; and (4) A brief example of a reflective tool to be used for assessing partnering in an RPP.

For more than four decades, researchers, policymakers, professional educators, and the philanthropic community in education have wrestled with how to scale up promising pockets of reform. Centering equity in scaling efforts means working with and prioritizing students who have historically lacked access to powerful learning opportunities, including deeper learning, and have not reached the ambitious outcomes we desire for everyone. This report seeks to help school district leaders and improvement teams define what it means to scale for equity. Presented in this report is a scaling-for-equity framework informed by the experiences of 10 research-practice partnerships (RPPs) and designed to provide readers with multiple, iterative opportunities to think about equity as it relates to scale. The scaling-for-equity framework supports a journey in which members of a team work to improve educational practice by traveling over time from their origin to a destination. The framework and accompanying tools are designed to: (1) help improvement teams navigate their scaling-for-equity journey; and (2) support school district leaders striving to transform their communities' public education systems into equitable ones by prompting improvement team members to consider equity-related questions during all stages of the work and to develop shared responses, describing and categorizing scaling strategies so that teams can identify the strategies, alone or in combination, that will bring them closer to their equity goals, and providing support for disentangling and addressing the many influences and tensions at play when improvement teams aim to bring about change.

Designing for and implementing deeper learning across classrooms and schools that serve communities disadvantaged by the U.S. educational system is challenging. This paper illuminates this challenge by asking the question: What would designers of interventions at the classroom, school, and district levels have to take into consideration when they want to powerfully set their organizations on a developmental path towards deeper learning? The thinking put forth in this paper is closely informed by the experiences of a number of district change projects aimed at furthering deeper learning. The projects were funded by the William & Flora Hewlett Foundation and organized as research-practice partnerships (RPPs) in which improvement teams worked together to design interventions or change activities over a period of three to four years. The approaches taken by the projects differed widely. While the projects were operating in different contexts and considering different developmental phases for deeper learning, the differences also reflected underlying ideas about what schools and districts need to do in order to make deeper learning a reality in schools and classrooms. The goal of this paper is to map out the encompassing nature of the design challenge so that designers come to (1) appreciate the spectrum of tasks to be addressed given the ambition and complexity of the deeper learning undertaking; (2) recognize the unique sets of tasks that specific designs tackle and the resulting trade-offs that may be required; and (3) draw from a common frame of reference for learning across projects that aim at deeper learning.

A burgeoning set of education-related organizations join networks each year, reflecting a shared belief that problems in education are too complex for any one educator or organization to solve on their own and that collaboration can support educational improvement. The goal of this paper is to describe how networked improvement communities (NICs) create a social structure to catalyze the type of community that can solve complex problems. We draw from prior theorizing, the research literature on network and learning community development, and observations of developing NICs to articulate a framework for network development. The framework is intended to be an analytic tool for thinking and reasoning about NIC emergence and maturation. [This report was presented at the American Educational Research Association Annual Meeting, Toronto, ON.]

This paper describes progress, challenges and opportunities of using policy to advance educational equity and excellence at scale in the United States. It begins with a look-back that examines two key elements of the standards-based education (SBE) era: First, it traces policy shifts and implementation challenges of the SBE era. Second, it examines student achievement trends over 30 years, with particular attention to learning outcomes for Black, Hispanic, and students from low-income backgrounds. The paper then provides a landscape analysis of key stakeholders that are actively engaged in shaping, guiding, developing and implementing educational improvement efforts. The landscape analysis also identifies those who have not been adequately involved but should be going forward, such as grassroots and community leaders, parents, teachers and students, particularly from people of color. To conclude, the report recommends bringing together a cross-section of diverse stakeholders from key educational sectors--such as policymakers, community-organizers, practitioners and researchers, students and families--to form a multi-year listening and learning community that engages in a series of field-based learning experiences. The learning engagements would allow stakeholders to consider strategies to: (1) broaden the vision of youth development; (2) build supports for equity; and (3) bolster community engagement.

Across the social sectors, a growing number of organizations are embracing continuous improvement approaches and organizing themselves as networks in order to attack complex problems. To succeed, networks must learn quickly by studying their own practices, continuously adapting to changing circumstances within their organizations and in the broader environment, and incorporating this learning into their ongoing work. In this paper, we describe an integrated approach--which we call Evidence for Improvement (EFI)--that explains how a variety of tools and practices drawn from diverse forms of program evaluation can inform the leaders of these networks in advancing productive change. We propose that improvement networks can be conceptualized and measured using a three-level nested model composed of a working theory of improvement, an improvement enterprise, and environmental contexts. As they engage in inquiry targeted at each level, those with evaluative expertise should become authentic partners with improvement leaders and fully align their analytic efforts with the improvement activities and social contexts of the network. The EFI approach we describe is designed to enhance a network's internal learning processes and, in turn, lead to more positive impacts for educators and students. It has implications for improvement practitioners, evaluators, and funders.

Traditional developmental mathematics sequences are a major impediment for community college students who need to acquire college-level mathematics credit in order to attain their long-term academic goals. Each year, approximately 500,000 students fail to complete their developmental mathematics coursework, jeopardizing their ability to earn an associate degree or technical certification, and/or transfer to a four-year institution for further education. In response to this national crisis, the Carnegie Foundation for the Advancement of Teaching founded the Carnegie Math Pathways. The Pathways include Statway® and Quantway®, two alternative courses that accelerate students' progress to and through college-level mathematics. Since launching in 2011, the Pathways have served over 27,000 students and consistently demonstrated significantly higher success rates than the traditional course sequence. In 2016-2017, Pathways success rates increased further, even as the program served nearly five times as many students as in its first year. This report provides descriptive statistics for 2016-2017 student outcomes as well as insights into potential areas for improvement, based on data from the Pathways' sixth year of implementation.

Since 2011, Statway®, an alternative pathway to completing college-level developmental mathematics, has evolved from a two-semester sequence into a single-semester course. As of the 2016-2017 academic year, five institutions have offered an accelerated one-term version of Statway. This paper describes why these colleges chose to implement accelerated Statway, the barriers to success they faced, and the benefits for implementing colleges. Accelerated Statway offers a rich illustration of how colleges have addressed their local needs by making adaptations to the successful program while maintaining the design integrity of the program as it was originally conceived.

The Carnegie Foundation for the Advancement of Teaching's Math Pathways seek to improve outcomes for community college students who take remedial math courses. The Pathways include two comprehensive instructional systems--Statway® and Quantaway® and are described in this report. They are designed to support students to achieve the necessary math credits, and ultimately, earn a degree or other credential, either at a two- or four-year institution. This report presents initial findings related to degree attainment and transfer for Pathways students. The goal of Quantway is to prepare students for success in college-level mathematics and to develop quantitatively literate students. Quantway 1 is focused on quantitative reasoning that fulfills developmental requirements of traditional developmental mathematics classes in a single semester or quarter. In 2012, Quantway 2 was added to create a complete Pathway to college credit. Results presented in this paper focus only on the post-participation effects of Quantway 1. As follow-on data for Quantway 2 becomes available, a subsequent report will be issued. Statway is focused on statistics, data analysis, and causal reasoning. It combines developmental mathematics and college-level statistics in a seamless and consecutive two semester or quarter program that allows students to fulfill both developmental and creditbearing mathematics in a single academic year. Statway is designed to teach mathematics skills that are essential for a growing number of occupations in which decision-making under conditions of uncertainty is a part of daily work.

Statway® is an accelerated developmental mathematics intervention program for college students who are not yet prepared to succeed in a college-level math course. The Carnegie Foundation for the Advancement of Teaching created the initiative to increase student success rates in developmental mathematics and subsequent credit-bearing, college-level mathematics. Statway differs from traditional developmental mathematics sequences in that it allows students to complete both their developmental and college-level mathematics requirements in a single pathway over the course of one academic year. A previous study (Yamada & Bryk, 2016) found that students--across all sex and race/ethnic groups--who enrolled in Statway, had higher rates of completion for both developmental and college-level math compared to carefully matched students in traditional developmental math programs. Statway provides a multifaceted approach to addressing challenges that impede student success through psychosocial and academic supports, as well as lessons based on examples relevant to students' lives. The course is structured so the same cohort of students is enrolled in a consecutive twoterm sequence with the same faculty instructor, students work collaboratively, and instructors learn and apply new and more interactive teaching methods. Faculty also receive professional development and support through a networked improvement community (NIC) that connects them with other practitioners, administrators, and researchers from diverse institutions across the nation. We assessed the effectiveness of the program in its third and fourth years by utilizing a multilevel modeling approach with propensity score matching to compare Statway student outcomes with their counterparts in traditional developmental mathematics programs. We found that in comparison to traditional developmental mathematics students, Statway students had a significantly higher likelihood of successfully attaining college-level mathematics credit--even when comparison group students were allowed twice the amount of time to achieve the same outcome. These results are consistent across all sex and race/ethnic subgroups, as well as in almost all participating classrooms and institutions. Additionally, in replicating earlier findings from the first two years of the program, the results suggest that Statway continues to advance equitable outcomes even as it scales to greater numbers of students in an increasing number of institutions. This report also suggests potential areas for future investigation and research. This report provides an important sequel to the Carnegie Foundation's previous assessment of the effectiveness of the first two years of Statway®, a mathematics program designed to expedite the process by which students may fulfill their developmental requirements and earn college mathematics credit. Employing a multilevel modeling technique with propensity score matching, the previous peer-reviewed study found that Statway considerably improved college mathematics success rates among students from the 2011 and 2012 cohorts (Yamada & Bryk, 2016). Since then, Statway has continued to scale within current institutions and spread to new ones, nearly doubling its enrollment and expanding its geographic reach to three new states, and expanding in total four-fold since it began five years ago (Huang, Hoang, Yesilyurt, & Thorn, 2016). Scaling innovations in education is notoriously difficult (Cannata & Nguyen, 2015). Thus, the current study, which focuses on the third and fourth years of Statway's implementation, seeks to further substantiate Statway as an evidence-based initiative by replicating results from the previous analysis. To better inform our efforts to address challenges in scaling Statway, the study also examines variation in performance across instructors and institutions.

College math is a gateway course that has become a constraining gatekeeper for tens of thousands of students annually. Every year, over 500,000 students fail developmental mathematics, preventing them from achieving their college and career goals. The Carnegie Math Pathways initiative offers students an alternative. It comprises two Pathways courses, Statway® and Quantway®, that provide post-secondary students with an accelerated method to complete developmental mathematics sequences and attain college-level math credit. Since its launch in classrooms at 29 colleges in 2011, Statway and Quantway have served over 20,000 students. In 2015-2016, the Pathways had achieved remarkable results even while serving over four times as many students as in its initial year. Student success rates in Statway, Quantway 1, and Quantway 2 reached 50%, 64%, and 67% respectively. New impact studies also show that Pathways students have higher completion rates, subsequent math enrollment and success, credit accumulation rates, and transfer rates from two-year to four-year colleges. This report provides descriptive statistics on the 2015-2016 student outcomes, as well as insights into potential areas for improvement based on data from the Pathways' fifth year of implementation.

This report analyzes evidence from a new study assessing the effectiveness of Quantway 1 (QW1), a single-term accelerated developmental math course with a focus on quantitative reasoning. Quantway® is one of the Pathways courses (along with Statway®) developed by a network of faculty and content experts convened by the Carnegie Foundation to accelerate the progress of college students through developmental mathematics with the aim of increasing success rates in earning college credit in mathematics. This report is the sequel to an earlier study (see ED582412) of the effectiveness of QW1, both of which applied a multilevel propensity score matching approach. The first study provided robust evidence that QW1 increases student success in fulfilling developmental math requirements and advances equity in student outcomes. The present study tracked student college math achievement through the year after QW1 enrollment. Results in this most recent analysis revealed that QW1 students were significantly more likely to enroll in credit-bearing college math courses within a year than their counterparts who followed traditional developmental math sequences were, and that while increasing subsequent math course taking, QW1 students demonstrated a comparable GPA. Significantly, QW1 effects were positive across all sex and race/ethnicity subgroups, as well as in nearly all classrooms and colleges. The current study provided additional empirical evidence of the persistence of QW1's effectiveness for diverse student populations across varied classroom and institutional contexts. Directions for future work are discussed. As indicated in the prior study, additional questions still need to be answered: Do the effects of QW1 persist in the following year? More specifically, we are interested in whether or not QW1 students are (a) more likely than matched comparison students to enroll in credit-bearing college math courses, and (b) perform comparably to or better than matched comparison students in college math courses. Because QW1 is designed not only to get students through their developmental math sequences but to help them meet their college-level math requirements, these questions will be particularly important to answer in determining QW1's effectiveness. Therefore, the objective of the current study is to assess the persistence of effects of QW1 by tracking student college math achievement one year after QW1 participation.

The Carnegie Foundation launched its Math Pathways initiative nearly six years ago at 29 colleges across the country with the aim of improving success rates in developmental math. Tens of thousands of students a year, who need additional preparation for college-level math, are shut out of earning degrees and fulfilling careers due to the huge failure rate in these classes. This narrative describes how Carnegie created the highly successful Pathway program by gathering math and design experts into a networked improvement community (NIC) that applied the principles of improvement science in its work. Improvement science is an evidence-based process of problem solving guided by a set of principles that includes identifying a specific problem, designing a solution that focuses on the needs of the users (such as students), and continually measuring and refining results to make sure they're working. For its user-centered approach, NIC members listened to students and identified both external and self-imposed barriers to success in traditional developmental math. They then addressed these specific challenges with their program design. Improvement science also required the Pathways NIC to understand the local contexts within each school and at schools across the country in order to design an intervention that colleges could adapt so that they would work for different subgroups of students, under varied constraints and circumstances. The result of this work are two accelerated developmental math programs--Statway® and Quantway®. Combined, Statway and Quantway have served more than 20,000 students since the 2011-2012 academic year with triple and double the success rates, respectively, in half the time as traditional developmental math, including when it comes to earning college-level math. As the initiatives were being implemented and expanded, Carnegie, as the organizing hub for the NIC, took on the task of identifying emerging problems and forming improvement teams to work on them. Results and new adaptations continue to be measured today in an effort to improve future results even more.

Quantway is a Carnegie Math Pathways initiative which redesigns the content, pedagogy, and structure of traditional developmental math courses to simultaneously tackle traditional barriers of student success and support a broader range of developmental students in achieving their math potential. Specifically, Quantway is a quantitative reasoning sequence that is comprised of a single term developmental math course called Quantway 1 and a college-level math course called Quantway 2. This study assessed the effectiveness of the developmental math course, Quantway 1, between the spring of 2012 and the fall of 2014. A hierarchical linear modeling technique was used to conduct propensity score matching across 37 student characteristics in order to compare the course performance of Quantway 1 students with matched comparison students in traditional developmental math courses. Quantway 1 students demonstrated significantly higher odds of success than matched comparison students in fulfilling developmental math course requirements. Additionally, Quantway 1 effects were positive across all sex and race/ethnicity subgroups as well as in nearly all classrooms and colleges. This study provided robust evidence that Quantway 1 increases student success in fulfilling developmental math requirements and advances equity in student outcomes. Directions for future work are suggested.

The Community College Pathways initiative is composed of two pathways, Statway® and Quantway® that provide post-secondary students with accelerated means to complete developmental mathematics sequences and attain college-level mathematics credit. Since its inception in 2011, the Pathways initiative has made tremendous strides toward reclaiming thousands of students' mathematical lives. In 2014-2015, the Pathways sustained the remarkable results seen in the Pathways' first three years of implementation, including successful course completion rates of approximately 50 percent for both Pathways. Notably, the program was able to achieve these outcomes while serving over three times as many students as in its first year. This report provides descriptive statistics on 2014-2015 student outcomes, as well as insights into potential areas for improvement based on data from the fourth year's Pathways implementation.

An evaluation system that is designed to support and improve teacher practice--rather than simply to assess and manage teacher performance--would have as its foundation effective and reliable processes to observe classroom practice and give teachers useful feedback. Such an improvement-focused evaluation system should be aligned with and contribute to a district-wide system that includes a shared definition of teacher quality, a clear set of district priorities, and a coherent strategy for improving teacher practice. High-quality observations demand skilled observers who deeply understand a common frame of reference for quality teaching (the rubric their district uses), who reliably interpret classroom evidence according to this rubric, and who give teachers timely feedback that is targeted to improve their practice. The challenge for districts and states now is how to train and support these observers. What types of training and support do observers need, and what is being provided? Does effective training improve observer skills and, ultimately, classroom practice? To address these questions, the authors examined recent research on observer training and explored the training efforts in a sampling of five districts: Boston, Massachusetts; the District of Columbia; Santa Fe, New Mexico; Maricopa County, Arizona; and New Haven, Connecticut. This brief describes key takeaways from their research and from conversations with trainers, researchers, and district officials.

Quitting in the face of hard work is never the response a teacher wants to see, but it's one that threatens to become more common as academic pressures rise. The new Common Core State Standards, the latest in a decades-long effort to drive educational improvement, soon will be setting unprecedented expectations for the performance of students, teachers, and schools. Reaching the Common Core bar will require more effective instruction than many students have traditionally received, along with assessments aligned to the standards. Students will also require something else: the motivation to meet the Common Core demands. As the academic requirements rise, so too must students' willingness to take on increasingly difficult tasks and to persist through the failures that often precede success. Motivating students, studies show, is already a considerable challenge. This report aims to help educators and policymakers understand the complex topic of student motivation. Key terms are defined, new research findings are discussed, and promising classroom strategies for improving students' engagement in learning are examined. The report is organized according to three major factors that contribute to student motivation: rewards and value, academic mindsets, and school-based relationships. It also explores the structural supports--measurement systems, teacher training, and the translation of research into practice--necessary for the scaling and long-term success of this work. A Bibliography and Glossary are provided.

On December 5, 2014, the Carnegie Foundation for the Advancement of Teaching's DC office hosted an invitation-only gathering of fifteen federal, state, and local education policy experts in both the public and nonprofit sectors. The goal of the meeting was to explore strategies to improve Title II, Part A of the federal Elementary and Secondary Education Act in anticipation of the Act's reauthorization. During this meeting participants were encouraged to think about the teacher-quality problem in new ways by first fully exploring the root causes of the issue, which often go unexamined in the fast-moving world of public policy. Using improvement-science techniques developed by the Carnegie Foundation to help educators and education policymakers see in their entirety the ways that systems produce problematic outcomes, participants explored the root causes of this problem statement: "The public education system is not adequately recruiting, retaining, and supporting a high-quality/effective teacher workforce for all students." The exploration process involved identifying the major sources of the recruitment, retention, and support problem, and then producing a comprehensive mapping of the contributing factors in each major category. It yielded a variety of insights that may not have surfaced had the group jumped right into solutions without first seeing the system as a whole. Recommendations emerged on the distribution of Title II funds, spending priorities, the collection and report of spending data, and the communication of program expectations to funding recipients. In exploring the reasons why public education does not adequately recruit, retain, and support a high quality teaching workforce, participants identified six major problem areas: preparation, leadership, development, job design, culture/norms, and compensation. These areas provided the basis for an in-depth analysis of the teacher-quality challenge. This report provides details of the following recommendations that were a result of the expert convention: (1) Changing the way states and LEAs receive Title II, Part A funds; (2) Changing which activities are funded by Title II, Part A; (3) Changing the data systems and reporting around Title II, Part A; and (4) Changing the communication about and purpose of Title II, Part A. The following are appended: (1) Complete List of Root Causes as Expanded from Figure 1; and (2) Complete List of Participants' Ideas for Changing Title II.

Early in the twentieth century, the industrialist Andrew Carnegie established the Carnegie Foundation for the Advancement of Teaching to create a pension system for the nation's college professors. The introduction of this pension system proved an ingenious educational reform. At the time, American higher education was a largely ill-defined enterprise with the differences between high school and colleges often unclear. To qualify for participation in the Carnegie pension system, higher education institutions were required to adopt a set of basic standards around courses of instruction, facilities, staffing, and admissions criteria. The Carnegie Unit, also known as the credit hour, became the basic unit of measurement both for determining students' readiness for college and their progress through an acceptable program of study. Over time, the Carnegie Unit became the building block of modern American education, serving as the foundation for everything from daily school schedules to graduation requirements, faculty workloads, and eligibility for federal financial aid. Today, the Carnegie Unit is under intensifying critique from educators and education policymakers who want to make student performance more transparent and the delivery of education more flexible. They see the Carnegie Unit as a significant impediment to the changes they seek. They advocate for innovations that support transparency and flexibility, including competency-based education models. In an effort to inform these reform conversations and serve a constructive catalyst for change, the Carnegie Foundation launched a study to revisit the role, function, and uses of the Carnegie Unit. They explored in detail the nature of the problems that reformers aim to address and the complexity of the systems in which these problems are embedded. They analyzed what a shift away from the Carnegie Unit toward a competency-based (rather than an instructional time-based) metric might entail for the operation of educational institutions and the students they serve. Finally, they considered the scope of innovations necessary to replace the Carnegie Unit, the ambitiousness and uncertainties associated with these tasks, and the vast array of practical problems that would need to be solved. The study's findings and the Carnegie Foundation's recommendations are presented herein. Appended are: (1) Improvement Science Principals; (2) Carnegie Foundation for the Advancement of Teaching Board of Trustees; and (3) Carnegie Unit Advisory Committee. A bibliography is included.

The Community College Pathways initiative consists of two pathways, Statway® and Quantway®, that accelerate post-secondary students' progress through their developmental mathematics sequence and a college-level course for credit. Launched in 2011, the Pathways have been remarkably successful, helping thousands of students achieve success in college-level mathematics. In 2013-2014, the Pathways maintained the positive outcomes attained in the first two years of implementation, including successful course completion rates of approximately 50 percent for both Pathways. The initiative was able to achieve these results while serving almost three times as many students as in its first year. This report provides descriptive statistics on 2013-2014 student outcomes as well insights into potential areas for improvement based on data from the third year's implementation of the Pathways. Tables are appended.

It is common knowledge that teacher quality is a key in-school factor affecting student achievement. While the quality of teaching clearly matters for how much students learn, this quality is challenging to measure. Evaluating teacher quality based on the level of their students' end-of-year test scores has been one method of assessing teacher quality, but this approach favors those teachers of students who begin the year already at a high academic level. Value-added methodology is an alternative use of annual test score data to assess teacher quality. As opposed to using the level of student achievement on a test at the end of the year to assess a teacher's effectiveness, value-added methodology seeks to isolate a teacher's contribution to student growth from other factors that contribute to student achievement, such as prior achievement or socio-economic status. Methodologists generally agree that value added estimates can serve as a gauge of some dimensions of teacher effectiveness. Yet the experts also agree that value-added measures have significant limitations. Most clearly, they do not capture a teacher's entire contribution to student learning because they are based solely on test performance, itself an imprecise signal of teacher effectiveness. There is disagreement among experts on the extent to which value added is biased. Thus there remains considerable debate about how value-added measures should be used to inform personnel policies, if they are to be used at all. As the real impacts of reforms using value added emerge and as researchers assess these effects, educators will learn more about the consequences of different uses of value added. Until then, the Carnegie Knowledge Network (CKN) briefs have provided a rich picture of current research on the use of value-added measures for teacher evaluation. Reviewing the briefs and their implications for practitioners, CKN arrived at the following recommendations: (1) When using value added, allow educational leaders to make judgments in interpreting the value-added results in light of other available measures of teacher quality and the principals' own assessments; (2) Use value added with other measures that are valid and have variation; (3) Choose a test that measures knowledge that is valued; (4) Consider differences in teaching contexts when using value added to compare teachers; and (5) Take specific steps to ensure the overall credibility of the teacher evaluation system. Data from millions of students and thousands of teachers as well as careful thought and analysis have taught us much about the statistical properties of value added. Yet these measures are only just beginning to be put into widespread use for teacher evaluation. As these new systems roll out, states and districts should be learning and experimenting, determining what works and what doesn't. They should use this time to collect data and monitor their evaluation systems, using what they learn to make revisions.

New teacher evaluation systems have emerged as the cornerstone of the recent movement to improve public school teaching. Fueled by incentives from the federal government, state and local policymakers have sought to replace the often-cursory evaluation models of the past with more comprehensive ones. In contrast to past evaluations, which often relied on a single classroom visit by an untrained administrator, new models evaluate teachers on the basis of their students' achievement, on surveys that capture students' perceptions of their teachers' practice, and on improved classroom observations. But as these new systems roll out, there is mounting evidence that principals alone cannot bear the time burden they impose. Nor can a single principal be depended upon to deliver effective feedback across content areas to teachers with vastly different strengths, weaknesses, and teaching assignments. In response to these challenges, a growing number of districts have adopted multi-rater systems, in which several observers watch teachers at work, score their performance, and provide feedback. Sometimes the raters observe together, sometimes independently. And more and more, they come to the process from different vantage points: Many districts now rely on combinations of peer teachers, master teachers, and administrators from different schools. By adding more eyes to these evaluations, districts aim not only to relieve principals but, more important, to lend new perspectives, deeper expertise, and greater objectivity to the evaluation process. This report explores the use of multi-rater evaluation systems in 16 districts with widely varying student populations, resources, and policy priorities. The districts range from New York City, the nation's largest school system, to Transylvania County, NC, which educates just 3,500 students each year. Drawing on document reviews and interviews with district officials, it examines the districts' varying aspirations for multi-rater models, as well as how the models are designed, how they operate, and the challenges they pose.

Statway® is one of the Community College Pathways initiatives developed by the Carnegie Foundation for the Advancement of Teaching to promote students' progress beginning in developmental math to and through college math credit. Early descriptive results suggested that the Statway program has tripled the success rate for students in half the time to achieve college math credit. This study presents a more rigorous causal analysis of hypothesized program effects. We used a multilevel approach with propensity score matching to evaluate the effectiveness of Statway. Propensity score matching results are consistent with earlier descriptive findings. They replicate across two different cohorts and hold up for varying genders, race/ethnicity groups, and math placement levels. Directions for future work are also suggested.

Carnegie's Community College Pathways (CCP) offers two pathways, Statway® and Quantway®, that reduce the amount of time required to complete developmental mathematics and earn college-level mathematics credit. The Pathways aim to improve student success in mathematics while maintaining rigorous content, pedagogy, and learning outcomes. It is essential that the Pathways' success in securing college credit for participants does not come at the expense of the rigor of the curriculum or level of student learning. This report describes the comparative analyses from the Pathways summative assessment system. All items on the summative assessments were rigorously validated by field testing with over 500 members of a comparison sample who had taken a college mathematics or statistics course. Subsequently, all Pathways students took the summative assessments at the end of each term. This report describes analyses that use the development sample for comparison to Pathways students to assess their relative levels of knowledge. This report then describes the relationship between students' summative assessments and their end-of-term grades. This analysis is to provide further evidence that Pathways students are not just receiving high grades and successfully completing the Pathways at a higher rate, but that their grades are reflective of actual learning, as corroborated by the independent measure. Also included in the report are appendices: (1) Sample Summative Assessment Items: Statway; (2) Statway and Quantway Modules; and (3) Tables.

Proposals to evaluate teachers based on their "value-added" to student test scores generate intense debate. Underlying the debate are concerns about three factors: "bias," "precision," and "relevance." Previous Carnegie Foundation briefs have detailed the reasons why the first two are significant concerns. But, even if value-added scores were unbiased and reasonably precise, their usefulness for evaluating teaching would still depend on the third factor--their relevance to the aims of schooling. The aim of this brief is to consider a key aspect of the relevance of value-added scores: their "predictive validity"--whether teachers who produce high value-added on achievement tests also engender lasting cognitive and non-cognitive skills that help prepare their students for success in later life. Herein, the author reviews 10 studies of the persistence of value-added scores. In each case, researchers compute value-added scores in an "initial year" and then in subsequent years. All studies show that value-added scores tend to fade out over time. Five years after the initial year, it appears that 75 percent to 100 percent of the initial impact has disappeared. This brief considers: (1) the size of the initial value-added effects; (2) the persistence of initial value-added; (3) reported impacts on adult outcomes; (4) potential explanations for these findings and suggestions for further research; and (5) implications for school practice.

Early career teachers make up an increasingly large proportion of the public school teaching force. Often less effective and facing greater challenges than their more experienced counterparts, new teachers tend to leave the profession at high rates and, given that the modal length of teaching experience has now dropped to one year, finding ways to support the growth and development of new teachers has become of paramount importance. One critical way to provide this support is through effective feedback that improves instruction. In this 90-Day Cycle, the authors investigate the factors and processes that contribute to an effective feedback system, paying particular attention to how the system affects early career teachers and how such systems can coexist with extant and emerging teacher evaluation systems. Identified are a set of interconnected drivers at the district, school, and individual levels, that unpack the feedback process into a set of sub-processes that are shaped by individuals, systems, and the larger context in which those systems operate.

Public school teaching is a profession in transition. Already the largest occupation in the United States, it is expanding faster than the nation's student population. Teachers of color are entering the profession at twice the rate of white teachers, reversing an exodus after civil rights victories opened many other doors to African Americans. And women are again entering the profession in greater numbers after years of bypassing the field for other opportunities. But what may be most significant--to students, schools, and the nation--is that teachers today are younger and markedly less experienced than a generation ago. Experts consider teachers with five or fewer years of experience to be still learning their craft. Although the recent recession pushed back the tide somewhat, and has likely raised the level of experience, the sheer number of novices in public school teaching has serious financial, structural, and educational consequences for public education--straining budgets, disrupting school cultures and, most significantly, depressing student achievement. Yet there has been scant discussion of the phenomenon by education policymakers. This report explores the causes, conditions, and consequences of what may be a permanent shift towards a less-experienced profession. It examines escalating levels of teacher attrition in public schools, a major source of the beginning teacher challenge. And it points to promising solutions, especially teacher induction strategies that provide the sort of targeted training and intensive support that recognizes the first years of teaching as the make-or-break opportunities they are.

Under laws providing for equal access to quality education, most American students with disabilities are taught largely in general education classrooms. And along with all other students, they are included in standards-based reforms--policies that hold their schools and teachers accountable for what they do or do not learn. In more and more cases, policies require that a teacher be judged partly by his value-added--a measure of what he contributes to student learning as determined by student scores on standardized tests. States and districts, of course, want to make sure they have accurate measures of these contributions. But students with disabilities pose several challenges for calculating value-added. They tend to score low--often very low--on regular state assessments, and most receive accommodations, such as extra time. The result can be scores that are unreliable or not comparable to those of other students or across years. In other cases, students with disabilities have test scores that cannot be used to calculate value-added because they take alternative assessments. At the same time, many students with disabilities are taught by multiple teachers; they may be taught by two teachers in the same classroom or by different teachers in separate general and special education classrooms. Students with disabilities also often receive help from aides and special services. Disentangling the contribution of each teacher from these other factors may be difficult. In this brief, the authors discuss the challenges of using value-added to evaluate teachers of students with disabilities. They consider the limited empirical research on the potential for systematic errors in value-added for these teachers, either because the models do not adequately account for the likely achievement growth of their students, or because they do not account for teachers being more or less effective for students with disabilities than they are for other students. They also consider the comparability of value-added for special education teachers and the value-added for other teachers.

Teacher evaluation systems introduced by states and school systems in the past several years have focused attention on improving the performance of public school teachers, but they have been cost- and time-intensive, placing a significant burden on states' and districts' resources. In Tennessee, for example, trained evaluators conducted nearly 300,000 classroom observations during the 2011-2012 school year, prompting administrators to complain that "the amount of time spent to implement TEAM [the state's new system] was unmanageable." Even among school leaders who did not feel overburdened by the number of hours spent observing classrooms, many felt the time had not been used efficiently, as the system treated all teachers with the same intensity, despite the fact that teachers' skills and needs for support varied widely. To address these concerns, Tennessee and other systems have replaced their one-size-fits-all evaluation approaches with more differentiated models, using past performance data to determine which teachers should be evaluated with more or less intensity in subsequent evaluation cycles and, in some cases, what that attention should include. This brief explores differentiation strategies in nine districts, two charter management organizations, and three states, Tennessee, Delaware, and Ohio. Interviews with system leaders and analyses of teacher evaluation policies reveal that these systems now vary the format or frequency of formal evaluation cycles, the format or frequency of classroom observations, or the type of observer conducting classroom observations, based on what is known about teachers' needs, strengths, and goals. A list of sources is included.

The question for this brief is whether education leaders can use value-added measures as tools for improving schooling and, if so, how to do this. Districts, states, and schools can, at least in theory, generate gains in educational outcomes for students using value-added measures in three ways: creating information on effective programs, making better decisions about human resources, and establishing incentives for higher performance from teachers. This brief reviews the evidence on each of these mechanisms and describes the drawbacks and benefits of using value-added measures in these and other contexts.

The Community College Pathways (CCP) program had an outstanding second year. In 2012-2013, the program reproduced the positive outcomes realized in the first year of implementation, including successful course completion rates of over 50 percent for both Pathways. Simultaneously, the administration of the Pathways has continued to develop and improve in varied instructional settings across the United States. This report provides descriptive statistics on 2012-2013 student outcomes as well as insights into the challenges and improvements in the second year's implementation of the Pathways. Altogether, the information presented in this report adds to the growing evidence of the Community College Pathways program's ability to reliably generate positive results at scale. A list of participating institutions in the 2012-2013 academic year is appended.

Teacher training programs are increasingly being held under the microscope. Perhaps the most notable of recent calls to reform was the 2009 declaration by U.S. Education Secretary Arne Duncan that "by almost any standard, many if not most of the nation's 1,450 schools, colleges, and departments of education are doing a mediocre job of preparing teachers for the realities of the 21st century classroom." Duncan's indictment comes despite the fact that these programs require state approval and, often, professional accreditation. The problem is that the scrutiny generally takes the form of input measures, such as minimal requirements for the length of student teaching and assessments of a program's curriculum. Now the clear shift is toward measuring outcomes. Much of the existing research on teacher preparation has focused on comparing teachers who enter the profession through different routes--traditional versus alternative certification--but more recently, researchers have turned their attention to assessing individual teacher preparation providers (TPPs). And in doing so, researchers face many of the same statistical challenges that arise when value-added is used to evaluate individual teachers. This document discusses what value-added methods can and cannot say about the quality of TPPs and details what is known about value-added assessments of TPPs.

The idea that multiple measures should be used when evaluating teachers is widely accepted. Multiple measures are important not only because education has multiple goals, but because each measure is an imperfect indicator of any given goal. For a variety of reasons, states and districts use multiple measures in one particular way: to make personnel decisions about teachers based on a weighted average of the separate measures. Also known as a "composite" or "index," the weighted average provides one bottom-­line metric through which teachers can be placed into performance categories. The federal Race to the Top (RTTT) initiative is one reason why states and districts use the weighted average. This competitive grants program required states to hold teachers accountable in a way that made student test scores a "significant factor" in personnel decisions. The meaning of this term is never explained, and the most likely way to meet the vague requirement was to assign large or significant weight--50 percent in some cases--to measures of student achievement growth, such as value-added. While weighted averages are a common and intuitive approach for using multiple measures, there are other options that have their own advantages. In this brief, Douglas Harris also considers the "matrix" and "screening" approaches, which do not involve combining multiple measures. After describing and comparing the weighting, matrix, and screening methods, Harris discusses their strengths and weaknesses according to specific criteria. More than anything else, this brief contributes some new and concrete ways of thinking about how value-added and other measures are used in accountability systems.

This brief considers the problem of using value-added scores to compare teachers who work in different schools. The author focuses on whether such comparisons can be regarded as fair, or, in statistical language, "unbiased." An unbiased measure does not systematically favor teachers because of the backgrounds of the students they are assigned to teach, nor does it favor teachers working in resource-rich classrooms or schools. A key caveat: A measure that is unbiased does not mean the measure is accurate. An unbiased measure could be imprecise--thus inaccurate--if, for example, it is based on a small sample of students or on a test with too few items. This brief focuses strictly on the bias that may arise when comparing the value-added scores of teachers who work in different schools. Raudenbush and Willms' linear model is appended.

Principals play a central role in how well a school performs. They are responsible for establishing school goals and developing strategies for meeting them. They lead their schools' instructional programs, recruit and retain teachers, maintain the school climate, and allocate resources. How well they execute these and other leadership functions is a key determinant of school outcomes. Many districts are trying to create value-added measures for principals much like those they use for teachers. The idea is compelling, but the situations are not necessarily analogous. Estimating value-added for principals turns out to be even more complex than estimating value-added for teachers. Three methods have been suggested for assessing a principal's value-added. One method attributes all aspects of school effectiveness (how well students perform relative to students at other schools with similar background characteristics and students with similar peers) to the principal; a second attributes to the principal only the difference between the effectiveness of that school under that principal and the effectiveness of the same school under other principals; and a third attributes school improvement (gains in school effectiveness) to the principal. This document reviews each of these distinct methods and discusses the current state of knowledge, what more needs to be known, and what can't be resolved by empirical evidence on this issue.

Value-added evaluations use student test scores to assess teacher effectiveness. How student achievement is judged can depend on which test is used to measure it. Thus it is reasonable to ask whether a teacher's value-added score depends on which test is used to calculate it. Would it change if a different test was used? Specifically, might a teacher admonished for poor performance or recognized for good performance have been treated differently if a different test had been used? It's an important question, particularly because most states will soon be adopting new tests aligned with the Common Core State Standards. In this knowledge brief the author discusses what is known about how sensitive value-added scores are to the choice of test and what more needs to be known. He also discusses issues about the choice of test that might not be resolved through empirical investigation, as well as the implications of these findings for states and school districts.

There is a growing body of research on the validity and reliability of value-added measures, but most of this research has focused on elementary grades. Driven by several federal initiatives such as Race to the Top, Teacher Incentive Fund, and ESEA waivers, however, many states have incorporated value-added measures into the evaluations not only of elementary teachers but of middle and high school teachers as well. Almost all states have committed to one of the two Common Core assessments that will test annually in high school, and there is little doubt that value-added will be expanded to the grades in which the new assessments are introduced. In order to assess value-added and the validity and reliability of value-added measures, it is important to consider the significant differences across grades in the ways teachers' work and students' time are organized. This document provides evidence that there are differences in the validity of value-added measures across grades for two primary reasons. First, middle and high schools "track" students; that is, students are assigned to courses based on prior academic performance or other student characteristics. Tracking not only changes our ability to account for differences in the students who teachers educate, but also the degree to which the curriculum aligns with the tests. Second, the structure of schooling and testing vary considerably by grade level in ways that affect reliability in sometimes unexpected ways. The problems are partly correctable, but, as shown herein, more research is necessary to understand how problematic existing measures are and how they might be improved.

Better teacher evaluation should lead to better instruction and improved outcomes for students, but more accurate classification of teachers requires better information than is now available. Because existing measures of performance are incomplete and imperfect, measured performance does not always reflect true performance. Teachers who are truly high-performing may be classified as lower-performing and vice versa. Teacher classification is of course a necessity, and policymakers and practitioners must decide who is eligible to teach in which classrooms and how teachers are promoted or compensated. School leaders have commonly used such measures as academic degrees, teaching experience, classroom observation ratings, and assessments by principals to make these decisions. Less common is the use of "value-added" estimates. Value added measures, however, are being used more and more as data systems are better able to make that student-teacher link. As they are, the measures are raising concerns about the implications for misclassification. Given the risk of misclassification, it is tempting to limit the number of personnel decisions that rely on these imperfect measures. Yet personnel decisions are unavoidable, and they hold significant potential for improving schools. This knowledge brief presents a discussion of what is known about errors in classifying teacher effectiveness, and asks what more needs yet to be known about this issue. The brief touches on what cannot be resolved by empirical evidence on this issue, and advises how and under what circumstances this issue should impact the decisions and actions that districts can make based on teacher evaluations. Better measures of teacher effectiveness, both value-added and other kinds, could reduce both types of error. Given that misclassification is inescapable, reducing the consequences of the problem is an important additional goal and one with potentially large payoffs--particularly for students.

Drawing on a research base developed over many years in education, Carnegie is testing a set of strategies to help students persist and succeed academically. This kind of persistence, what the researchers and faculty who developed the Pathways call "productive persistence," is a key driver of Quantway® and Statway®. Broadly defined, productive persistence is the package of skills and tenacity that students need to succeed in an academic setting. These interventions and others, as well as the story of how Carnegie developed this new program of work addressing the alarming failure rates of students in developmental mathematics, are covered in this report.

In recent years, "continuous improvement" has become a popular catchphrase in the field of education. However, while continuous improvement has become commonplace and well-documented in other industries, such as healthcare and manufacturing, little is known about how this work has manifested itself in education. This white paper attempts to map the landscape of this terrain by identifying and describing organizations engaged in continuous improvement, and by highlighting commonalities and differences among them. The findings classify three types of organizations engaged in continuous improvement: those focused on instructional improvement at the classroom level; those concentrating on system-wide improvement; and those addressing collective impact. Each type is described in turn and illustrated by an organizational case study. Through the analysis, six common themes that characterize all three types of organizations (e.g., leadership and strategy, communication and engagement, organizational infrastructure, methodology, data collection and analysis, and building capacity) are enumerated. This white paper makes four concluding observations. First, the three case studies provide evidence of organizations conducting continuous improvement work in the field of education, albeit at different levels and in different ways. Second, entry points to continuous improvement work are not mutually exclusive, but are nested and, hence, mutually informative and comparative. Third, continuous improvement is not synonymous with improving all organizational processes simultaneously; rather, research and learning cycles are iterative and gradual in nature. Fourth, despite being both iterative and gradual, it is imperative that improvement work is planned and undertaken in a rigorous, thoughtful, and transparent fashion. The following are appended: (1) Selected Continuous Improvement Methodologies; (2) Baldrige Education Criteria for Performance Excellence Framework; (3) Examples of Improvement Artifacts from the School District of Menomonee Falls, Wisconsin; (4) Examples of Improvement Artifacts from Montgomery County Public Schools, Maryland; and (5) Examples of Improvement Artifacts from Strive Partnership Cincinnati.

Building a stronger teacher workforce requires the thoughtful orchestration of multiple processes working together in a human capital system. This white paper presents a framework that can be used to take stock of current efforts to enhance the teacher workforce in school districts or educational organizations, as well as their underlying theories of how the teacher workforce improves over time. The paper refines and provides evidentiary support for a human capital system framework composed of four subsystems that ideally work together to build a stronger teacher workforce: (1) getting the right teachers in the right positions on time (Acquire); (2) supporting professional growth in school-based learning communities (Develop); (3) nurturing, rewarding, and challenging high-performing teachers (Sustain); and (4) informing evidence-based personnel decisions (Evaluate). Attention to this framework will engender a corps of teachers with the capacity and expertise to collectively facilitate enhanced educational outcomes.

A teacher's value-added score is intended to convey how much that teacher has contributed to student learning in a particular subject in a particular year. Different school districts define and compute value-added scores in different ways. A variety of people may see value-added estimates, and each group may use them for different purposes. Teachers themselves may want to compare their scores with those of others and use them to improve their work. Administrators may use them to make decisions about teaching assignments, professional development, pay, or promotion. Parents, if they see the scores, may use them to request particular teachers for their children. And, finally, researchers may use the estimates for studies on improving instruction. Using value-added scores in any of these ways can be controversial. Some people doubt the validity of the achievement tests on which the scores are based, some question the emphasis on test scores to begin with, and others challenge the very idea that student learning gains reflect how well teachers do their jobs. The authors' purpose is to answer a more limited, but essential, question: How might educators reasonably interpret value-added scores? Social science has yet to come up with a perfect measure of teacher effectiveness, so anyone who makes decisions on the basis of value-added estimates will be doing so in the midst of uncertainty. Making choices in the face of doubt is hardly unusual--they routinely contend with projected weather forecasts, financial predictions, medical diagnoses, and election polls. But as in these other areas, in order to sensibly interpret value-added scores, it is important to do two things: understand the sources of uncertainty and quantify its extent. The authors' aim is to identify possible errors of interpretation, to consider how likely these errors are to arise, and to help educators assess how consequential they are for different decisions. They will begin by asking how value-added scores are defined and computed. Next, they will consider two sources of error: statistical bias and statistical imprecision. (Contains 4 figures, 1 table, and 11 endnotes.)

Value-added models have caught the interest of policymakers because, unlike using student tests scores for other means of accountability, they purport to "level the playing field." That is, they supposedly reflect only a teacher's effectiveness, not whether she teaches high- or low-income students, for instance, or students in accelerated or standard classes. Yet many people are concerned that teacher effects from value-added measures "will" be sensitive to the characteristics of her students. More specifically, they believe that teachers of low-income, minority, or special education students will have lower value-added scores than equally effective teachers who are teaching students outside these populations. Other people worry that the opposite might be true--that some value-added models might cause teachers of low-income, minority, or special education students to have "higher" value-added scores than equally effective teachers who work with higher-achieving, less risky populations. In this brief, the author discusses what is and is not known about how well value-added measures level the playing field for teachers by controlling for student characteristics. He first discusses the results of empirical explorations. He then addresses outstanding questions and the challenges to answering them with empirical data. Finally, he discusses the implications of these findings for teacher evaluations and the actions that may be based on them. (Contains 26 endnotes.)

There are good reasons for re-thinking teacher evaluation. Evaluation systems in most school districts appear to be far from rigorous. A recent study showed that more than 99 percent of teachers in a number of districts were rated "satisfactory," which does not comport with empirical evidence that teachers differ substantially from each other in terms of their effectiveness. Likewise, the ratings do not reflect the assessment of the teacher workforce by administrators, other teachers, or students. Evaluation systems that fail to recognize the true differences that are known to exist among teachers greatly hamper the ability of school leaders and policymakers to make informed decisions about such matters as which teachers to hire, what teachers to help, which teachers to promote, and which teachers to dismiss. Thus it is encouraging that policymakers are developing more rigorous evaluation systems, many of which are partly based on student test scores. Yet while the idea of using student test scores for teacher evaluations may be conceptually appealing, there is no universally accepted methodology for translating student growth into a measure of teacher performance. In this brief, the authors review what is known about how measures that use student growth align with one another, and what that agreement or disagreement might mean for policy. (Contains 3 tables and 44 endnotes.)

In the recent drive to revamp teacher evaluation and accountability, measures of a teacher's value added have played the starring role. But the star of the show is not always the best actor, nor can the star succeed without a strong supporting cast. In assessing teacher performance, observations of classroom practice, portfolios of teachers' work, student learning objectives, and surveys of students are all possible additions to the mix. All these measures vary in what aspect of teacher performance they measure. While teaching is broadly intended to help students live fulfilling lives, educators must be more specific about the elements of performance that contribute to that goal--differentiating contributions to academic skills, for instance, from those that develop social skills. Once educators have established what aspect of teaching they intend to capture, the measures differ in how valid and reliable they are in capturing that aspect. Although there are big holes in what they know about how evaluation measures stack up on these two criteria, they can draw some important conclusions from the evidence collected so far. In this brief, the author will show how existing research can help district and state leaders who are thinking about using multiple measures of teacher performance to guide them in hiring, development, and retention. (Contains 30 endnotes.)

Value-added models measure teacher performance by the test score gains of their students, adjusted for a variety of factors such as the performance of students when they enter the class. The measures are based on desired student outcomes such as math and reading scores, but they have a number of potential drawbacks. One of them is the inconsistency in estimates for the same teacher when value added is measured in a different year, or for different subjects, or for different groups of students. Some of the differences in value added from year to year result from true differences in a teacher's performance. Differences can also arise from classroom peer effects; the students themselves contribute to the quality of classroom life, and this contribution changes from year to year. Other differences come from the tests on which the value-added measures are based; because test scores are not perfectly accurate measures of student knowledge, it follows that they are not perfectly accurate gauges of teacher performance. In this brief, the authors describe how value-added measures for individual teachers vary across time, subject, and student populations. They discuss how additional research could help educators use these measures more effectively, and they pose new questions, the answers to which depend not on empirical investigation but on human judgment. Finally, they consider how the current body of knowledge, and the gaps in that knowledge, can guide decisions about how to use value-added measures in evaluations of teacher effectiveness. (Contains 3 tables and 17 endnotes.)