Search

Many districts and states have implemented incentives to recruit teachers to low-performing schools, and previous research has found evidence that these incentives are effective at attracting teachers. However, effects on the schools and students these teachers leave behind have not been examined. This study focuses on the spillover effects of recruiting effective teachers to Tennessee’s Innovation Zone (iZone) schools. We find the short-term effects of losing these teachers range from −0.04 to −0.12 SDs in student test score gains, with larger negative effects when more effective teachers leave. However, combining both these negative effects in schools teachers leave and the positive effects in iZone schools yields overall net positive effects.

Many early childhood programs are not prepared to meet the needs of children who have significant social, emotional, and behavioral challenges. Program-Wide Supports for Pyramid Model Implementation (PWS-PMI) provides a systematic approach to supporting early childhood programs using Pyramid Model practices and enhancing children's social-emotional outcomes that is grounded in implementation science. We designed the current study to test the PWS-PMI intervention and examine its feasibility of implementation in community-based early childhood programs serving children from low-income environments. In this study, we found programs increased their implementation of PWS-PM and improved classroom practices after only one year of support. Our findings indicate a program-wide approach is effective, although more time and support will be necessary to sustain high fidelity implementation and produce robust effects on children.

Purpose: While previous research has examined the impact of school turnaround models, less is known about the principals who lead these turnaround schools. This study examines the personal demographics, experience, educational background, prior school performance, salaries, and turnover of principals who led two turnaround models in Tennessee's lowest performing schools: a state-run Achievement School District (ASD) that has not yielded positive nor negative effects and local Innovation Zones (iZones) that averaged positive effects on student achievement over six years. Methods: We analyze longitudinal, administrative data from the Tennessee Department of Education from 2006–2007 to 2017–2018 to compare pre- and post-reform means and trends in principal characteristics between ASD, iZone, and similarly low-performing comparison schools. Results: ASD schools had higher principal turnover rates and lost principals whose schools performed higher while iZone schools retained more principals and lost principals whose schools performed lower. Moreover, iZone schools employed more experienced principals, more Black principals, and principals with higher graduate degree attainment and paid their principals more than ASD schools. Salary differences between ASD and iZone schools were not explained by principals’ characteristics, such as years of experience. Implications: Our findings reveal differences in leadership characteristics between iZone and ASD schools that were consistent with differences in the effectiveness of the two turnaround approaches.

In contrast to prior federally mandated school reforms, the Every Student Succeeds Act (ESSA) allows states more discretion in reforming their lowest performing schools, removes requirements to dis...

Attracting and retaining teachers can be an important ingredient in improving low-performing schools. In this study, we estimate the expressed preferences for teachers who have worked in low-performing schools in Tennessee. Using adaptive conjoint analysis survey design, we examine three types of school attributes that may influence teachers’ employment decisions: fixed school characteristics, structural features of employment, and malleable school processes. We find that teachers express a strong preference for two malleable school processes, administrative support and discipline enforcement, along with a higher salary, a structural feature. Estimates indicate these attributes are 2 to 3 times more important to teachers than fixed school characteristics like prior achievement. We validate our results using administrative data on teachers’ revealed preferences.

A growing body of research evaluates the effects of turnaround on chronically low-performing schools. We extend this literature by formally testing factors that may either mediate or suppress the effects of two turnaround initiatives in Tennessee: the Achievement School District (ASD) and local Innovation Zones (iZones). Using difference-in-differences models within a mediational framework, we find that hiring effective teachers and principals partially explains positive iZone effects. In the ASD, high levels of teacher turnover suppress potential positive effects. Also, in iZone schools, increased levels of student mobility and chronic absenteeism suppress potentially larger positive effects. Policies that increase capacity within turnaround schools, such as financial incentives for effective staff, appear to be important ingredients for realizing positive effects from turnaround reforms.

Using unique administrative data from North Carolina that allow us to separate classroom teacher turnover during the school year from end-of-year turnover, we find students who lose their teacher during the school year have significantly lower test score gains (on average −7.5 percent of a standard deviation unit) than those students whose teachers stay. Moreover, the turnover of other teachers during the year lowers achievement gains, whereas end-of-year teacher turnover appears to have little effect on achievement. The harmful effects of within-year turnover cannot be explained by other extraneous shocks or the quality of departing teachers. Teachers who depart from December through April have the most harmful effects on achievement, although these vary somewhat by level of schooling and subject.

Online credit recovery (OCR) refers to online courses that high school students take after previously failing the course. Many have suggested that OCR courses are helping students to graduate from high school without corresponding increases in academic skills. This study analyzes administrative data from the state of North Carolina to evaluate the efficacy of OCR using full data from public and private OCR providers. Findings indicate that students who fail courses and enroll in OCR are 20 percentage points more likely to earn course credit, have lower test scores of up to two tenths of a standard deviation, and are about eight percentage points more likely to graduate high school within 4 years than students who repeat courses traditionally. Test score differences are particularly large for Biology compared to Math I and English II. Hispanic and economically disadvantaged OCR students are more likely to graduate high school than their peers.

Teacher turnover occurs during and at the end of the school year, although documentation of within-year turnover currently rests on anecdotal evidence. On average, over 4.6% of teachers turn over during the school year, which amounts to 25% of total annual turnover. Teachers transfer within districts at higher rates at the beginning of the school year and leave teaching at higher rates at the beginning of the spring semester. Higher performing teachers are less likely to turn over during the school year and less likely to turn over within the year than at the end of the year. Teach for America corps members are much less likely and teachers from out of state much more likely to turn over during the school year than traditionally prepared teachers.

The Individuals with Disabilities Education Act stresses the importance of educating students with disabilities (SWDs) in the least restrictive environment, often with peers who do not have disabilities. Prior research has examined the extent to which SWDs are included in general education classrooms, but not the characteristics of the peers with whom SWDs are educated. We examined the math classmates of fourth- and fifth-grade SWDs from one state. On average, SWDs were grouped with twice as many other SWDs, about four per class, than students without disabilities. Students with learning disabilities had fewer peers with disabilities in their classrooms than students with other disabilities. Students with intellectual disabilities, autism, or emotional/behavioral disorders more often had peers with disabilities, often their same disability. Our results provide directions for future research regarding peer effects and understanding how schools group SWDs.

In recent years, the federal government has invested billions of dollars to reform chronically low-performing schools. To fulfill their federal Race to the Top grant agreement, Tennessee implemented three turnaround strategies that adhered to the federal restart and transformation models: (a) placed schools under the auspices of the Achievement School District (ASD), which directly managed them; (b) placed schools under the ASD, which arranged for management by a charter management organization; and (c) placed schools under the management of a district Innovation Zone (iZone) with additional resources and autonomy. We examine the effects of each strategy and find that iZone schools, which were separately managed by three districts, substantially improved student achievement. In schools under the auspices of the ASD, student achievement did not improve or worsen. This suggests that it is possible to improve schools without removing them from the governance of a school district.

Locally-scored teacher candidate performance assessments offer teacher preparation programs (TPPs) formative performance data, common language and expectations, and information to guide program improvements. To best use these data, TPPs need to understand the validity and reliability of local scoring and assess whether scores predict candidates’ performance as teachers. Examining locally-scored performance assessments, we find that local scores are significantly higher than official scores. However, local scores identify three factors partially-aligned with the assessment’s construct blueprint and significantly predict teachers’ performance outcomes. These analyses provide a framework for research and highlight the utility of locally-scored performance assessments for evidence-based TPP improvement.

Between 2006 and 2010, the North Carolina Department of Public Instruction intervened in 128 low-performing schools, combining approaches consistent with school restructuring and transformation. In improved schools, local educators reconstructed key school functions, a distinctly nonlinear process more like the work of skilled craftsmen than that of design engineers that we refer to as “scaffolded craftsmanship.” We interviewed key stakeholders in 12 high schools to learn about the dynamics accounting for the improvement or stalemate at each school. In sum, in the improved schools we studied, the turnaround process was not a matter of initial external design and subsequent implementation, but a non-linear process of planning, inventing, adjusting, and re-planning as well as a process of learning, doing, and learning from doing. The improvement generally began with the installation of new leadership and involved four main components: new commitment, climate, and culture; improved knowledge and skills; strategically organized and managed structures and supports for instruction; and strengthened external support. Our findings suggest that judicious personnel replacement followed by professional development and coaching targeted to key functions may be a more effective method for implementing school turnaround than the structural approaches promoted via NCLB sanctions and Race to the Top.

Many states have responded to teacher shortages by granting certification to individuals traditionally prepared out-of-state; now, out-of-state prepared teachers comprise a sizable percentage of the teacher workforce in many states. We know little about these teachers, and therefore, in the present study, we estimate the effectiveness of out-of-state prepared teachers in North Carolina elementary schools. We find that out-of-state prepared teachers are significantly less effective than in-state prepared and alternative entry teachers; however, there is a substantial overlap in the distributions of effectiveness across groups. Upon testing hypotheses to explain these findings, results indicate that differences in human capital help explain out-of-state prepared teachers’ underperformance and suggest the utility of research evidence to inform state policy and local hiring decisions.

Background: Nascent empirical research demonstrates the sizable impacts of principals on student achievement. More research is needed on the pathways to the principalship and how principals’ characteristics and training experiences influence their performance. Purpose: (1) To describe the characteristics of first-time principals and the schools that hire them and (2) to assess the extent to which the characteristics of early-career principals and the environments in which they previously worked are associated with changes in student achievement in the schools they lead. Setting: North Carolina public schools. Sample: All first-time principals (981 in total) from 2006-2007 through 2009-2010 and the students attending the schools where they serve. Data: Administrative data on students, school personnel (teachers, assistant principals, principals), and schools provided by the North Carolina Department of Public Instruction. Research Methods: Covariate adjustment and fixed effects value-added models. Findings: We find that first-time principals are “homegrown”—hired from within the district. On average, first-time principals wait 5.12 years between completing formal principal preparation and assuming school leadership and spend 4.15 years as assistant principals in North Carolina public schools. Several principal characteristics, including holding a doctorate from an in-state private or out-of-state institution (negative) and serving as an assistant principal in a high–value-added school (positive), are significantly associated with student achievement gains. Conclusions: This study suggests that the effectiveness of early-career principals may be affected by the environment where they served as assistant principals. Further analyses are needed to better understand the attributes of meaningful assistant principal experiences.

State policies affect the qualifications of beginning teachers in numerous ways, including regulating entry requirements, providing incentives for graduate degrees, and subsidizing preparation programs at public universities. In this paper we assess how these policy choices affect student achievement, specifically comparing traditionally prepared with alternative-entry teachers; in-state traditionally prepared with out-of-state traditionally prepared teachers; teachers beginning with undergraduate degrees with those beginning with graduate degrees; and teachers prepared at in-state public universities with those prepared at in-state private universities. Using school fixed effects to analyze data from North Carolina, we find that: Teach For America corps members are more effective than traditionally prepared teachers; other alternative-entry teachers are less effective than traditionally prepared instructors in high school mathematics and science courses; and out-of-state traditionally prepared teachers are less effective than in-state traditionally prepared teachers, especially in elementary subjects where they constitute nearly 40 percent of the workforce. © 2014 Association for Education Finance and Policy

Performance-based accountability along with budget tightening has increased pressure on publicly funded organizations to develop and deliver programs that produce meaningful social benefits. As a result, there is increasing need to undertake formative evaluations that estimate preliminary program outcomes and identify promising program components based on their effectiveness during implementation. By combining longitudinal administrative data, multiple comparison group designs, and a progressive series of analyses that test rival explanations, evaluators can strengthen causal arguments and provide actionable program information for key stakeholders to improve program outcomes. In this article, we illustrate the application of rigorous methods to estimate preliminary program effects and rule out alternative explanations for preliminary effects, including site selection bias, individual selection bias, and resentful demoralization through the evaluation of the Collaborative Project, a North Carolina educational improvement project that incorporated multiple components aimed at boosting student achievement.

The current teacher workforce is younger, less experienced, more likely to turnover, and more diverse in preparation experiences than the workforce of two decades ago. Research shows that inexperienced teachers are less effective, but we know little about the effectiveness of teachers with different types of preparation. In this study, we classify North Carolina public school teachers into portals—fixed and mutually exclusive categories that capture teachers’ formal preparation and qualifications upon first entering the profession—and estimate the adjusted average test score gains of students taught by teachers from each portal. Compared with undergraduate-prepared teachers from in-state public universities, (a) out-of-state undergraduate-prepared teachers are less effective in elementary grades and high school, (b) alternative entry teachers are less effective in high school, and (c) Teach For America corps members are more effective in STEM subjects and secondary grades.

Calls for evidence-based reform of teacher preparation programs (TPPs) suggest the question: Do the current indicators of progress and performance used by TPPs predict effectiveness of their graduates when they become teachers? In this study, the indicators of progress and performance used by one program are examined for their ability to predict value-added scores of program graduates. The study finds that rating instruments, including disposition surveys, clinical practice observation ratings, and portfolio assessments, each measure a single underlying dimension rather than the multiple constructs they were designed to measure. Neither these instruments nor teacher candidates’ scores on standardized exams predict their later effectiveness in the classroom based on value-added models of student achievement. Candidates’ grade point averages during their preparation program and number of math courses were positively associated with their students’ math score gains. These findings suggest a need for better instruments to measure prospective teachers’ progress toward proficiency.

To address gaps in achievement between more- and less-affluent students, states and districts need to ensure that high-poverty students and schools have equitable access to educational resources. Traditionally, assessments of resource equity have focused on per-pupil expenditures and more proximal inputs, such as teacher credentials and class size, despite the inconsistent and/or weak relationships between these measures and student performance. Given the sizable and direct effects of teachers on student achievement, we argue that (1) teachers’ value-added scores should be incorporated into assessments of resource equity and (2) providing schools with greater flexibility for setting salaries or using strategic staffing initiatives may be necessary to achieve an equitable distribution of effective teachers. To illustrate these assertions we incorporate teacher value added into a case study of resource allocation in the public high schools of Wayne County, North Carolina, which have been the target of a complaint by the North Carolina National Association for the Advancement of Colored People. © 2013 Association for Education Finance and Policy

New federal and state policies require that teacher preparation programs (TPP) be held accountable for the effectiveness of their graduates as measured by test score gains of the students they teach. In this article, the authors review the approaches taken in several states that have already estimated TPP effects and analyze the proposals for incorporating students’ test score gains into the evaluations of TPP by states that have received federal Race to the Top funds. The authors organize their review to focus on three types of decisions that are required to implement these new accountability requirements: (a) selection of teachers, students, subjects, and years of data; (b) methods for estimating teachers’ effects on student test score gains; and (c) reporting and interpretation of effects. The purpose of the review is to inform the teacher preparation community on the state of current and near term practice for adding measures of teacher effectiveness to TPP accountability practices.

Is a popular innovation for increasing human capital in the teaching profession—competitive college scholarships for teachers—effective? The authors show that one large and long-standing merit-based scholarship program (a) attracts teacher candidates who have high academic qualifications; (b) yields graduates who teach lower performing students, although not as challenging as the students of other beginning teachers; (c) produces teachers who raise high school and third- through eighth-grade mathematics test scores more than other traditionally prepared teachers do; and (d) produces teachers who stay in public school classrooms for 5 years or more at higher rates than alternative entry or other traditionally prepared teachers.

Research on teacher development reports significant early-career increases in teacher effectiveness, but the extent to which this is attributable to the development of teachers who persist or to the attrition of less effective teachers is unclear. In this study of novice teachers in North Carolina public schools, the authors investigated the development of teachers’ effectiveness during their first five years in the classroom and contrasted the effectiveness of teachers who stayed with that of those who left. Across grade levels, teachers’ effectiveness increased significantly in their second year of teaching but flattened after three years. The teachers who left the profession were less effective, on average, than those who stayed at least five years, but this finding is somewhat less consistent than the findings of an initial jump in effectiveness and diminishing returns to on-the-job development.

Evaluating the impacts of public school funding on student achievement has been an important objective for informing education policymaking but fraught with data and methodological limitations. Findings from prior research have been mixed at best, leaving policymakers with little advice about the benefits of allocating public resources to schools or how it might best be done. In this study, the authors take advantage of a pilot supplemental funding program in North Carolina that used a quantitative index of educational advantage to select the most educationally disadvantaged districts in the state to receive funding. The targeted districts received supplemental funds of $250 per pupil or $840 per academically disadvantaged pupil for the 2 years of the pilot. Using a regression discontinuity design and multilevel models with extensive controls, the authors estimate that the marginal average treatment effect of the supplemental funding was 0.133 standard deviation units and that the effect on educationally disadvantaged students was 0.098 standard deviation units. The treatment effect represents approximately one third of the difference between the average score in top performing and low performing high schools.

In an effort to provide high-quality preschool education, policymakers are increasingly requiring public preschool teachers to have at least a Bachelor's degree, preferably in early childhood education. Seven major studies of early care and education were used to predict classroom quality and children's academic outcomes from the educational attainment and major of teachers of 4-year-olds. The findings indicate largely null or contradictory associations, indicating that policies focused solely on increasing teachers' education will not suffice for improving classroom quality or maximizing children's academic gains. Instead, raising the effectiveness of early childhood education likely will require a broad range of professional development activities and supports targeted toward teachers' interactions with children.

Currently, a disjuncture exists between the economic literature on young children's outcomes from early education or child care and the literature on schooling outcomes of older children and adolescents. Peer effects have been found to be both theoretically important and empirically significant in school settings, yet the effects of peers have not been incorporated into research estimating the child development production function. In this study, we estimate the value-added effects of peer abilities on the educational outcomes of a probability sample of four year olds who attended Head Start, publicly subsidized pre-kindergarten, or private preschool in Georgia. We use a longitudinal data set that includes measures of preschool quality, child and family characteristics, peer abilities, and assessments of the skills of pre-kindergarteners both before and after attending preschool. The ability level of the peers in a child's classroom has direct and positive effects on the child's cognitive skills, pre-reading skills, and expressive language skills after controlling for preschool resources, family characteristics, and the child's skills at the beginning of preschool. Neither time spent on discipline, nor contextual effects of classroom composition, nor teachers’ motivation appear to be the mechanisms that explain the influences of peers on children's skill development.

The debates over the 2003 reauthorization of Head Start highlighted a controversy about the devolution of federal early education policy. At the center of the debate is the concern that state control of early education programs will reduce the quality and effectiveness of federal support for children living in poverty, and their families. The current fragmentation of early education policy, with both federal Head Start programs and state-subsidized prekindergarten programs operating in close proximity, presents an opportunity to compare the programs’ quality and effectiveness within a region of common support. In this study, propensity score techniques were used to match a probability sample of Head Start participants in Georgia with a group of children who were eligible for Head Start but who attended the state prekindergarten program in Georgia. The two groups were statistically similar at the beginning of their preschool year on three of four direct assessments (p < .05), but by the beginning of kindergarten the children attending the state prekindergarten program posted higher developmental outcomes on five of six direct assessments (p < .05) and 14 of 17 ratings by kindergarten teachers (p < .05). This study indicates that economically disadvantaged children who attended Georgia’s universal prekindergarten entered kindergarten at least as well prepared as similar children who attended the Head Start program.

Why is there an increased need for quantitative indicators of student socioeconomic status at the school level? What alternative approaches have been used to measure SES? Can principals accurately estimate the SES distribution of their student populations? How do their estimates correlate with other indexes o f family SES?

As a part of efforts to evaluate and monitor the increasing public investment in early childhood education, teachers are being asked to assess children's school readiness. In this study, preschool teachers and kindergarten teachers rated children's skills in three areas (kindergarten readiness, academic skills, and communication skills), and these ratings were compared with direct assessments of the children's skills. Ratings by both groups of teachers tended to be more highly related to basic skills, such as counting and number naming, than to abilities such as solving applied problems and using expressive and receptive vocabulary. Preschool teachers' ratings had a lower association with children's observed skills and abilities than kindergarten teachers' ratings. Ratings of children attending Head Start were systematically inflated, but this relationship was mediated to a significant extent by the teachers' levels of education. More educated teachers rated children in a manner consistent with the children's directly assessed skills. Implications of these findings for informing future efforts to assess school readiness by using teacher ratings are discussed.

The emergence of publicly subsidized preschool raises important policy questions about the role of market forces and, in places where competition to provide these services exists, presents a setting in which the effects of competition on educational outcomes can be tested. We test neo-institutional hypotheses concerning the effects of competition to provide publicly-funded prekindergarten (pre-k) services on the performance of public schools and private organizations. We use student-level data collected over a five-year period on a large sample of children who attended publicly subsidized prekindergarten in Georgia. Overall, we find that more competition improves third grade reading and math test scores but does not significantly affect retention or school readiness ratings during elementary school. Not all children are equally affected by competition; for example, greater competition significantly decreases the likelihood of retention for children of the working poor. Contrary to the expectations of some theorists, both public schools and private organizations respond to increased competition in ways that improve test scores but not retention. However, children attending private prekindergarten have higher language arts scores and lower retention across the range of competition when compared with children who attended public school pre-k. © 2006 by the Association for Public Policy Analysis and Management

The specifics of the case (the context) are highly important to how different theorists will react in conducting an evaluation. By this, we think of contexts in a very broad sense. A context is not only a disciplinary field (say, education) or a specific kind of program within that field (Head Start, for example), but also all the particularities that relate to the organization, the individuals within that organization, and those who are affected by the program. The context is also influenced by how broadly the evaluator chooses to view the program. In the instance of Bunche‐Da Vinci, we have an elementary school in a low-income, heavily bilingual area. We also have a program that is a unique partnership of the school and the Da Vinci Learning Corporation. Each of the theorists has done evaluations related to education, though not necessarily elementary schools. The four theorists admittedly have differential levels of experience in dealing with instances like Bunche‐Da Vinci. King and Greene have had the most firsthand experience with situations like that presented to them. Henry has dealt with the evaluation of educational programs, but within the context of public policy. Thus, his experience with evaluating education is at a more macro level. Donaldson has performed evaluations in a variety of fields, with his primary exposure to education outside the context of K‐12. What impact does this have on the way in which the theorists deal with Bunche‐Da Vinci? The way this exercise was constructed also has implications for how the theorists might approach the evaluation. Some theoretical approaches

In 1993, the creation of Georgia’s HOPE Scholarship Program accelerated interest in understanding the effects of merit-based student financial aid. This article compares a sample of “borderline” HOPE recipients (students just above the eligibility threshold) with similar nonrecipients to examine differences on four college performance outcomes. The HOPE Scholarship recipients accumulated more credit hours, achieved slightly higher grade point averages, and were more likely to have graduated after 4 years of college. In addition, HOPE recipients who attended 4-year institutions of higher education were more likely to persist in college. Most merit aid recipients lost their scholarships, however, which slightly reduced recipients’advantages on grade point average and credit hour accumulation. Differences in persistence and graduation are significant only for those who maintain eligibility for the scholarship, suggesting that scholarship retention is critical if merit aid programs are to help achieve several of the broad goals of higher education.

Past literature has identified several putative precursors of use, as well as alternative forms of use. However, important shortcomings still exist in previous work on use. In particular, inadequate attention has been given to the underlying processes that may mediate the effects of evaluation on attitude and action. In essence, a key part of the theory of change for evaluation itself is missing. To help fill this gap, we describe a framework designed to capture key mechanisms through which evaluation may have its effects. The framework includes change processes that have been validated in various social science literatures. It identifies three levels of analysis (individual, interpersonal and collective), each with four kinds of processes (general influence, attitudinal, motivational and behavioral). With a more comprehensive view of the mechanisms underlying evaluation’s influence, the field can move forward in relation to its understanding and facilitation of evaluation’s role in the service of social betterment.

Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie between the evaluation and its ultimate goal—social betterment—have been sufficiently developed. We draw a number of these change mechanisms, such as justification, persuasion, and policy diffusion, from the social science research literature, and organize them into a framework that has three levels: individual, interpersonal, and collective. We illustrate how these change processes can be linked together to form “pathways” or working hypotheses that link evaluation processes to outcomes that move us along the road toward the goal of social betterment. In addition, we join with Kirkhart (2000) in moving beyond use, to focus our thinking on evaluation influence. Influence, combined with the set of mechanisms and interim outcomes presented here, offers a better way for thinking about, communicating, and adding to the evidence base about the consequences of evaluation and the relationship of evaluation to social betterment.

Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie between the evaluation and its ultimate goal—social betterment—have been sufficiently developed. We draw a number of these change mechanisms, such as justification, persuasion, and policy diffusion, from the social science research literature, and organize them into a framework that has three levels: individual, interpersonal, and collective. We illustrate how these change processes can be linked together to form “pathways” or working hypotheses that link evaluation processes to outcomes that move us along the road toward the goal of social betterment. In addition, we join with Kirkhart (2000) in moving beyond use, to focus our thinking on evaluation influence. Influence, combined with the set of mechanisms and interim outcomes presented here, offers a better way for thinking about, communicating, and adding to the evidence base about the consequences of evaluation and the relationship of evaluation to social betterment. Use is a core construct in the field of evaluation. 1 Most if not all evaluators strive to have their evaluations used. Many if not most evaluators accept the idea that, at least in part, the merit of their work—the success or failure of their evaluation efforts—can be judged in terms of

Evaluations can influence perceptions about social problems, the selection of social policies, and adapting policy implementation. In this paper, exemplars of influential evaluations are described. These evaluations and the qualities which characterize them give us evidence to consider as we develop a clearer picture of what evaluation should look like in the future.

Using Christie's research as an example, the authors describe a variety of forms that a more evidence-based approach to evaluation theory could take and offer some suggestions to help increase the amount and impact of evidence in evaluation theory.

In the wake of the 1990 amendments to the Clean Air Act, localities across the United States initiated public information campaigns both to raise awareness of threats to air quality and to change behavior related to air pollution by recommending specific behavioral changes in the campaign messages. These campaigns are designed to reduce the health hazards associated with poor air quality and to avoid federal sanctions resulting from the failure to meet air quality standards. As in many other communities across the country, a coalition of government agencies and businesses initiated a public information campaign in the Atlanta metropolitan region to reduce certain targeted behaviors, mainly driving. A two-stage model used to analyze data from a rolling sample survey shows that the centerpiece of the information campaign—air quality alerts—was effective in raising awareness and reducing driving in a segment of the population. When the overall information campaign was moderated by employers' participation in programs to improve air quality, drivers significantly reduced the number of miles they drove and the number of trips they took by car on days when air quality alerts were sounded. Public information campaigns can be successful in increasing awareness, but changing well-established behaviors, such as driving, is likely to require institutional mediation to provide social contexts that support the behavioral change, as well. © 2003 by the Association for Public Policy Analysis and Management.

What we expect from public programs determines how the programs will be judged. This article presents a systematic values inquiry used to examine how much importance citizens and stakeholders attached to specific indicators in determining the success of a public preschool program. Four groups were surveyed: teachers, administrators, parents and the public. All four groups showed significant differences with respect to the importance of 29 possible indicators. The four groups agreed on the importance of high-quality services and about the lack of importance associated with economic benefits for families and certain educational outcomes for the preschoolers. Using confirmatory factor analysis, the opinions of the groups were shown to have similar underlying structures; however, teachers made greater distinctions between the groups of potential outcomes and the public distinguished between outcomes the least. Systematic values inquiry can be an important tool in designing evaluations that will produce information that can influence the judgments of citizens and stakeholders about a program's value.