Search

Between 2006 and 2010, the North Carolina Department of Public Instruction intervened in 128 low-performing schools, combining approaches consistent with school restructuring and transformation. In improved schools, local educators reconstructed key school functions, a distinctly nonlinear process more like the work of skilled craftsmen than that of design engineers that we refer to as “scaffolded craftsmanship.” We interviewed key stakeholders in 12 high schools to learn about the dynamics accounting for the improvement or stalemate at each school. In sum, in the improved schools we studied, the turnaround process was not a matter of initial external design and subsequent implementation, but a non-linear process of planning, inventing, adjusting, and re-planning as well as a process of learning, doing, and learning from doing. The improvement generally began with the installation of new leadership and involved four main components: new commitment, climate, and culture; improved knowledge and skills; strategically organized and managed structures and supports for instruction; and strengthened external support. Our findings suggest that judicious personnel replacement followed by professional development and coaching targeted to key functions may be a more effective method for implementing school turnaround than the structural approaches promoted via NCLB sanctions and Race to the Top.

Many states have responded to teacher shortages by granting certification to individuals traditionally prepared out-of-state; now, out-of-state prepared teachers comprise a sizable percentage of the teacher workforce in many states. We know little about these teachers, and therefore, in the present study, we estimate the effectiveness of out-of-state prepared teachers in North Carolina elementary schools. We find that out-of-state prepared teachers are significantly less effective than in-state prepared and alternative entry teachers; however, there is a substantial overlap in the distributions of effectiveness across groups. Upon testing hypotheses to explain these findings, results indicate that differences in human capital help explain out-of-state prepared teachers’ underperformance and suggest the utility of research evidence to inform state policy and local hiring decisions.

To address gaps in achievement between more- and less-affluent students, states and districts need to ensure that high-poverty students and schools have equitable access to educational resources. Traditionally, assessments of resource equity have focused on per-pupil expenditures and more proximal inputs, such as teacher credentials and class size, despite the inconsistent and/or weak relationships between these measures and student performance. Given the sizable and direct effects of teachers on student achievement, we argue that (1) teachers’ value-added scores should be incorporated into assessments of resource equity and (2) providing schools with greater flexibility for setting salaries or using strategic staffing initiatives may be necessary to achieve an equitable distribution of effective teachers. To illustrate these assertions we incorporate teacher value added into a case study of resource allocation in the public high schools of Wayne County, North Carolina, which have been the target of a complaint by the North Carolina National Association for the Advancement of Colored People. © 2013 Association for Education Finance and Policy

Why is there an increased need for quantitative indicators of student socioeconomic status at the school level? What alternative approaches have been used to measure SES? Can principals accurately estimate the SES distribution of their student populations? How do their estimates correlate with other indexes o f family SES?

In 1993, the creation of Georgia’s HOPE Scholarship Program accelerated interest in understanding the effects of merit-based student financial aid. This article compares a sample of “borderline” HOPE recipients (students just above the eligibility threshold) with similar nonrecipients to examine differences on four college performance outcomes. The HOPE Scholarship recipients accumulated more credit hours, achieved slightly higher grade point averages, and were more likely to have graduated after 4 years of college. In addition, HOPE recipients who attended 4-year institutions of higher education were more likely to persist in college. Most merit aid recipients lost their scholarships, however, which slightly reduced recipients’advantages on grade point average and credit hour accumulation. Differences in persistence and graduation are significant only for those who maintain eligibility for the scholarship, suggesting that scholarship retention is critical if merit aid programs are to help achieve several of the broad goals of higher education.

Past literature has identified several putative precursors of use, as well as alternative forms of use. However, important shortcomings still exist in previous work on use. In particular, inadequate attention has been given to the underlying processes that may mediate the effects of evaluation on attitude and action. In essence, a key part of the theory of change for evaluation itself is missing. To help fill this gap, we describe a framework designed to capture key mechanisms through which evaluation may have its effects. The framework includes change processes that have been validated in various social science literatures. It identifies three levels of analysis (individual, interpersonal and collective), each with four kinds of processes (general influence, attitudinal, motivational and behavioral). With a more comprehensive view of the mechanisms underlying evaluation’s influence, the field can move forward in relation to its understanding and facilitation of evaluation’s role in the service of social betterment.

In the wake of the 1990 amendments to the Clean Air Act, localities across the United States initiated public information campaigns both to raise awareness of threats to air quality and to change behavior related to air pollution by recommending specific behavioral changes in the campaign messages. These campaigns are designed to reduce the health hazards associated with poor air quality and to avoid federal sanctions resulting from the failure to meet air quality standards. As in many other communities across the country, a coalition of government agencies and businesses initiated a public information campaign in the Atlanta metropolitan region to reduce certain targeted behaviors, mainly driving. A two-stage model used to analyze data from a rolling sample survey shows that the centerpiece of the information campaign—air quality alerts—was effective in raising awareness and reducing driving in a segment of the population. When the overall information campaign was moderated by employers' participation in programs to improve air quality, drivers significantly reduced the number of miles they drove and the number of trips they took by car on days when air quality alerts were sounded. Public information campaigns can be successful in increasing awareness, but changing well-established behaviors, such as driving, is likely to require institutional mediation to provide social contexts that support the behavioral change, as well. © 2003 by the Association for Public Policy Analysis and Management.

What we expect from public programs determines how the programs will be judged. This article presents a systematic values inquiry used to examine how much importance citizens and stakeholders attached to specific indicators in determining the success of a public preschool program. Four groups were surveyed: teachers, administrators, parents and the public. All four groups showed significant differences with respect to the importance of 29 possible indicators. The four groups agreed on the importance of high-quality services and about the lack of importance associated with economic benefits for families and certain educational outcomes for the preschoolers. Using confirmatory factor analysis, the opinions of the groups were shown to have similar underlying structures; however, teachers made greater distinctions between the groups of potential outcomes and the public distinguished between outcomes the least. Systematic values inquiry can be an important tool in designing evaluations that will produce information that can influence the judgments of citizens and stakeholders about a program's value.

One highly touted use of performance measurement systems is to assess the merit and worth of reforms. In this study, the effect of the League of Professional Schools, a democratic reform initiative in Georgia, was evaluated using performance measures from the state’s educational performance measurement system. The findings indicate that the League, in combination with an antecedent condition, motivated leadership, produced more widespread participation in staff development than other schools. In addition, schools that were relatively successful in implementing the tenets of the program exhibited modestly improved levels of student achievement over similar schools. However, the League schools did not outperform schools involved in another school reform that was instituted with the same antecedent condition—motivated leadership. Although both reforms were associated with modestly better student performance, the League seemed to trigger more teacher involvement in school governance than did the alternative reform. There is, however, no evidence that the antecedent condition, motivated leadership, was not sufficient by itself to cause the higher levels of student performance. Analysis of the performance measurement data allowed the merit of the League to be assessed against several different performance standards. Although the performance of the League’s schools was positive relative to several of these performance standards, in the end it was impossible to use performance measures to show that the League was a necessary component of the causal package that resulted in improved performance.

This chapter argues that pursuing use as the defining goal for evaluation can distort the allocation of evaluation resources and reduce the contributions of evaluation to broader social goals, such as social betterment.

We comment on a set of papers in this journal issue that have examined three evaluation theories by constructing a logic model for each and conducting a set of comparative content analyses. We consider the set of papers as an instance of research on evaluation. For instance, we question the extent to which linear logic models convey the contingent decision making emphasized by some evaluation theories. We also offer specific comments about the papers’ treatment of one theory, so-called emergent realist evaluation.

Performance monitoring systems have typically been “top-down” evaluation approaches. In Virginia, however, a stakeholder approach was used to develop an educational-performance monitoring system. Four stakeholder groups (teachers, school superintendents, school board members, and education group representatives) were identified, and representatives of the groups were involved in developing the system. Stakeholder group representatives viewed their impact on the development of the system positively. Teachers, who might be expected to resist a performance monitoring system, were the most positive about their efficacy in the process and the most committed to the system. The responses of the stakeholders generally indicate that some of the preconditions for utilization of the results were met through the approach of involving stakeholders. However, it is not clear whether the opinions of the representatives will spill over to the rest of the education community.

While interest in issues shifts daily, tracking these changes is currently done by multiple surveys, which are conducted months apart. In this article, we adapt political tracking poll methods to model the functional form of issue attention changes through three phases, including equilibrium, developing interest, and decline. We find that opinions do follow discernible cycles. Further, we test and find strong support for the "boredom" explanation that reactions to external events that are brought to the attention of the public through the media are much greater during the developing interest phase than the saturation phase. The evidence gathered from the newly adapted method has implications for modeling the issue-attention cycle and testing the effects of media coverage and external events on public interest in issues.

In contrast to education reform efforts that target teachers and schools, merit-based financial aid for college increases the incentives for high school students and their families to directly affect the quality of education by investing more time and effort in schoolwork. Large-scale merit-based aid programs, such as Georgia's HOPE Scholarship, seek to improve education by encouraging students to meet higher standards, in this case by obtaining a 3.0 grade point average in high school and college. Since the HOPE program began in 1993, the number of high school graduates qualifying for the aid has steadily increased to more than 38,000 graduates in the class of 1998, or 59.5 percent of the graduating class. At the same time, the relationship between grades and achievement has remained consistent or, in some cases, improved since HOPE began. In fact, African–American males and females with a 3.1 high school core course grade point average have increased their average Scholastic Assessment Test (SAT) scores by more than 20 points. This indicates that merit-based aid has improved the quality of K–12 education in Georgia and reduced racial performance disparities by motivating students and their families to commit greater effort to schooling. © 2002 by the Association for Public Policy Analysis and Management.

Currently there is considerable interest in quality and customer service in both business and government, as total quality management philosophy and practices (Deming, 1982; Scherkenback, 1986) have become widespread in the private sector and are making substantial inroads in the public sector (Krone, 1991; Milakovich, 1992; Hyde, 1992). With respect to the public sector, however, the move toward quality comes at a low point in the popular impression of the efficacy of government and the public service. In its widely heralded report, the Volcker Commission voiced alarming concern with public perceptions of corruption, waste, and ineffectiveness in government. The commission warned that this general distrust undermines the public service and in the long run could jeopardize the democratic process itself (National Commission on the Public Service, 1989, pp. 12-13). Indeed, conventional wisdom seems to take for granted that the private sector is far superior to the public sector in terms of overall performance. Downs and Larkey (1986) begin the preface to their tract on efficiency in government by stating: "If you are like most Americans, you believe that governments are inherently inefficient.... You believe that there should be less government and that what remains should be run as if it were a private business" (p. v). Although their subsequent analysis finds that governments are more efficient and businesses less efficient than popularly believed, the book's opening premise is that public services are generally not highly regarded. It often appears that the perceived performance gap between the public and private sectors pertains to both quality and customer service as well as the efficiency of service delivery. Joseph Sensenbrenner, a former mayor of Madison, Wisconsin, says, "Increasingly, people don't believe that government knows how to help or wants to bother. They find concepts like |total quality,' |customer driven,' and |continuous improvement' foreign to everything they know about what government does and how it works" (Sensenbrenner, 1991, p. 199). He continues with an example of the perceived inferior quality of public sector service delivery: "People are making comparisons," says one quality expert. "They can call American Express on Monday and get a credit card in the mail by the end of the week, but it takes six weeks to get a lousy driver's license renewed. You might not think the motor vehicles division competes with American Express, but it does in the mind of the customer." In the preface of Reinventing Government, Osborne and Gaebler (1992) refer to a deep-rooted contempt of government, as reflected in several frequently repeated epithets that have become part of the popular culture. For example, "It's close enough for government work" suggests that expectations for accuracy and reliability, two core components of service quality, are not held as high for public sector products and services as for those produced by the private sector, while the sarcastically inflected "I'm from the government and I'm here to help" captures a supposedly widespread sentiment that public programs and those responsible for them are often lacking in responsiveness and sensitivity, two other dimensions of service quality. However, assumptions of the perceived inferiority of public services are not always sustained by more careful analysis. Charles Goodsell (1985) describes the "grand myth," deeply ingrained in American popular culture, that citizens are not satisfied with their governmental agencies and the services they perform, but in reviewing numerous surveys designed to solicit citizen ratings of public services, he finds generally favorable responses. Furthermore, client evaluations of specific services they receive tend to be more positive than ratings of these same services by the general public (Katz et al, 1975). …

How can a useful performance monitoring system be designed? Gary T. Henry and Kent C. Dickey describe the use of a research and development model to develop an effective performance monitoring system as part of the State of Virginia's efforts in educational reform. They provide details on how the approach was implemented and discuss both the limits and risks involved. A combination of public interest, political pressure, and increased technical capacity has thrust public accountability, and with it the concept of performance monitoring, once again into the fore. When the U.S. Department of Education sounded an alarm with A Nation at Risk in 1983, the nation renewed its concern for public education. The presidential Educational Summit in Charlottesville, Virginia, the advent of significant community and parental control of Chicago's public schools, and the court-ordered complete overhaul of the public school system in Kentucky amplified the alarm. Dissatisfaction with public education services in many quarters has led to a variety of proposals ranging from dismantling educational bureaucracies by allowing school choice to increasing the accountability of the schools to the public.

Ranking urban affairs graduate programs is a difficult undertaking because there is no objective or consensual measure of quality. This study utilizes three empirical measures- reputation, productivity, and quality of individual output—to rank urban affairs programs. These measures as well as an overall ranking are presented for each program. In addition, per capita productivity and quality of individual output ratings are presented to control for program faculty size; the same measures are presented with the most productive member and the most cited member of each faculty removed from the respective rankings to control for a "superstar" effect.

A central theme in much of the literature on policy formulation centers on the actors involved. Questions such as these are central: Who are they? To what extent are they involved? In what types of policy are they involved? Recently Svara has merged various strains of both the empirical and theoretical literature that deal with these questions to address the extent to which policy and administrative authority is shared between administrators and elected officials.' Svara's research indicates that the extent of involvement by the actors varies considerably but that elected officials are more involved with setting broad mission statements, while administrators are more involved in day-to-day management. Both midlevel policy making (such as the design of programs) and broad administrative or implementation decisions are likely to be shared to some extent by administrators and elected officials. Research using the four-component model of the policy-administrative continuum mission, policy, administration, and management has only dealt with relationships at the local level.2 It is useful to examine the involvement at the state level also. Svara's conceptualization led to the reanalysis of data collected for a review of state boards in Virginia.3 At the state level, appointed boards are an important group of actors. Boards, while not elected, are generally oriented to a policy-making role. Thus the patterns of their involvement with policy making are interesting to analyze to see if their involvement is primarily at the mission or agenda-setting level or if involvement is extensive at the adminstrative and management level.