Search

The second article in this series on the history of evaluation use has three sections. The first and longest develops a functional definition of the term use, noting that a thorough definition of evaluation use includes the initial stimulus (i.e., evaluation findings or process), the user, the way people use the information, the aspect of the program considered, and the purpose. It then defines evaluation use’s unethical companion, misuse, detailing the distinction between the two. The second section briefly discusses a broadened concept of evaluation impact that expands to include evaluation influence. Finally, the third section summarizes the factors that research has shown to be related to evaluation use.

This discussion comments on the approximately 20years history of writings on the prescriptive theory called Empowerment Evaluation. To do so, involves examining how "Empowerment Evaluation Theory" has been defined at various points of time (particularly 1996 and now in 2015). Defining a theory is different from judging the success of a theory. This latter topic has been addressed elsewhere by Michael Scriven, Michael Patton, and Brad Cousins. I am initially guided by the work of Robin Miller (2010) who has written on the issue of how to judge the success of a theory. In doing so, she provided potential standards for judging the adequacy of theories. My task is not judging the adequacy or success of the Empowerment Evaluation prescriptive theory in practice, but determining how well the theory is delineated. That is, to what extent do the writings qualify as a prescriptive theory.

Our third article on the history of evaluation use affirms its importance in evaluation practice and related literature. It first highlights the centrality of use in the field's professionalizing documents, extant theories, and the persistence of continuing research. Next, it discusses the challenge of evaluation theories in general, including the prevalence of prescriptive theories, and provides criteria for a good descriptive use theory that even the most detailed use theory does not currently meet. The third section reviews existing use "theories," nine from Alkin's evaluation theory tree, two from the literature, and two existing influence theories. This article concludes with a discussion of research on evaluation use past and present, including the effects of competing definitions, along with thoughts for future inquiry.

Here, we describe the development of logic models depicting three theories of evaluation practice: Practical Participatory (Cousins & Whitmore, 1998), Values-engaged (Greene, 2005a, 2005b), and Emergent Realist (Mark et al., 1998). We begin with a discussion of evaluation theory and the particular theories that were chosen for our analysis. We then outline the steps involved in constructing the models. The theoretical prescriptions and claims represented here follow a logic model template developed at the University Wisconsin-Extension (Taylor-Powell & Henert, 2008), which also closely aligns with Mark's (2008) framework for research on evaluation.

The act of valuing in an evaluation may be perceived in different ways. We consider the multiple theoretic perspectives that govern an evaluator's behavior and present a typology of evaluator valuing roles. Within this typology we describe three ways in which value judgments are typically reached-by stakeholders alone, stakeholders and evaluators in consort with each other, or by evaluators only. This heuristic offers a more explicit understanding of how valuing occurs in evaluation. © 2012 Wiley Periodicals, Inc., A Wiley Company, and the American Evaluation Association.

As is the case with other fields, there is motivation for studying the impact that the body of evaluation theory literature has within and outside the field. The authors used journal articles written by theorists included on the evaluation theory tree by Alkin and Christie (2004; Christie & Alkin, 2008) and published in the Web of Science, an online academic database, as a data source to address the questions: "What fields of knowledge do evaluation theorists draw upon in their publications?" and "What fields draw upon the published articles of evaluation theorists in their work?" The bibliometric analysis of 731 journal articles published by evaluation theorists shows that evaluation is an intellectual field that is strongly informed by psychology and education, as well as a range of other subjects. There are some consistencies in the publishing patterns of the theorists across the three branches of the theory tree (methods, use, and valuing), but multidimensional scaling maps show that each branch also exhibits a distinctive character of its own. References to a random sample of 500 articles from a subset of 9 theorists indicate that these theorists were cited not only in the areas that they themselves cite, but also in areas beyond where they routinely publish. © The Author(s) 2010.

When examining various evaluation prescriptive theories comparatively, we find it helpful to have a framework showing how they are related that highlights features that distinguish theoretical perspectives, thus a "theory" about theories. The evaluation theory tree that we presented in Alkin's recent book, Evaluation Roots [Eisner, E. (2004). The roots of connoisseurship and criticism: A personal journey. In M. Alkin (Ed.), Evaluation roots: Tracing theorists' views and influences. Thousand Oaks, CA: Sage; Guba, E., & Lincoln, Y., (2004). The roots of fourth generation evaluation: Theoretical and methodological origins. In M. Alkin (Ed.), Evaluation roots: Tracing theorists' views and influences. Thousand Oaks, CA: Sage; Stake, R. (2004). Stake and responsive evaluation. In M. Alkin (Ed.), Evaluation roots: Tracing theorists' views and influences. Thousand Oaks, CA: Sage; Wholey, J. S. (2004). Using evaluation to improve performance and support policy decision making. In M. Alkin (Ed.), Evaluation roots: Tracing theorists' views and influences. Thousand Oaks, CA: Sage], is such a framework. Just as theorists modify their views over time, in this paper we suggest modifications to the theory tree presented in the Roots book, including a repositioning of a few theorists, the addition of theorists, and a reconceptualization of the valuing branch. © 2008 Elsevier Ltd. All rights reserved.

Program theory plays a prominent role in many evaluations, not only in theory-driven evaluations. This paper presents a case study of the process of developing and refining a program's theory within a user-oriented evaluation. In user-oriented (or utilization-focused) evaluations, primary users can play a role in defining their own program theory. This is different, however, from the typical process by which a program theory is developed when using theory-driven evaluation framework. This case study concerns a university's academic outreach program with three local school districts. The program's objective is to increase the number of University of California eligible and specifically University of California, Los Angeles (UCLA) admissible students from targeted local public schools. The authors helped develop and refine outreach staff's program theory. The evaluation procedures are described and results of the theory building process presented.

This study investigated the nature of barriers to the implementation of (external) program evaluation. The purpose of the study was to expand the scarce empirical research on this topic by adding program staff perspectives. Eighteen staff members of a university outreach program that had been subjected to external evaluation were interviewed (1) about their general thoughts on barriers to evaluation implementation, and (2) about their views on the explanatory value of factors found in research related to evaluation utilization and which we believed to be theoretically relevant for barriers to evaluation implementation. These factors are summarized as human, evaluation, and context factors. Asked what they considered to be barriers to evaluation implementation, the interviewees mentioned human factors most frequently. They talked about evaluation and context factors much less frequently. With regard to human factors, interviewees focused on the evaluator's social competence and program staff's lack of trust in evaluators and evaluation process. Taking into account the context of the study, particularly the participants' prior experiences with evaluation, the findings reinforce our understanding that in order to avoid barriers to evaluation implementation, it is important to create a trusting relationship with those affected by the evaluation through continuous participation and communication, and to conduct carefully planned, methodologically appropriate evaluations.

This paper describes the use of role-play in creating an experiential learning environment for graduate students enrolled in a comparative evaluation theory and/or an evaluation procedures course. Role-play exercises have been designed for each course that increase student involvement in the learning process, which promotes a deeper understanding of and engagement in course content. The specific exercises used, which are described in detail, require students to think beyond the words they read—to learn by doing in a safe environment.

This article examines Guttman's facet theory (FT) and compares it to factor analysis (FA) in the context of two research studies. FT is examined in terms of its advantages and disadvantages compared to FA for theory development and confirmation. Two studies provide insights into the utility of FT. The first describes ideal student traits as perceived by prospective teachers. Using FT and smallest space analysis (SSA) confirmed the theory by displaying the accord between the facets in the mapping sentence and the distribution of the items in a multidimensional space. The second describes teacher professionalism and teaching context as perceived by teachers who completed a version of the Talbert and McLaughlin questionnaire. FT and SSA confirmed the theory and identified the subscales and dynamic relationships between them. By using the definitional mapping sentence of FT and SSA, the multidimensional theory and structural validity of the findings in the two studies were confirmed.

Parents, professionals, and researchers have been concerned about the most appropriate placement for children with mental retardation. To shed light on the efficacy of integration, 36 studies were reviewed on the academic and social attainments of school-age children with mental retardation. Results show that children in general education classes do not attain social acceptance ratings at as high a level as do their typically developing peers. When comparing children with mental retardation in general education and special education classrooms, integrated students perform better than their comparable segregated students on measures of academic achievement and social competence. Other variables are discussed that might affect child outcomes, and related recommendations are given for future research.

This study examined the effects of the child's diagnosis (autism vs. Down syndrome), age, and current educational placement on parental perceptions toward inclusion for their child with disabilities. Parents of children with autism and with Down syndrome completed surveys regarding their opinions on their child's current educational placement, their desire for changing the current placement, and their views on inclusive education. Results indicated that diagnosis, age, and current placement influenced parental opinion on the ideal educational placement for their child. Parents of children with Down syndrome were significantly more likely to endorse inclusion (full-time placement in general education) as the ideal educational program for their child whereas parents of children with autism were more likely to endorse mainstreaming (consistent part-time placement with general education students). Parents of younger children and parents whose children were already placed in general education programs were more positive towards inclusion than parents of older children or students currently in special education. Findings are discussed in terms of child characteristics and prevailing educational practices.

This study surveyed 291 parents of children with Down syndrome about their satisfaction with their child's current educational program, as well as their desire and reasons for considering change. Perceptions were compared across age, current educational placement, ethnicity, and mother's educational level. Parents of younger children, children currently enrolled in both early intervention and general education, and mothers with an education beyond a bachelor's degree were more satisfied with their children's current programs. Parents of children in general education were least likely to want a program change. Wanting change centered on seven themes: the approach of a school transition point, the desire for greater inclusion, placement in a neighborhood school, the need for additional services and support, influences of peers, parents' financial resources, and ongoing information acquisition/decision-making issues.

As the evaluation profession has continued to grow and develop, there has been a corresponding concern about how to properly train future evaluation practitioners. Those who teach evaluation strive to develop training opportunities that create the appropriate balance of the practical, how-to knowledge of evaluation with the burgeoning theoretical knowledge that undergirds responsible evaluation practice. There is a recognized need to move beyond traditional teaching methods to ones that are more engaging and "hands on" to help students understand the interactive nature of program evaluation. Problem-based learning is an experiential learning approach that can integrate the need to balance self-study of theory and practice, along with the need to familiarize students with the dynamic, interactive nature of program evaluation. This article serves as an introduction to the problem-based learning approach and describes this instructional method applied to teaching a graduate-level course in evaluation procedures. (Contains 1 figure and 2 notes.)

The editors give each of the theorists a chance to respond to questions posed about the context of the situation in relation to their own experience in the field, exploring how the exercise had an impact on their evaluation designs.

The theorists whose work appears in this issue have confronted the exercise presented to them in vastly different ways. The editors of this journal asked these theorists to consider how they would evaluate the Bunche-Da Vinci program, and to make and specify assumptions about the program context. The primary task, however, was to describe "a course of action, which may have consequences that lead to other evaluation design decisions." All of the theorists did indeed do this. There was great diversity in the evaluation approaches presented and in the presentation of approaches. As mentioned in "What Theorists Say They Do: A Brief Description of Theorists' Approaches" (Christie and Azzam, EJ791770), the editors anticipated diversity on both of these dimensions and were pleased to see that it emerged. This article explores the differences in approaches that are evident, as well as the similarities on several dimensions of evaluation practice. These authors comment first, however, on each theorist's overall approach to the task before moving on to the discussion of comparing approaches. (Contains 1 table.)

The Bunche-Da Vinci case described in this article presents a situation at Bunche Elementary School that four theorists were asked to address in their evaluation designs (see EJ791771, EJ719772, EJ791773, and EJ792694). The Bunche-Da Vinci Learning Partnership Academy, an elementary school located between an urban port city and a historically blue-collar suburb, is a dignified and well-maintained school structure and a much-sought-after refuge for African American and Latino students from the neighboring smaller suburb, well known for poverty, crime, racial tension, and low-performing schools. The school is a beacon of hope and stability in an industrial area of the large city and contrasts sharply with the nearby schools of the smaller, more notorious, adjacent suburb. The district recognized early on that to combat the problems that Bunche's students face, innovative solutions were needed. When approached by Da Vinci Learning Corporation, the district chose Bunche Academy to enter into a unique arrangement with the company to create a partnership that combined elements of a charter school while still remaining a regular part of the district. When the academy's decreasing state test scores contrasted sharply with the students' performance on Da Vinci's own testing system, the principal and district superintendent agreed that an outside evaluation was needed. This issue of "New Directions for Evaluation" includes four articles that examine, comparatively, the practical application of theorists' approaches to evaluation by presenting four evaluations of this case. (Contains 2 tables.)

Provides a conceptual framework for considering the distinction between evaluation use and evaluation influence and the relationship between evaluation use and knowledge use. Clarifies and expands some of the established definitions of concepts related to evaluation use. (SLD)