Search

This paper offers a look into the future of educational program auditing and presents a detailed discussion of the notion of a districtwide educational audit. To date, virtually all program audit activities have occurred as a result of specially funded State or federal education programs. Once the program ends, or changes its requirements, the audit is eliminated. The author argues that the only way in which educational program auditing can have a long-range impact on improved decisionmaking and accountability is to have the audit conducted at the school district level as a regular and continuing activity. (Author/DN)

Suggestions for broadening the utilization of evaluation research findings are discussed. Major program changes, based on evaluation, may not become obvious for several years; program actions may contradict findings and still be based on rational decision making processes; evaluation findings may have far-reaching effects which were not recognized and formally accounted for; personal interaction between evaluator and user may help to shape and redirect the evaluation process. In order to take these factors into account, naturalistic research methods, using techniques such as case studies, field investigations, and participant observations, are recommended. The evaluator's approach is one variable of naturalistic research. The evaluator's influence on the utilization of evaluation information may be determined by: role choice; the extent of which user involvement is encouraged; the amount of attention given to the performance on mandated evaluation tasks; rapport established between the evaluator and users; and the extent to which the evaluator facilitated and stimulated the use of information. Evaluation is defined as a dynamic process; the procedures and outcomes of evaluation studies are influenced by multiple factors as well as interactions between them. Five case studies of evaluation utilization are briefly described. (MH)

This speech concerns the dimensions of context-oriented evaluation and begins with a discussion of the differences between measurement, research, statistical methodology, and evaluation. Decision-oriented evaluation is described as a process in which the evaluator has, as a major concern, the responsibility to determine likely decisions prior to data collection or analysis. The assumption that the link between evaluation and decision-making is direct and immediate, and that this approach is appropriate for all situations, is seen as a drawback which limits the decision-oriented evaluation approach. On describing the context-oriented evaluation approach, the author provides a three dimensional method for classifying the evaluation context. The first dimension concerns evaluation intent and entails process, results with a predisposition, and openminded results. The second dimension of evaluation context involves whether evaluations are commissioned for individual or organizational purposes. Several organizational and personal motivation factors in this dimension are discussed. The third dimension, the evaluation structure, entails the extent to which those commissioning the evaluation have a specific set of expectations as to the areas to be investigated, the goals to be examined, and the role of the evaluator. The three dimensions comprise a method for providing an evaluation consistent with the contextual needs of a given situation. (Authro/JAC)

A new role for evaluators is proposed, which reconstructs the evaluator into an information specialist who is part of a project team engaged in self evaluation. The evaluator's role has traditionally been complicated by disagreements about what is expected from the evaluator; by differing perceptions about the evaluator's power to effect change; by the influence of personal values and methodological preferences; and by the conflict between internal reports of findings (to be used for program improvement), and external reports (to be used to make judgments regarding program accountability). The proposed role of the information specialist includes five major functions: (1) program clarification--asking questions about the evaluation plans and criteria for assessment; (2) information structuring--developing forms and tests; (3) information acquisition--collecting data; (4) information provision--analyzing data and presenting results; and (5) technical assistance--giving advice in his or her areas of expertise, conducting pilot tests, or searching the literature. Self evaluation (by the project staff) is emphasized as a means of increasing the likelihood of program improvement. Suggestions regarding the accountability role of external audits are included. (GDC)

The general goals of the study were: (1) to obtain the decision maker's perspectives on evaluation, how he defines it, how he carries it out; (2) to better understand the role of evaluation in site level decision making; and (3) to identify important characteristics of evaluation and of the school setting that affect the level of evaluation utilization at the school site. The methodology of the Decision Maker Study is detailed, describing the selection of the respondent sample, the elaboration of the interview strategy and format, the training of the interviewers, the collection of the interview data, and the several analysis activities--some complete, others in progress. Multiple strategies for data analysis included directed discussions, selected reviews of written data summaries, human data banking and an independent "quotation analysis." This many-faceted approach gave added validity to the themes and patterns of interactions uncovered in the study. (Author/RL)

Recommendations to increase the relevance of information, improve the communication of information, and increase user responsiveness to information are presented. Recommendations include the following: (1) the Department of Education (ED) should institute a flexible planning system for evaluations of federal education programs; (2) congressional requests for evaluation should identify the kind of question(s) to be addressed; (3) in evaluation initiated by the ED, the kinds of evaluation activities to be carried out should be specified clearly and should be justified in terms of program development or program implementation; (4) the ED should ensure that dissemination of evaluation results achieves adequate coverage; (5) the ED should observe the rights of any parties at interest and the public in general to information generated about public programs; (6) the ED should test various mechanisms for providing linkage between evaluators and potential users; and (7) the ED should provide funds for training programs in evaluation to increase the skills of individuals currently charged with carrying out or using evaluations and to increase the participation of minorities. While no formal recommendation was made to increase user responsiveness to evaluation, it was suggested that the ED consider a plan requiring evaluation use reports. (RL)

This investigation documents the impact of the Van Leer Study, a large-scale evaluation study of achievement in the primary schools of Israel. It is intended to increase understanding of the process of evaluation utilization, showing how evaluation findings and other kinds of information can work together, over time and in a variety of ways, to influence decision making. The Van Leer Study provided a comprehensive review of three major areas: resources and their allocation among the different types of Israeli public schools; student achievement in reading, arithmetic, Biblical studies, geography, and science; and home and school variables associated with achievement. The study was designed for multiple clienteles and was reported in a variety of formats. When released, the evaluation report generated considerable controversy among its major audiences. The ultimate impacts of the lengthy report were that the various groups were led to a closer examination of the issues; that a number of Parliamentary recommendations were made; that a variety of program changes were initiated at the Ministry of Education; that additional research in the form of replication studies and supplementary analyses resulted; and that further study is currently ongoing. (PN)

The misuse of evaluation is discussed. Boundaries of misutilization are not well understood, and distinctions between misutilization and non-use or abuse are not clear. In general, the responsibility for evaluation misuse is with the user, while the responsibility for misevaluation is with the evaluator. A distinction is also needed between intentional and unintentional misuse. Misuse of evaluation may be further defined as misuse of commissioning the evaluation, misuse of the evaluation process, and misuse of the findings. A more precise definition will refine the nature of misutilization in order to estimate how often it occurs. (SLD)

How can Title I evaluations be improved so that they are more likely to increase the impact of evaluation data on informed actions within the schools? An examination of evaluation practice leads to the conclusion that most LEA practitioners view Title I evaluation as externally imposed and primarily designed to serve state and federal needs. A broader conception of evaluation utilization should focus on the actions and involvements of individual evaluators interacting with designated users. Based on this conception, technical assistance efforts should focus on evaluation as a process. The attainment of greater utilization of evaluation is dependent upon changes in the Title I evaluation which would allow increased local determination of evaluation questions of concern. (Author/CTM)