Search

The aim of this paper is to present a logical structure for the evaluation of evaluation (meta-evaluation) and to suggest ways of conducting such evaluations. Part I contains an analysis of background factors and problems associated with meta-evaluation--that is, the evaluation of evaluation. This part discusses the need for meta-evaluation and summarizes some of the pertinent literature. Suggestions are made concerning what criteria should guide the development of a meta-evaluation methodology. The final and major portion of Part I is an enumeration of 6 classes of problems that jeopardize methodology. The second part of the paper is a conceptual response to the first part. Part II contains a definition of meta-evaluation and a set of premises to undergird a conceptualization of meta-evaluation. Most of part two is devoted to a logical structure for designing meta-evaluation studies. The third part of the paper is an application of the logical structure presented in Part II. Basically Part III contains 5 meta-evaluation designs. Four of the designs are for use in guiding evaluation work, and the fifth is used in judging completed evaluation work. Taken together the three parts of the paper are intended to provide a partial response to the needs for conceptual and practical developments of meta-evaluation. (Author/MLP)

The CIPP Evaluation Model was originally developed to provide timely information in a systematic way for decision making, which is a proactive application of evaluation. This article examines whether the CIPP model also serves the retroactive purpose of providing information for accountability. Specifically, can the CIPP Model adequately assist educators, after the fact, to account for their decisions and actions? If so, the CIPP Model is a powerful tool for making and implementing decisions and for post hoc accounting for those decisions and actions. The remainder of the article describes the CIPP Model, presents a conceptualization of accountability, and analyzes the potential of the CIPP Model for meeting the information requirements of accountability. (Author)

This is a proposal submitted by the Ohio State University Research Foundation for design of a planning and assessment system for the Division of Manpower and Institutions (now Division of Research and Development Resources), U.S. Office of Education. It responds to a need for an expanded planning, management, and assessment system that would enable DMI leaders to make responsible decisions under its new institutional support policy. The proposal suggests a background study; the convening of two advocate teams to develop alternative evaluation strategies; a convergence team to review the competing strategies and generate a third, which could be an elaboration of one of the advocate team strategies, a merging of the best features of the two, or a completely new strategy; and a means for judging the competing strategies. (Author)

This study focused on the relationships between attitude congruence and the selector's decisions about teacher candidates. Selectors rated a teacher applicant as part of a simulated teacher selection situation. Applicants were represented by written information in which the proportion of attitude congruence with the selector and the relevance of the attitudes to the vacancy were varied. The data indicated that the selector's decisions about teacher candidates are influenced by the proportion of congruent attitudes rather than by the relevance of the attitudes to the position vacancy. The implications of the findings for teacher preparation programs and teacher recruitment are discussed. (Author)

This report submitted by Advocate Team No. 1 to the U.S. Office of Education, Division of Research and Development Resources (formerly Division of Manpower and Institutions) presents a proposed evaluation system for R&D institutions and programs. It includes: (1) an organizational framework within which the evaluation system could function; (2) procedures to be followed in implementing the evaluation system; and (3) a set of criteria and related guidelines for use in applying the proposed evaluation approach. (Author/MS)

Seventy-two Ss attempted to solve four "who-done-it" type deductive reasoning problems which varied in mode of representation, type of logical connective employed, and affirmative versus negative statement of information. Affirmative and conjunctive problems were solved more frequently than negative and disjunctive problems (all are defined in the text of the articles); however, the mode of representation in which the problem was presented had no effect. Analysis of the mode of representation actually employed (based on Ss' written protocols) revealed a marked tendency for Ss in various conditions to change the original representation of the problem. When these changes were taken cognisance of, the matrix representation was superior to other forms on all but negatively stated problems. The somewhat poorer performance of females on this task was related to their greater reluctance at changing the representation of the problems. The original hypothesis of the instrumental role of representation of information in problem-solving was supported. Suggestions are offered for subsequent investigation. (Author/KS)

This paper presents a guide for the planning and evaluation of student teachers' performance based on performance objectives. According to the author performance objectives are written sets of behaviors that the student may perform in order to demonstrate achievement. A minimal pattern of performance objectives should be mandated for all student teachers. Research suggests a hierarchy of ten levels of role behavior engaged in by student teachers. The students progress from inner directed to outer directed activities. Students vary in their degree of proficiency at the onset of student teaching thus prohibiting structured time periods. Performance objectives and the ten classifications of behavior enable the student to analyze the whole role of the teacher and decide where his university classroom experiences fit. These objectives help to orient the student teacher to the role of teachers, enable him to interpret the behavior he observes, and help him to act in response to his observations. Performance objectives also serve as a planning guide for supervising teachers enabling them to help the student see the total picture of the teacher's role. Student teachers, supervising teachers, and university supervisors may use performance objectives to improve communication, plan activities, perceive behavior, and assess performance. Ten checklists of performance objectives and an appendix with a weekly behavioral objective report are included. (MJM)

This report describes the results of an experiment assessing the appropriateness of Delphi procedures for formulating group value judgments. Two groups of subjects--upperclass and graduate students from UCLA--were asked to generate and rate value categories relating to higher education and the quality of life. The initial lists (300 and 250 items respectively) were aggregated by the experimental teams to 45 and 48 items respectively. The subjects then rated all possible pairs of these items with respect to their similarity. The average similarity ratings were analyzed by a clustering routine. Subjects from both groups made estimates of the relative contributions of each of the educational categories to each of the quality of life categories. The primary data analyses concerned the importance rating. Three aspects were examined: the quality of the distributions of the responses, the correlation between ratings by different groups and different rating techniques, and the amount of change and degree of convergence upon iteration with feedback. The experiment furnished support for the conclusion that Delphi procedures are appropriate for processing value material as well as factual material. The substantive data concerning higher education and individual life also indicated the possibility that current notions of the role of the university are somewhat loosely tied to the basic interests of the students. A 13-item bibliography is included. (Author/MJM)

The decade of the sixties saw the appearance of more formal curriculum development efforts in industrial arts than had occurred in its total prior history as a school subject. This paper seeks to explore, analyze, and compare these curriculum development efforts and to suggest priorities for future development, dissemination, and research activities. The programs discussed are grouped according to those based upon either industry or technology, those centered around the individual projects with a career development or occupational emphasis, and those characterized by an evolutionary approach. Although the literature reveals a widespread concern and involvement in an effort to improve the industrial arts program, there is still no instance where an innovative program has been implemented across all grade levels. Nor has there been much coordination with programs in other school subjects. Materials presently available vary from philosophical guidelines for adaptation by the teacher, to highly specific text materials and laboratory manuals, available commercially. Problem areas of research, development, and dissemination are identified by the author. (GEB)

This guide to accompany "Similarity and Congruence" contains all of the student information in SE 015 346 plus supplemental teacher materials. A summary of terminal objectives and teaching aids and equipment is given. With each section are listings of objectives, teaching aids, suggested approaches, and discussion questions. Related documents are SE 015 334 - SE 015 346. This work was prepared under an ESEA Title III contract. (LS)

This instructional unit is an introduction to the common properties of similarity and congruence. Manipulation of objects leads to a recognition of these properties. The ASA, SAS, and SSS theorems are not mentioned. Limited use is made in the application of the properties of size and shape preserved by similarity or congruence. A teacher's guide is available. Related documents are SE 015 334 - SE 015 345 and SE 015 347. This work was prepared under an ESEA Title III contract. (LS)

Following a brief investigation of the need for performance objectives specifying teacher behavior, this report develops a performance base for field experience. The purpose of performance objectives in field experiences and the relationship of these objectives to the student teacher, cooperating teacher, and university supervisor are discussed in detail. The development of performance objectives for field program is based on the following considerations: identification of clientele (groups) providing performance statements about expected teacher behavior; a survey of clientele to secure goal and performance statements; preparation of performance statements based on evaluation behavior, instructional behavior, planning and organizing classroom behavior, classroom management behavior, and guidance and couseling behavior; analysis and revision; field testing the performance statements; and legitimization of performance objectives. Finally, possible problems in developing performance objectives include intrusion resistance, apathy and a complexity maze. A four-item bibliography is included. (MJM)

An evaluative symposium conducted during the 1971 Annual Meeting of the AERA offered the following critiques on "Educational Evaluation and Decision Making," a book prepared by the Phi Delta Kappa Study Committee on Evaluation: "A Critique of the Report of the Phi Delta Kappa Study Committee on Evaluation" (Henry M. Brickell); "A Critique of the Measurement and Instrumentation Aspects of 'Educational Evaluation and Decision-Making'" (John C. Flanagan); "A Critique of the Methodology of Evaluation in the PDK Volume, 'Educational Evaluation and Decision Making'" (William B. Michael); "Evaluation: Noble Profession and Pedestrian Practice" (Michael Scriven); "Determining 'Most Probable' Causes: A Call for Re-examining Evaluation Methodology" (James L. Wardrop). (MS)

Part 1 of this monograph discusses the status of educational evaluation and describes several problems in carrying out such evaluation: (1) defining the educational setting, (2) defining decision types, (3) designing educational evaluation, (4) designing evaluation systems, and (5) defining criteria for judging evaluation. Part 2 proposes an approach to educational evaluation which would alleviate problems of definition and design. Different types of decision settings and appropriate evaluation strategies are explored. Types of decisions are matched with types of evaluation. The structure of evaluation design including the collection, organization, analysis, and reporting of information is outlined. Finally, criteria for judging evaluations such as validity, reliability, significance, scope, and efficiency are presented. Figures are given, including the CIPP (context, input, process, and product) Evaluation Model. (AL)

The need for competent formal evaluation programs, particularly for new federally assisted programs, is expressed. Problems in defining educational evaluation and its requirements, in designing such evaluations, and 3 possible sources of faulty conceptual bases for evaluations, are presented. An attempt is made to define evaluation in general, to analyze emergent problems of educational change, and to identify the types of decisions for which evaluations are needed in these programs. Four strategies for evaluating educational programs are outlined. These include context, input, process, and product evaluation, each of which are used at a distinct stage in the development of a program. Finally, a general guide for developing evaluation designs to implement a given evaluation program is provided. The logical structure of evaluation design is presented in a step by step format. (PR)

A 60-item questionnaire measuring believed effects of sensitivity training, evaluations of these effects, and logical inconsistencies of respondents is presented. The questionnaire was developed from phrases describing sensitivity training found in newspaper articles written during a week of intense publicity following an incident at a local high school. (Author)

Computers are beginning to be used more frequently as instructional tools in secondary school social studies. This is especially true of "new social studies" programs; i.e., programs which subordinate mere mastery of factual content to the recognition of and ability to deal with the social imperatives of the future. Computer-assisted instruction (CAI) in such programs is educationally justified because the computer provides great information storage and retrieval capacities. In addition, CAI is economically feasible for secondary school social studies because of the large enrollments in those programs. Examples of simulations which have been used include MARKET, ELECT, and USPOP; these introduce the student, respectively, to dynamic marketing, political and demographic analyses. (PB)

The report of the Advisor to American Indian students covers 3 quarters of the academic year 1969-70 at Montana State University (MSU). During that period, the advisor worked on a 3/4 time basis to give help and guidance to American Indian students trying to finish a 4-year degree program. In preparation for this, the advisor visited 6 Montana reservations in August and September 1969 to meet with students who would attend MSU that fall. The most prevalent student problems were finances, low grades, dropping courses, and absenteeism. The document gave, for each quarter, the number of Indian students enrolled, counseling and tutorial services given, student activities, and end of quarter grades. (KM)

This document provides an analysis and synthesis of the research in industrial arts education during the period 1966 through 1968, inclusive. Journal articles and doctoral dissertations were the primary sources for studies, while staff studies and reports of funded research were used when available to the reviewers. Major topics are: (1) History, Philosophy, and Objectives, (2) Curriculum Development, (3) Instructional Media and Methods, (4) Student Personnel Services, (5) Facilities and Equipment, (6) Teacher Education, (7) Administration and Supervision, (8) Research, and (9) Summary. An extensive bibliography of the reviewed literature is included. This report complements a 1968 conference on research in industrial arts, a report of which is available as ED 029 986. (GR)

A study explored the relationships between 1) congruence of attitudes between principals and teacher candidates with different job qualifications, and 2) principals' attraction to teacher candidates. Concern was to test relationships expressed in Newcomb's ABX system of interpersonal attraction. In a simulated selection situation, elementary principals (N=140) rated an applicant for a teaching position. Principals were volunteers from a random sample of 500 drawn from all full-time elementary principals in New York State excluding New York City and Long Island. Instruments were the Education Scale (Kerlinger and Kaya, 1959) to secure principals' attitudes toward general educational practices and the Interpersonal Judgment Scale (Byrne, 1961) to measure their attraction to the hypothetical candidates presented to them. Teacher applicants were represented by written information in which the attitude congruence with the principal and the level of teacher qualifications were varied. The results of data analysis (two-way analysis of variance) suggest that 1) the similarity-dissimilarity of attitudes between principal and teacher candidates does affect principal attraction to the candidate with principals more attracted to those holding attitudes similar to their own and 2) effect of similarity-dissimilarity of attitudes seems to be more powerful than the effect of teacher candidate qualifications. (Author/JS)

Experimental design has only limited utility inthe field of educational evaluation and, in general, the methodology of the former does not equal the methodology of the latter; in fact, experimental design plays a limited role in the total framework of educational evaluation. Experimental design does have potential utility in the areas of input and product evaluation; however, it has little utility within the areas of context and process evaluation. The utility of experimental design can be increased by following a set of procedures that do not require the use of a common criterion instrument and a uniform decision rule for all students in the experiment. This will allow an investigator to judge a program in terms of the number of students for whom it was successful. A schema for the development of product evaluation designs is included. (Author/CK)

Evaluation is discusses as a primary element in the change process. Defined as a procedure for obtaining data to make decisions, evaluation is regarded as a recurring determination of the extent to which objectives have been reached, within a four-stage sequence of research, development, diffusion, and adoption. A conceptual scheme for viewing the role of evaluation is explained and distinctions are made between evaluation procedures applicable to decision making on Federal and State administrative levels and procedures applicable to decision making on the local level. Six recommendations are made for the implementation of evaluation of projects developed under the ESEA Act of 1965. (JK)

PAPERS OR STUDIES CONCERNED WITH THE APPLICATION OF BEHAVIOR MODIFICATION PRINCIPLES TO CHILDREN AND RETARDATES ARE INCLUDED IN THIS ANNOTATED BIBLIOGRAPHY. AUTHORS ARE LISTED ALPHABETICALLY. A SHORT PARAGRAPH DESCRIBES THE GENERAL NATURE OF EACH STUDY CITED, AND BIBLIOGRAPHIC REFERENCES ARE GIVEN. THERE ARE 130 ENTRIES IN THIS BIBLIOGRAPHY. (MS)

INFORMATION RELATED TO INNOVATIVE ELEMENTARY AND SECONDARY SCHOOL PROGRAMS IN OHIO IS PRESENTED. DATA WERE OBTAINED FROM SCHOOL ADMINISTRATORS AND TEACHERS THROUGH THE USE OF QUESTIONNAIRES. ENTRIES INCLUDE--(1) GEOGRAPHIC REGION, (2) COUNTY, (3) CITY OR VILLAGE, (4) LOCAL DISTRICT, AND (5) PROJECT TITLE. DETAILED DESCRIPTIONS OF SELECTED PROGRAMS ARE INCLUDED. A FUTURE REPORT WILL CONTAIN AN ANALYSIS OF THE TYPES, ORIGINS, FUNDING, AND EFFECTIVENESS OF CURRENT INNOVATIONS. THIS PUBLICATION IS ALSO AVAILABLE FROM PUBLICATIONS OFFICE, THE OHIO STATE UNIVERSITY, 242 W. 18TH AVENUE, COLUMBUS, OHIO 43210, FOR $2.25. (AG)