Search

Policymakers are fond of saying that we have enough research knowledge; however, one problem is that nobody is applying the knowledge we already have. This paper offers a discussion of the kinds of knowledge needed to improve students learning, what to do about this in the assessment and testing arena, and where assessment falls short. Types of knowledge are discussed, including research knowledge, which must be both usable and useful. Distinctions are made between usable and useful knowledge, and a case is made for how we might ultimately design our systems and our own actions to help us act with greater intelligence. (Author)

This paper presents a model for the evaluation of postsecondary online distance education courses and programs. To better address the unique nature and audience for these courses and programs, and the related institutional needs for assessing their success or failure, the focus is on a model from corporate training that provides a comprehensive, multi-level evaluation of a distance course or program. This model is based on Kirkpatrick's (1975) levels of evaluation. A general description is provided of how this model can be implemented via online means to provide the assessment information required by postsecondary institutions. (Contains 12 references.) (Author)

This paper offers a broader framework for assessment and evaluation of teachers in developing countries. Teacher is seen not only as some one who imparts knowledge and skills but in the developing world, teacher has a wider role to play. (Contains 2 tables, 3 figures, and 13 footnotes.)

In 1998, the Australian Research Council provided funding for a three year research project to develop professional standards and assessments for the English teaching profession. The project team is a consortium of researchers from three major universities together with the two national English teaching associations and representatives from state government education departments. In this paper, two of the researchers, Margaret Gill and Brenton Doecke from Monash University, and a teacher, Doug McClenaghan, reflect on the collaborative nature of the project and the central role the profession itself is playing in developing and validating nationally recognized sets of standards which will define what both the beginning and accomplished English teacher should believe, know, and be able to do. (Contains 13 notes.) (Author/EF)

This background paper details Head Start, a federally funded program serving preschool age children from low-income families, and focuses on the program's effectiveness and the adequacy of historic federal funding levels. The paper provides an overview of the Head Start Program, describes federal requirements for local programs, and describes Head Start services in Idaho. The paper also details federal Head Start appropriations, enrollment, and costs per child from fiscal year 1995 to 1999, both nationally and for Idaho. In addition, the paper lists the states providing no state funding for Head Start, those providing direct state funding for Head Start, and those with state-funded pre-kindergarten programs other than Head Start. The paper notes although it is difficult to determine the overall impact and effectiveness of Head Start nationwide because of the variation in local programs, recent research concerning local programs has generally found them to be effective in preparing children for school. However, the General Accounting Office has concluded that the existing body of studies was not sufficient to develop a national estimate of the impact of Head Start and recommended that the Department of Health and Human Services conduct a nationwide assessment of the impact of Head Start programs. The Department of Health and Human Services maintains that existing research has already provided clear evidence of the positive impacts of Head Start services. The paper concludes with a list of proposed evaluation questions for Idaho Head Start programs. (KB)

A study examined whether or not the composition faculty at the University of Northern Iowa agreed on criteria for evaluating student writing and to what extent they would give similar scores even if their criteria did not agree. Using 15 sample student papers written for the Writing Competency Examination (WCE), the scores given by current writing faculty using a 4-point holistic scale in addition to one 3x5 index card's worth of comments were analyzed. Results indicated that scores reflected a strong tendency toward the middle range. Statistical analysis on the subgroups revealed that of several factors the most significant was the relationship of scores to whether or not the rater made more content or form comments, followed by percentage of positive comments and teaching classification. Findings suggest that since significant differences were not found among the subgroups, standards are not sacrificed in spite of the current practice which reflects a more diverse and possibly less experienced group of teachers. The study suggests that: (1) the current system produces scores which tend toward the average with few highs and fewer lows; (2) no standardization of criteria exists among teachers of the College Reading and Writing (CRW) course; and (3) the university is served by promoting academic freedom for all teachers of first-year composition. (Contains 7 tables of data and 28 references; various sample forms--a recruitment letter for the study, a demographic information questionnaire, and two additional tables of data--are appended.) (CR)

This paper addresses the methodological issues encountered when attempting to evaluate a college course's effectiveness by determining whether students who enroll in it perform at higher levels than students who did not enroll. The paper identifies resource people who may be helpful in the evaluation, and offers guidelines for the research question, which defines the purpose of the project, and the null or research hypothesis. The necessity of operational definitions is also emphasized, with examples and definitions of key words. Suggested factors to consider in planning the research project include cost, ethics, time, choice of students for the study, availability of students, commitment, faculty, the experimenter effect, student retention during the study, outcome measures and when to examine them, treatment rigor, and necessary sample size. The four research models provided for evaluation of the college success course are: (1) the experiment with randomly assigned partially matched students; (2) the experiment with randomly assigned unmatched students; (3) the post-facto design; and (4) the time series design. Selection of a particular model depends on the collective purpose of the evaluation. The paper concludes with a suggested outline for the ensuing research report.(YKH)

This volume brings together the reports and discussion connected with a roundtable convened by the Departments of Education and Labor and the National School-to-Work Office to discuss issues surrounding the conduct of a net impact evaluation of school-to-work. After an introduction that summarizes the approach and intentions of the School-to-Work Opportunities Act and describes evaluation activities that have been commissioned separately from a net impact effort, six commissioned papers are presented. The papers are as follows: "Evaluating Early Program Experiences in the School-to-Work Opportunities Act: Policy and Design Issues" (Gary Burtless); "Net Impact of School-to-Work: Exploring Alternatives" (Charles Dayton); "Net Impact Evaluation of School-to-Work: Desirable but Feasible?" (Robert Glover, Christopher T. King); "Evaluation of School-to-Work Transitional Programs" (James J. Heckman); "Evaluating the School-to-Work Opportunities Act of 1994" (Robert A. Moffitt); and "Net Impact Evaluation of School-to-Work: Contending Expectations" (Hillard Pouncy, Robinson Hollister). Following the papers is a summary of the roundtable discussion and an epilogue that synthesizes both the papers and the discussion. Contains a list of participants, 190 references, and some papers includes tables/figures. (MO)

Many students view statistics as their worst college course. Four heuristics that can improve students' proficiency in statistics and in interpreting reports of research are presented in this paper. The heuristics guide students' judgments about significance, generalizability, cause-and-effect, and strength of independent-dependent variable relationships. Previous studies of students' abilities to interpret research indicated that few of them understood random sampling and random assignment research methodologies as determinants of generalizability and cause-and-effect conclusions. It is supposed that students may fail to interpret research correctly because of the exclusive attention given to factual knowledge and statistical procedures rather than to interpretation abilities. It is recommended that a large set of vignettes (research-report summaries) can provide the critical methods needed for assessing student interpretations. One way to improve students' interpretation-of-research abilities is to develop a taxonomy that guides students in their judgments. A set of taxonomies is provided that can further aid students in answering questions about vignettes. A sample interpretation-of-research vignette is presented in which seven questions are posed to students when interpreting any research report. The students' answers to the questions can identify specific interpretation problems. (RJM)

This paper chronicles the Teaching Mathematics Using Interactive Videodisk (TMMUIV) pilot project. The project emerged in response to the need to train pre-service teachers in, among other things, the use of mathematics manipulatives. The project now can offer a video database and pilot instructional program that combines full-motion video, digitized audio, text, and graphics. Two instructional modes, classroom presentation and stand alone tutorial, help the teacher develop a repertoire of techniques. The report catalogs all the system's components and provides a brief history of the genesis of the project and subsequent improvements. Formative evaluation was conducted at approrpiate times throughout the course of the project and included informal review by project staff, review by outside experts and the advisory board, and field trails conducted over a 2-year period in mathematics methods classes. Summative research was conducted when the system was close to completion. Four elementary mathematics methods classes were used to test the utility of the TMMUIV classroom presentation system. Two classes received only conventional instruction and two received the same instruction enhanced by TMMUIV. Overall, the group receiving the pilot TMMUIV enhancement reported being better prepared to teach and more highly motivated to teach. Also, they reported gaining more knowledge for teaching than the group receiving conventional instruction alone. The TMMUIV group also scored higher on each of three objective performance measures. These findings suggest the potential impact on teacher training for pre-service and in-service teacher utilization of interactive multimedia. (BEW)

A recent conference on statistics education recommended that more emphasis be placed on the interpretation of research (IOR). Ways for developing and assessing IOR and providing a systematic framework for creating and selecting instructional materials for the independent assessment of specific IOR concepts are the focus of this paper. The recommended assessment procedure to evaluate IOR abilities consists both of vignettes (research-report summaries) and questions designed to assess the students' interpretations of those vignettes. Vignettes could be selected to systematically vary on different features, such as random sampling as opposed to using an available group of subjects. A systematic framework is introduced here, consisting of 4 features, that can be used to write sets of vignettes for each cell of a 16-cell taxonomy. The features are: (1) random assignment versus classificatory independent variable; (2) a dependent variable that is either life-experience meaningful or not; (3) results that are counter to popular beliefs or for which there is no clear expectation of outcome; and (4) the independent variable having levels that are quantitatively different versus the independent variable not having an underlying continuum. A booklet containing a sample vignette for each cell of the taxonomy is appended. (RJM)

This study of 20 preservice teachers in a secondary English methods course explored whether materials on diversity, information about language of wider communication, and literature from parallel cultures would have a positive effect on their ability to fairly assess work by high school students from diverse populations. At the beginning of the semester, the students scored and discussed two sets of essays written by tenth- and eleventh-grade students. These essays had already been scored by experienced teachers; these scores and the authors' ethnic identities were hidden from the education students. During the rest of the semester students studied pedagogy, assessment, and issues of diversity including differences in prose development from culturally diverse writers. They also read secondary level literature by nonmainstream writers. At the end of the semester students evaluated the essays again. A comparison between student ratings in the first and second round found that: (1) all but one preservice teachers' scores were lower than those of the expert raters; (2) in all cases but one, preservice teachers' first scores either remained constant or were raised substantially by the end of the semester; and (3) essays which received large jumps in scores were written by students whose ethnicities were very apparent in their work. The paper includes scoring guide, table of the experts' and preservice teachers' scores, essay assignment, and a student essay, "The 'Addquin' Streets of the 'Barrio'." (JB)

Recent calls for high standards and hard content for all students have brought a growing interest in Opportunity-To-Learn (OTL) measures. OTL indicators provide valuable information for anyone interested in knowing what instruction is like, whether it is equitably distributed, and whether it is improving over time. This paper describes an effort to develop OTL indicators for science education through a collaborative project of 14 states. The Council of Chief State School Officers is working with the State Collaborative on Assessment and Student Standards (SCASS) Science Project to develop innovative approaches to assessment in elementary and secondary science education. A project component of high interest and potential application for local educators is information on students' opportunity to learn the science, content, knowledge, and skills that are assessed. The SCASS member states anticipate using methods of collecting and reporting on opportunity to learn the science content in conjunction with science assessments. The paper discusses the approach to OTL data used in SCASS and some of the issues in designing OTL data collection instruments. The questionnaire is appended. (LZ)

Technology is not often a factor in classroom activities because teachers are untrained in its use. In 1990, International Business Machines (IBM) contributed over 30 million dollars' worth of workstations, networking hardware and software, IBM courseware, cash, and training to prompt 144 selected teacher preparation programs to integrate technology into their curricula. This paper describes a study evaluating the impact of these contributions, based on data and comments received from the sites in the fall of 1992 (response rate 92%). The study discovered that 63% of teacher preparation faculty were trained in the use of IBM equipment, 44% of them because of the grant program, and that the numbers of pre-service and in-service teachers so trained increased in three successive academic years. (Contains 6 figures and 15 references.) (Author/BEW)

The suggestions offered in this paper are based on the experiences of two scientific researchers of how evaluations undertaken in a competitive arena, in which true experimental designs are not viable, can be designed in such a way that meaningful comparative data can be examined. Case studies of the Science and Engineering Research Semester and the Laboratory Graduate Research Participation programs at the Oak Ridge Institute for Science and Education illustrate how one can make use of existing comparison groups when the establishment of control groups for each cohort in an educational study is not economically feasible. Data from national surveys are used for comparative purposes. These surveys include: (1) the Survey of Earned Doctorates of the National Science Foundation (NSF); (2) the Survey of Doctorate Recipients, another NSF survey; (3) the NSF New Entrants Survey; (4) the Survey of Experienced Scientists and Engineers, also an NSF component survey; and (5) the National Survey of College Graduates. Items from these surveys that are clearly relevant are selected for comparisons. In conclusion it is remarked that this approach is not a panacea, and requires thorough understanding and careful evaluation of the national survey, but it can be a useful and cost-effective alternative to traditional control-group designs. Ten figures illustrate survey use. (Contains nine references.) (SLD)

The objectives of this program evaluation were to gather information and normative data about the students referred for psychoeducational evaluation; to determine the diagnostic profile of students referred for special education; and to use the information gathered to determine future program direction and possible program modifications. Students (N=55) in non-public schools referred for psychoeducational evaluation because of learning problems were evaluated. Students were evaluated using the Wechsler Intelligence Scale for Children III and the Wide Range Achievement Test-Revised (WRAT-R). After the evaluation these students were assigned to one of three dispositions which determined an intervention of increasing intensity in behalf of the student. The first level was teacher/psychologist consultation. The second level was multidisciplinary team meeting including the parent. The third level was a multidisciplinary team meeting with referral for an individualized educational planning meeting. Results indicated that there were no significant differences in psychologists' case dispositions using intelligence scores as dependent variables. As achievement scores in reading and spelling on the academic screening measure, the WRAT-R, yielded the only significant differences among case dispositions, a classroom based assessment of academic skills would be a viable alternative to the varied assessment batteries currently being administered. (ABL)

This paper questions whether it is appropriate to evaluate music educators exclusively on general competencies, or whether effective music teaching involves certain teaching behaviors, characteristics, and attributes that are so significantly unique that they demand their own set of evaluative criteria. The paper reviews selected relevant research from 1989 to 1993 and discusses pertinent practical issues involved in the evaluation of public school music teachers, and synthesizes eight general areas for further development as potential criteria for music teacher evaluation. Both empirical and nonempirical evidences are studied, and attempts are made to reconcile the discontinuity that exists between evaluative criteria appropriate for all teachers and that which is appropriate for teachers of music. An appropriate evaluation instrument for in-service music teachers should include the following broad areas that would serve as defining attributes: (1) personal characteristics, as evidenced through student teacher, teacher faculty, and teacher administration interactions; (2) musical competence and performance skill, as evidenced when teaching a lesson; (3) effective use of nonverbal strategies, such as modeling and demonstrating; (4) effective use of verbal strategies, such as eliciting performance from students; (5) classroom management, including discipline, group participation, and the creation of a positive learning environment; (6) effective planning for concept learning and aesthetic appreciation across a wide age span; (7) an objective assessment of teaching style based on empirically supported criteria; and (8) relevant and appropriate professional development activities. Contains 10 references. (DK)

Recent attention has focused on the decreasing levels of scholastic achievement of youth in the United States, particularly in areas of mathematics and science. In particular, concern has been expressed about the involvement and achievement of girls in traditionally sex stereotyped curricula, such as mathematics and science. This paper reports on an investigation exploring the course-taking and achievement patterns of academically talented girls and boys, selected by Scholastic Aptitude Test (SAT) scores. The subjects in this study were enrolled in the Talent Identification Program's Summer Residential Program at Duke University, an intensive 3-week academic experience. The study found that girls and boys performed equally well in all types of classes. Main effects in achievement and motivation were found only for type of class. The success of the students indicated that the SAT is a valid selection instrument. In addition, the usefulness of the SAT as an identifier of very high academic ability is described. (DK)

This paper describes a needs assessment process that identifies a range of performance improvement strategies using the steps of the Evaluation Research Model (Geroy and Wright, 1988). A detailed description of a project conducted by the Professional Development Committee of a mid-sized community college in suburban Philadelphia to identify professional development activities for the administrators, faculty, and support personnel of the college is used to illustrate this process. All of the challenges identified by outside consultants were addressed: (1) all employees (stakeholders) were involved equally in identifying professional development needs; (2) the planning committee recognized that certain non-training actions were required to maximize the worth of training; (3) data were elicited from all 280 employees that represented both their wants and their needs; and (4) decision makers were provided with a systematic process for selecting professional development activities that will yield maximum results for cost and effort. The project illustrated the number and kind of responses generated through the nominal group technique (NGT), as well as the range of professional development options that emerge from NGT responses. It reflected the cutting edge of performance technology by virtue of its application in an academic setting, involving all employees of the organization, addressing the desire of the organization for training while simultaneously presenting alternative performance improvement solutions, and introducing performance technology as the decision making model for all HRD (human resource development) functions of the organization. The project also provided techniques, references, and examples that can be used in convincing decision makers of the value of conducting a needs assessment. (5 references) (BBM)

Participatory evaluation is an assessment approach that involves the participants of a study from beginning to end. The evaluation strategy was implemented in a public school district to assess teacher involvement in decisionmaking, determine whether teachers desire more participation, and evaluate the assessment strategy itself. Data were collected through interviews with 12 teachers and 1 principal as well as through results from a survey instrument developed by the interviewees and distributed to their district's entire elementary teacher population. Results from the data analysis indicate that teachers participate in decisions involving curriculum/instruction and student personnel but do not desire more participation in the two areas. However, teachers do want more participation in decisions regarding staff personnel, supervision, budget/finance, and school facilities. Observations made while implementing the evaluation strategy indicate that the participants became actively immersed in the project, explorations of issues were facilitated due to the trusting relationship between participants, discussions of topics were easier because of the participants' broad knowledge base, and representatives of the teacher population felt that the survey findings were accurate. Recommendations for implementing the participatory evaluation strategy include making certain that all participants are knowledgeable insiders. Three tables and a figure are included. (41 references) (EJS)