Search

In 1992-93 a client satisfaction plan for teachers was developed and implemented in the Virginia Beach (Virginia) public school system. This report describes the processes of design and implementation as well as preliminary findings from the field test of a survey soliciting parent perceptions of teacher performance. Although students were recognized as the clients of teachers, parent were the target client group in this study. Forty-six percent of the district's teachers volunteered to participate, and 40,128 forms of a survey developed for the study were sent to parents of their students. The overall return rate was slightly over 50%. These were compiled for each participating teacher. Results were positive, with parent comments about teachers generally favorable. Parents supported the idea of client satisfaction and generally appreciated the teachers' efforts. Results were used to refine the survey for later use. Five tables present survey findings. (SLD)

A secondary mathematics curriculum of open-ended non-routine problems developed by a teacher of high school math over 10 years is described. An open-ended non-routine problem is one that requires problem recognition and orientation, effort, and persistence. It is open-ended in that it allows for various solutions, and requires the student to evaluate a variety of approaches and solutions. Every student can solve the problem, and each solution requires at least a few hours of work. In a sense, it corresponds to an essay in the language arts. A 4-year high school curriculum of non-routine problems has been developed and field tested. The core is a set of 60 non-routine problems, 16 for each year. Each item gives the student a chance to practice problem solving. In field tests over 10 years with the instructor's students, the problems have been refined. Students who have practiced these problems demonstrate "mathematical maturity" in their problem-solving approaches. Appendix A summarizes the non-routine problems for each grade level. Appendix B gives sample problem statements. Appendix C is a draft of teaching suggestions. (SLD)

The impact of portfolio assessment, specifically writing portfolios, on the norms of classroom assessment practices were studied in two English Language Arts classes in a middle class suburban high school in the Los Angeles (California) area. Data were gathered through observations and interviews, and the examination of 25 student portfolios, with student reflections and teacher comments, and 56 student compositions regarding portfolios. Students maintained working portfolios all semester, and selected items for permanent inclusion at the end of each semester. Portfolios counted heavily for grades in both classes. Portfolios in these classes met the qualifying criteria, but no impact on assessment practice was discernible. A single significant criterion was apparent in both classes, and that was completion of the assignment on time. Neither quality of work nor student growth appeared to be a consideration. Both teachers expressed discomfort regarding assessment. Students generally expressed favorable attitudes toward portfolios, but most (9 of the 11 interviewed) viewed the teacher as the primary variable in determining grades. Findings suggest that even expert instructors may lack the awareness, knowledge, and skills to participate in professional assessment practices, particularly with portfolios. One graph and one chart illustrate the discussion. (SLD)

This report contains results of a field test conducted to determine the relationship between a Graduate Records Examination (GRE) linear computer-based test (CBT) and a paper-and-pencil (P&P) test with the same items. Recent GRE examinees participated in the field test by taking either a CBT or the P&P test. Data from the field test indicated that examinees were able to navigate through the CBT with very little difficulty and that their overall reaction to the CBT was favorable. No substantive item-level mode effects were detected. No test-level mode effects were found for the verbal and analytical measures, but a small test-level effect was found for the quantitative measure. The raw-to-scale equating conversions of the CBT nearly matched those of the P&P version of the CBT for each measure, and therefore P&P conversions were used to report CBT scores. Six appendixes provide supplemental information, including item response theory parameter estimates, the plots of differences, the CBT questionnaire results, and seven additional tables. (Contains 31 tables, 13 figures, and 6 references.) (Author/SLD)

InterGeo II, a project of the Commission on Geographical Education (CGE) of the International Geographical Union (IGU), has developed a broadly based, field-trialed testing instrument for making cross-national comparisons of achievement in geography. Field trials of InterGeo II were held in 23 countries. Data were analyzed for national achievement levels as well as cross-national comparisons on six subtests. The data analyses suggest a wide variation in basic geography achievement between and among countries. Despite design problems regarding sample selection and curriculum validity, the test results provide general patterns of information regarding achievement in geography to educators worldwide. The discussion of results from InterGeo II may serve as the basis for further research on cross-national achievement. Research on InterGeo II was supported by the German Research Foundation and the Commission on Geographical Education. Three graphs illustrate data on number of usable answer sheets by country, average scores on InterGeo II by subtest and composite, and correct responses by item on InterGeo II. Appendix A provides a list of national coordinators in the 23 countries participating in the study. (Author/CK)

This document presents a summary of the field test results from the first follow-up of the 1992 Beginning Postsecondary Students Longitudinal Study (BPS:90/92). The BPS:90/92 represents a departure from previous longitudinal studies of high school students in that it starts with a cohort of individuals beginning their postsecondary education regardless of when they completed high school. Consequently, information will be available from non-traditional students who have delayed completion of their educations. The study considers how and why students continue, how they pay for their educations, what they study, and what the impact of study is. An introductory chapter considers the background and purposes of the overall study and the field test. Chapter 2 describes the field test design and methods for 4,501 students in the field test sample, including 3,256 students from 73 institutions who provided interview data. Chapter 3 discusses determinations of first-time beginners, and Chapter 4 reviews evaluation of locating procedures and institutional data collection. Chapter 5 covers interview administration and related procedures, and Chapter 6 evaluates experimental conditions and data quality. There are 31 tables and 5 figures with field test data. Supporting documentation for survey material, the surveys used in the field test, and 18 tables are provided in 3 appendixes. (SLD)

Since 1985, the National Crime Prevention Council and the National Institute for Citizen Education in the Law have been supporting the implementation of the Teens, Crime, and the Community (TCC) curriculum for use in a variety of settings. The curriculum is intended for teenagers aged 14 to 18. The 1992 evaluation was an outgrowth of smaller evaluations conducted in the previous 2 years. The evaluation includes: (1) a site visit to the Justice Resource Institute/Evaluation Program to document changes in the program in the past year; (2) pretest and posttest attitude and knowledge testing of 10 delinquent youth; (3) a site visit to a probation unit to document changes and perceptions of students; (4) site visits to two rural schools; (5) pretest and posttest data from 10 Iowa schools; and (6) interviews with 10 teachers. Attitude change for students appeared to be related to the program's focus on prevention and the use of resource people. Data support the hypothesis that TCC prevents or reduces future delinquent behavior for "regular" students. Students in juvenile justice facilities showed appropriate trends, but results did not reach statistical significance. Successful implementation of TCC appeared to depend on teacher commitment, administrative support, careful attention to the program, student empowerment, and adequate financial resources. (Contains eight tables.) (SLD)

The National Household Education Survey (NHES) was conducted for the first time in 1991 as a way to collect data on the early childhood education experiences of young children and participation in adult education. Because the NHES methodology is relatively new, field tests were necessary. A large field test of approximately 15,000 households was conducted during the fall of 1989 to examine several methodological issues. This report describes the approach used to increase the number of Black and Hispanic American households and youth in the sample. During the field test, an approach that uses demographic information at the telephone exchange level to develop sampling strata was used to oversample Black and Hispanic American households. The yield of the field test sample design was compared to that which would have been expected without oversampling, and the effects of oversampling on the precision of survey estimates are reported. Oversampling did improve the precision of estimates of characteristics of Blacks and Hispanic Americans. Precision losses for overall totals and for non-Black and non-Hispanic American estimates are the price paid in order to improve the reliability of the estimates for Blacks and Hispanic Americans. Two tables and two figures report study data, and two appendixes contain screening ratios used to locate households of various groups and expected and observed sample sizes. (SLD)

The National Household Education Survey (NHES) was conducted for the first time in 1991 as a way to collect data on the early childhood education experiences of young children and participation in adult education. Because the NHES methodology is relatively new, field tests were necessary. A large field test of approximately 15,000 households was conducted during the fall of 1989 to examine several methodological issues. This report focuses on measurement errors arising from the use of proxy respondents. During the 1989 Field Test, a knowledgeable household member was used as a source of information on the school enrollment of each sampled 14- to 21-year-old in the household. In addition, this age group was asked to report on its own school enrollment. The correspondence between the responses given by proxy respondents and the youths themselves is reviewed. Field test findings indicate that it will be possible to estimate the size of measurement error associated with using proxy respondents, but that it may be prudent to collect data from a household member and from the youths themselves to improve the reliability of dropout estimates. Fourteen tables and two figures illustrate the discussion, and an appendix contains statistics on the measurement errors for variables used to derive event and state dropout rates. (SLD)

The National Household Education Survey (NHES) was conducted for the first time in 1991 as a way to collect data on the early childhood education experiences of young children and participation in adult education. Because the NHES methodology is relatively new, field tests were necessary. A large field test of approximately 15,000 households was conducted during the fall of 1989 to examine several methodological issues. This report examines a technique that was used to increase the coverage of 14- to 21-year olds and to capture more dropouts in the sample. The effectiveness of the multiplicity sample in achieving these goals is described. The multiplicity sample results in more older youths and status dropouts, but does little to add to the sample size of younger youths and the event dropouts. The multiplicity sample is effective in reducing undercoverage bias for some statistics, but not others. Ways to improve estimation are discussed. Since the cost of screening households to find those with 14- to 21-year-old members (less than 1 in 4 households) is large relative to the cost of conducting the Household Respondent Interview, it is recommended that the multiplicity sampling approach be implemented in any future survey on dropouts. One table and six figures supplement the discussion. An appendix contains five detailed tables of field test findings. (SLD)

The National Household Education Survey (NHES) was conducted for the first time in 1991 to collect data on the early childhood education (ECE) experiences of young children and participation in adult education. Because the NHES methodology is relatively new, field tests were necessary. A large field test of approximately 15,000 households was conducted during the fall of 1989 to examine several methodological issues. This report analyzes data from the Current Population Survey to identify the extent of telephone undercoverage for 14- to 21-year-olds and 3- to 5-year-olds and bias related to undercoverage for estimates of school dropouts and ECE program participation. Methods for adjusting survey estimates to reduce this bias partially are developed and evaluated. Recommendations are given to improve sampling accuracy for both populations. For estimation of 14- to 21-year-olds in the NHES, it recommended that the mean adjusted poststratified estimator be used because it incorporates an additional smoothing over the within-cell adjusted estimator. Poststratification variables that are more closely related to household income should be considered for the NHES estimation phase, and the use of tenure in addition to or in place of some of the other poststratification variables may be useful in this respect. For estimation of 3- to 5-year-olds in the NHES, the poststratified estimator appears to perform reasonably well for the range of statistics available, and it is recommended for use with this target population. Problems concerning undercoverage bias due to households without telephones were not substantial. Fourteen tables and eight figures present field test findings. An appendix discusses the source and reliability of estimates. (SLD)

The National Household Education Survey (NHES) was conducted for the first time in 1991 as a way to collect data on the early childhood education experiences of young children and participation in adult education. Because the NHES methodology is relatively new, field tests were necessary. A large field test of approximately 15,000 households was conducted during the fall of 1989 to examine several methodological issues. This report describes the design of the field test and the outcomes of its collection activities. Unit and item response rates and the burden associated with survey participation are described. The telephone methodology of the NHES survey offers some important advantages for the program; it is less expensive than a comparable personal interview survey, and it can also be instituted relatively quickly, address several subject matter areas at the same time, and provide results in a very timely fashion. The NHES offers the potential for examining education issues that have been difficult to address through the more traditional National Center for Education Statistics surveys that are based on institutional samples. The field test has shown that many of the concerns can be handled adequately. Fifteen tables and six figures illustrate the overview. (SLD)

This report covers progress in a Comprehensive Teacher Training Project (CTTP) from its inception in 1987 to 1992 including background, current status, problems and solutions, upcoming internal evaluation, future events, and feedback on the project. CTTP, a project of the Organization for Cooperation in Overseas Development, a Canadian-based organization, is designed to increase significantly the knowledge and skills of untrained elementary school teachers in the West Indies in English language, mathematics, social studies, and integrated science, with a view to preparing these individuals to enter the teacher training programs there. Rising populations, economic cutbacks, stronger regionalism, and more strictly enforced entrance requirements to local teacher colleges and universities have all generated the need for academic upgrading and development of pedagogical skills for untrained teachers. Phase I of the CTTP was designed for curriculum development and field testing. Section 1 outlines the background of the development of the CTTP. Section 2 outlines training for participants. Section 3 examines the past and present role of the Regional Office in the CTTP. Section 4 provides a production update for all four courses offered by the CTTP. Section 5 addresses the development of the field-testing process. Section 6 looks at the future of the CTTP. Section 7 concludes with recommendations for the balance of Phase I and proposals for Phase II. Includes a glossary, 16 tables, and a list of 25 appendixes (which are not included). (JB)

In 1991, a project studied implementation of the American Society for Training and Development's Workplace Basics model. Field testing was conducted in two Virginia industries: Corning Incorporated in Waynesboro and American Safety Razor (ASR) in Verona. Corning had fewer than 100 employees and an 85 percent female work force; it had shifted from manufacturing to service and distribution. ASR was a larger (650 employees) manufacturing facility with a strong union. Project outcomes at Corning were lauded as positive, valuable, and worth the time and energy. The most significant outcome at ASR was the new structure for communication and decision making. The basic skills in these sites were reading, writing, mathematics, communication, and problem solving. Work design, work environment, and management practices determined the program's scope. Information needs included clear statements of what the job entails, how to accomplish it, and its relationship to the larger organization. Program development and implementation took time and required significant employee involvement. Skilled leadership was required. Training offered in weekly 1-hour segments allowed participants time to reflect and practice within the work environment. Continued lack of clear understanding of how organizational needs are related to skills, skills to performance, and performance to results suggested difficulty in getting support for evaluation. (Appendixes contain the following: a blueprint for program implementation; national and local economic climate; survey results; and a 24-item bibliography.) (YLB)

The Institute for the Study of Adult Literacy developed and field-tested a curriculum and instructional materials for teaching occupationally specific vocabulary in the health care field. The materials, which are in the form of a narrative about a woman who learns medical terminology from a friend, were designed to serve as a vehicle for training students to use structural analysis to understand vocabulary relevant to the responsibilities of beginning licensed practical nurses (LPN). Project staff reviewed existing medical vocabulary texts and assembled a list of medical vocabulary word parts that were then each reviewed by three working registered nurses to determine their relevance to a beginning LPN. A pretest/posttest format was used to field-test 11 adult learners at two sites in Pennsylvania. The mean pretest score was 6.91 out of 45 (range, 3 to 15), and the mean posttest score was 39.27 (range, 33 to 44). Both the instructors and adult learners who used the new curriculum and materials reacted positively to them when interviewed. (MN)

A pilot study was conducted to field test instruments that could be used to assess, evaluate, and improve curricula to be used in the Agri Tech Prep 2000 (ATP 2000) project. The project features a 4-year tech prep program linking New York high school and postsecondary curricula to prepare students for careers in agriculture or acceptance into a college-level agriculture program. A survey instrument focusing on students' economic/family backgrounds, career interests, knowledge/perceptions of agriculture, educational background/interests, work experience, and personality traits was administered to 172 10th-grade students at 5 pilot schools associated with ATP 2000. The Strong Interest Inventory was administered to 108 students. The content validity of the researcher-developed instrument was confirmed, and its findings were summarized for use in future curriculum development. According to the survey instruments, most students did not perceive the integration of agriculture and academic courses, 30.9% planned to attend a four-year college, 19% planned to attend a two-year college for their terminal degree, and an additional 17% planned on attending a two-year college before transferring to a four-year institution. (Appendixes constituting over half this document contain the survey, survey administration instructions, selected findings, and the Strong Interest Inventory.) (MN)

Test items are designed to be representative of the subject areas that they measure and to reflect the importance of specific domains or item types within those subject areas. Content validity is achieved by content specification and number of items in each content domain included in the design of the test. However, largely due to the normal attrition of less desirable items during the field test phase, the averaged proportional domain information of the 1990 National Assessment of Educational Progress (NAEP) reading assessment deviated from the original design. The relative value of polytomous and dichotomous items in a 1990 NAEP science assessment booklet was evaluated. The current study included 1,248 students from the NAEP grade 8/age 13 years sample who were administered booklet 20, which contains 3 science blocks. Results show that the polytomous items were only as informative as dichotomous items in regard to both the model-based relative information values as well as the reduction in the variance of posterior proficiency distribution. It is contended that in the future, there should be a greater concern that the reporting score reflects the content validity of the instrument specifications. The components of information should be monitored routinely by the item response models or by the content, and the reduction of the posterior variance should be monitored through additional items. Six tables and four graphs illustrate the analysis. (Author/SLD)

An empirical model for evaluating a school-TAFE (technical and further education) link program was derived from the results of an evaluation of a Queensland, Australia, program. The program established a TAFE center run by the county TAFE college on high school grounds, in which students could take 50% of their courses. The evaluation gathered information from the high school, TAFE, and community. The following criteria were used: suitability of location, effectiveness of organizational structure, use of resources, use of staff, suitability of curriculum, cost effectiveness, achievement of expected outcomes, and alternative solutions. For each criterion, relevant questions were generated to gather both qualitative and quantitative information. Field test results were as follows: the location was not suitable due to lack of potential growth and employment opportunities and small town population; layout of the building inhibited offerings of new programs; center and school staff did not communicate regularly; with no updated written policies on center organization and operation, resources and facilities were not being used to their potential; subjects offered provided very limited employment prospects; and although TAFE students acquired job skills and experience, they were not offered classes for entrance to university education. Options included closing down the center, improving it, or reorganizing it. (YLB)

Alternative mathematical items administered as prototypes at the spring 1989 Field Trials of the Scholastic Aptitude Test (SAT) are evaluated for differential item functioning (DIF) and differential speededness. Results for Algebra Placement (AP) and Student Produced Response (SPR) items are presented and contrasted with results obtained on two current SAT-Math (SAT-M) items: Regular Math and Quantitative Comparison. Analyses on comparisons between female and comparable male examinees, and between Asian-American, Black, and Hispanic examinees in comparison to comparable White examinees indicate that both of these alternative items appear to have DIF. Additional DIF analyses comparing the use of an internal versus an external matching criteria for the SPR items show evidence of negative DIF with either criteria. Results using the Mantel Haenszel Delta DIF statistic are more extreme than DIF results using the standardization p metric DIF index. Differential speededness results indicate that the two Math prototypes have slightly higher levels of differential speededness than the SAT-M. The SPR items pose an interesting problem for DIF. The definition of an appropriate DIF matching criterion for constructed response items and metric differences between methods and their effect on difficult or easy items also need further exploration. One figure and 13 tables illustrate the analysis. (Contains 15 references.) (Author/SLD)

The Second Follow-Up Field Test (SFUFT) of the National Education Longitudinal Study of 1988 (NELS:88) was conducted in 1990 and 1991 by the National Opinion Research Center and the Educational Testing Service. The SFUFT evaluated free response items (FRIs) for possible inclusion in the Second Follow-Up Main Study (SFUMS) and tested survey instruments, procedures, and forms. This report focuses on a methodological examination of the SFUFT data and the questionnaires developed. The SFUFT included four major survey components: students, dropouts, parents, and school administrators. Four questionnaires and five cognitive tests were developed, including the: Student Questionnaire, Not Currently In School (Dropout) Questionnaire, Parent Questionnaire, School Administrator Questionnaire (SAQ), and Cognitive Test Battery. Most of the students surveyed were in the last half of their senior year in high school. The major purpose of the parent component field test was to gather sufficient observations for evaluating the performance of questionnaire items. Data were obtained from: 2,254 students in 94 schools (overall response rate of 70.5%); 108 dropouts; 506 parents (response rate of 73.5%); and 65 schools/principals. The results show that the goals set for this pretest effort were attained, the FRIs were successfully administered, and the FRIs measure the same/similar domains as do multiple-choice questions. Recommendations for implementation of the SFUMS are provided. Numerous tables and figures are included. Documentation of 1990 instrument development meetings, documentation supporting questionnaire experiment analyses and cognitive test analyses, and a summary of cognitive interviews are appended. (RLC)

As a step in considering the addition of dropout statistics to the Common Core of Data (CCD) collected by the National Center for Education Statistics, telephone and personal interviews about the feasibility of implementing a standard national dropout statistics collection were conducted. Interviews were held from May through September of 1990 with different survey instruments for 30 participating states and 24 nonparticipating states. Open-ended and forced-choice questions were included to identify perceived barriers to and incentives for full-scale implementation of dropout statistic collection. Twenty-eight participating states believed that the data collection was feasible for statewide implementation. Nineteen of the non-participants reflected a similar attitude, responding that there were no serious barriers to providing dropout data if such an item was added to the CCD. Major areas to be addressed before adding a dropout statistic to the CCD include: (1) logistical problems in data collection and reporting; (2) problems of complicated and rigorous definition of dropouts; (3) determining the possibility of assessing the status of school leavers with sufficient accuracy to provide a reliable dropout count through school district records; (4) problems of classifying school leavers as dropouts with sufficient comparability when state policies and procedures differ; and (5) determining the effects of differences in operations among state agencies. Four tables present the survey results. (SLD)

One aspect of the development of a performance assessment is addressed--the field testing of the tasks. In a performance-based assessment, the tasks must allow each examinee a fair and equal opportunity to give the best possible demonstration of his or her ability, and must elicit a performance sample that enables scorers to evaluate each examinee adequately. A method of field testing is proposed that collects information via two supplemental instruments administered during the field test: (1) one completed by examinees; and (2) one completed by the field test raters. Each instrument produces quantitative data via machine-readable response forms and qualitative data via written comments. The method is illustrated by field testing four parallel forms in French and Spanish of the Texas Oral Proficiency Test, a performance-based assessment for certification of Spanish, French, and bilingual education teachers in Texas. The field test involved 160 examinees and eight raters. The use of the two instruments during field testing allows both examinees and raters to identify poorly functioning items on a performance-based assessment. This method, which encourages examinees and raters to provide focused written feedback, enables both to be involved in test development and improvement. Three tables present data, and an appendix provides the supplemental data collection forms. (SLD)

The University of Southern Queensland (USQ) Distance Education Centre compared the cost and instructional effectiveness of the Optel Telewriter to that of telephone tutorials and face-to-face instruction. Thirty enrollees in a data analysis class used the Optel Telewriter, which permits two-way voice and graphic communication, and 30 other students did not. The Optel Telewriter users passed the course's final examination at a higher rate than did the students in the control group (73% versus 66%) and achieved a higher mean score than the controls (57.46 versus 55.80). Both the students and teachers who used the Optel Telewriter felt that it is a viable alternative to the face-to-face component of distance education. Students indicated that the Optel Telewriter enhanced their academic performance, and teachers stated that it significantly expanded the pedagogical possibilities of the audio teleconferencing system with which they had worked previously. Although tutorials conducted via the Optel Telewriter cost approximately twice as much as telephone tutorials, they were still far more cost-effective than face-to-face instruction. (Appended are a chart detailing the organizational structure of the USQ Distance Education Centre and the student attitude questionnaire. Twelve tables/figures and 13 references are included.) (MN)

Effects of assessment on curriculum and instruction were studied in an investigation that considered local educators' reactions to statewide minimum competency testing, the instructional effects of implementing these tests, and variations within each state. The two states examined were Pennsylvania, a low-stakes situation with relatively minor consequences for the student, and Maryland, a high-stakes situation where high school graduation depends on passing the tests. Pennsylvania students were tested in grades 3, 5, and 8, while Maryland students were tested beginning in grade 9. Fieldwork was conducted in six sites in each state, and over 250 local educators and students were interviewed. A survey completed by 277 of 501 Pennsylvania districts and 23 of 24 Maryland districts provided additional information. The impact of the testing program was far greater in the high-stakes situation, and this impact seemed positive as long as the districts were not under too much pressure. Both positive and negative consequences are discussed. Five tables present study findings. (Contains 14 references.) (SLD)

In contrast to multiple-choice test questions, figural response items call for constructed responses and rely upon figural material, such as illustrations and graphs, as the response medium. Figural response questions in various science domains were created and administered to a sample of 347 fourth, 365 eighth, and 322 twelfth graders. Data were gathered in conjunction with field testing for the 1990 National Assessment of Educational Progress. Item and test statistics from parallel sets of figural response and multiple-choice questions were compared. Figural response items were generally more difficult, especially for questions that were difficult (p<.5) in their constructed-response forms. Figural response questions were also slightly more discriminating and reliable than their multiple-choice counterparts, but had higher omit rates. The paper addresses the relevance of guessing to figural response items and the diagnostic value of the item type. Plans for future research on figural response items are discussed. An appendix describes figural response items. (Contains 3 tables, 10 figures, and 20 references.) (Author/SLD)

This paper posits a taxonomy for categorizing issues that arise in the evaluation of Intelligent Tutoring Systems (ITSs). The taxonomy has three dimensions: Life Cycle of Evaluation, Research Issues, and Methodological Issues. The Life Cycle dimension has four levels: pre-experimental, laboratory study, field study, and initial operational test and evaluation. The three levels of the Research Issues dimension--functionality, effectiveness, and cost--are subsequently further divided into several sublevels. The Methodological Issues dimension is discussed in the context of each of the Research Issues levels. A recommendation from this work is that ITS evaluation studies should adopt multi-dimensional, multi-method designs. (Author)

This paper discusses the applicability and adaptability for evaluating reference interviews in library science of the Arizona Clinical Interview Rating Scale (ACIRS), a process-oriented scale originally developed in 1976 at the Arizona College of Medicine to evaluate the interview performance of medical students. The analysis surveys the content and reliability and validity data and discusses the appropriateness of the criteria and the presentation for reference interviews. It is concluded that the ACIRS is an effective interview scale emphasizing process-related criteria that can be modified fairly easily for application to pre-search interviews. (9 references) (GL)

It is argued that the three basic issues that have typically been the concern of formative evaluation (comprehensibility, appeal, and usefulness) tell us very little about the role that an instructional product can play in the ongoing environment of the classroom. The focus in this paper is on the field test, i.e., on research situated within the framework of existing classroom activity. Four issues considered central to situated research are discussed: (1) How is the instructional prototype interpreted by teachers? (2) How does the prototype function as part of the ongoing content curriculum that is both the backbone and blueprint for classroom goals and activities? (3) How does the prototype function within the organizational context of the classroom? and (4) Within the social context of the classroom, how is the prototype interpreted by students? Each of these questions is addressed using examples from formative research with two different prototype products: a computer-based set of interactive cognitive tools to support inquiry practices in science ("Inquire"), and a television series that uses a narrative format to introduce students to different arenas of scientific and mathematical inquiry ("The Second Voyage of the Mimi"). (8 references) (GL)