Search

The National Assessment Summit was designed to bring together users and producers of assessments and engage them in a discussion of unmet assessment needs and untapped assessment capacities. The meeting was truly a summit in that national foreign language organizations and associations were asked to nominate representatives to bring their organization's perspective to the forum and also to take back to their organization ideas for action. Specifically, the National Assessment Summit aimed to identify a set of priorities for the next two years (2005-07). At a follow-up National Assessment Summit in 2007 in Portland, Oregon, under the sponsorship of the Center for Applied Second Language Studies (CASLS) at the University of Oregon and the Center for Advanced Research in Language Acquisition (CARLA) at the University of Minnesota, these organizations will gather to report on progress made and to consider an agenda for the following two years (2007-09). The National Assessment Summit identified seven priorities for the coming two years. Each priority is addressed in a section of this series of papers and is authored by one of the participants. These priorities are: (1) Articulation; (2) Assessment Literacy; (3) Suite of Assessments; (4) Test Database; (5) Models and Anchors; (6) Research; and (7) Advocacy. Taken as a whole, these papers describe state-of-the-art language assessment and articulate a two-year agenda for improving assessment and educational practice. This publication includes: (1) Introduction (Carl Falsgraf); (2) Achieving Articulation through Assessment (Jacque Bott Van Houten); (3) Assessment Literacy (Peggy Boyles); (4) Suite of Assessments (Ann Tollefson); (5) Foreign Language Assessment Database (Margaret E. Malone and David W. MacGregor); (6) Models and Anchors (Elvira Swender); (7) Assessments Research (Ursula Lentz); (8) Promoting Language Advocacy Through Assessment (Martha G. Abbott); and (9) Conclusion (Paul Sandrock). Individual papers contain references and footnotes.

This paper presents a new method to test for grade inflation. Defining grade inflation as the decreasing value of grades in the coin of student achievement, the study avoids the assumption that a rise in mean grade is a necessary condition of grade inflation. The study, which was conducted at a single university, also sought to determine whether grade inflation could be demonstrated if mean grades have leveled off. To measure general academic abilities, Graduate Record Examination (GRE) data was used. Data was grouped into three time spans (1976-80, 1984-88, and 1992-96). To examine the influence of students' majors, data were also separated into two broad groups, "humanistic" and "scientific," because literature had shown roughly that the former revealed more grade inflation than the latter. Findings suggested grade inflation relative to GRE total and verbal scores, but no evidence of grade inflation when GRE quantitative or GRE analytical scores were used, and it is postulated that this is a result of declining academic standards in the sense that verbal reasoning ability associated with grade point average is lower than it used to be. Differences were also found between disciplines, with some evidence of grade inflation found for science majors. (Contains 17 references.) (RH)

This paper describes efforts to devise a more effective method of assessing oral proficiency in English using a computer-based testing and computer-based test result management system. This new test would be designed to directly operationalize the second language constructs it measured, and would be developed with reference to the best practices revealed by recent research in the field. Scores are automatically sent to databases at the completion and evaluation of each task. Student performance is then used to drive an adaptive algorithm, determining the difficulty of successive tasks. This computerized testing would aid in assigning skills or constructs to individual tasks, apportion the skills required for each task, and identify the features of interaction elicited in individual tasks. The study results revealed not only which tasks elicited most assessor/interlocutor interaction, but also that a great deal of the interaction took place outside the task boundaries. There has been a substantial move towards acknowledging the importance of interaction in oral proficiency and, therefore, oral proficiency testing featuring discourse and conversation analysis. The research detailed in this paper indicates that greater interaction does seem to be associated with greater oral proficiency, yet there are nagging issues of test validity still to be addressed in future research. (Contains 47 references.) (KFT)

This document contains the five papers presented at a meeting at which key issues in evaluating workplace literacy programs were discussed. In "Key Components of Workplace Liteacy Projects and Definitions of Project 'Modules,'" Judith A. Alamprese describes the context for evaluating the National Extension Program, components of workplace literacy programs, and validation of program components. Design alternatives for a national program evaluation and design options for an effective practices study are explored by Jorie W. Philippi in "Measuring Workplace Literacy Program Effectiveness: Evaluation Purposes and Methodologies." In "Advantages and Disadvantages of Longitudinal Designs for Evaluating Workplace Literacy Programs," Elisabeth Hayes examines the following: the short-term nature of literacy gains, random assignment versus other methods, availability of comparison/control groups, and practical feasibility of alternative designs and procedures. The last two papers, "Articulation and Measurement of Participant Outcomes in Workplace Literacy Programs" (Larry Mikulecky) and "Articulation and Measurement of Program Outcomes" (Anthony R. Sarmiento), focus on the relationships between curriculum/instructional design and various participant/program outcomes and assess the availability and appropriateness of data and various data collection methods. Most papers include substantial bibliographies. Appended are the work group conference agenda, biographies of the presenters, and a list of work group participants. (MN)

The European Association for Quality Assurance in Higher Education (ENQA) realises that there is a steadily growing interest among students and all stakeholders of Higher Education and the public at large in accessing detailed and reliable information on the quality of individual study programmes, faculties and higher education institutions. ENQA is aware that external quality assurance activities in higher education set up by quality assurance agencies are producing only part of the information. Besides quality assurance agencies that enshrine their activities in the ESG, there is a growing number of providers of information about higher education. The proliferation of ranking schemes and other transparency tools is one aspect of this increasing need for information about higher education institutions and their activities. The purpose of this seminar was to discuss expectations from higher education institutions and other stakeholders concerning information on quality of higher education, and the contribution of quality assurance and transparency tools to meet these expectations. The outcome of the discussions is a position paper on information and transparency tools, which can be found in Part 2 of this report. This paper is divided into two parts: (1) Key messages; and (2) ENQA position paper on transparency tools. Programme of the Seminar is appended. (Contains 2 footnotes.)

The purpose of this paper is to summarize the research on service learning pedagogy into the realm of professional practice for students with disabilities. At this point in the development of a research base for service learning, an emerging need for studies is reported. Limited research exists which studies the impact of service learning activities with respect to the retention of students with disabilities, particularly those who are also African American. The authors suggest that one role that university faculty can take is to create, implement, and evaluate service-learning programs in urban communities that incorporate students, adults with disabilities, the local public school k-12, and teacher educators and researchers from nearby colleges and universities. (Contains 2 tables and 2 footnotes.)

This paper discusses the use of focus groups to investigate criteria used by college students to determine their satisfaction with higher education, and presents the results of a pilot study. It reviews research on the use of focus groups, discusses the development of a focus group guide for the current study, and examines problems associated with recruiting students to participate in focus groups, such as incentives for volunteers and no-shows at group sessions. The paper discusses the role of the rapporteur and the scribe in group sessions, as well as the actual conduct of the sessions. The pilot study found that the overall undergraduate student criterion for determining satisfaction or dissatisfaction with their college experience at a regional university included: (1) finding pride and inner satisfaction with accomplishments; (2) a flexible curriculum; (3) a university with status and prestige; (4) a university degree that opens the door to career opportunities; (5) the encouragement of student involvement; (6) a caring faculty; (7) an opportunity for independence; (8) a student-oriented university administration; and (9) a university experience that provides an opportunity for growth and development. An appendix contains a copy of the focus group guide. (Contains 21 references.) (MDM)

After decades of disinterest, evaluation of the performance of elementary and secondary teachers in the United States has become an important educational policy issue. As U.S. states and districts have tried to upgrade their evaluation processes, one of the models that has been increasingly used is the Framework for Teaching. This paper summarizes validity evidence pertaining to several different implementations of the Framework. It is based primarily on reviewing the published and unpublished studies that have looked at the relationship between teacher evaluation ratings made using systems based on the Framework and value-added measures of teacher effectiveness. The research results summarized suggest that teacher ratings made using evaluation systems based on or related to the Framework have a correlation with value-added estimates of teacher effectiveness, in the 0.2 to 0.3 range. Measurement error in both the value-added estimates and evaluation ratings likely bias these correlations downward. Though fragmentary, the evidence reviewed suggests that Framework-based evaluation ratings can show substantial inter-rater agreement and, if multiple observers and multiple occasions of observation are used, the ratings can be reliable. However, a score based on a single observation by one observer is likely to have low reliability. The paper also discusses ways to extend and improve validity research on teacher evaluation systems. "Inter-rater Agreement Details for Cincinnati" is appended to this document. The contributions to this research of current and former colleagues Steven Kimball, Herbert Heneman, Allan Odden, H. Alix Gallagher and Bradford White, are gratefully acknowledged. (Contains 6 tables, 2 figures, and 6 footnotes.)

The Comparing Options in Secondary Mathematics: Investigating Curriculum (COSMIC) project is a longitudinal study of student learning from two types of mathematics curricula: integrated and subject-specific. Previous large-scale research studies such as the National Assessment of Educational Progress (NAEP) indicate that numerous variables are associated with mathematics performance; however, consideration of too many variables can result in biased models that inflate the amount of variation explained. This paper addresses key issues in the design of studies of curricular effectiveness including data collection, data reduction, and coherence in modeling student achievement in Year 1. At the student-level, demographic and prior achievement data was collected for 2621 secondary school students in five states. These scores were transformed onto NAEP scales to yield a COSMIC Prior Achievement (CPA) score that provided a control for prior achievement across the entire sample. At the teacher-level, data related to teaching experience, professional development, teacher beliefs, and curriculum implementation was collected. Student learning was measured using the Iowa Test of Educational Development: Mathematics (Level 15) and two project-developed assessments (Test A and Test B) that embodied curricular validity (National Research Council, 2004). Item Response Theory (IRT) scaled student scores were adjusted for CPA and subsequently aggregated by teacher. Seven factors were extracted from 24 teacher-level variables through principle components analysis, four related to curriculum implementation and three to teacher characteristics. Curriculum Type, Percent Free/Reduced Lunch (%FRL), and Lesson Development variables were examined in relation to student outcomes. Full and partial correlations procedures were used to identify key variables associated with student mathematics learning. Significant correlations included Curriculum Type with student scores on Test A and Test B; %FRL correlated (negatively) with scores on Test A and ITED-15; Technology & Collaboration with scores on Test B and ITED-15; Opportunity-to-Learn (OTL) with performance on all three outcome measures; and Knowledge of Standards with scores on Test A and Test B. When controlling for %FRL, Curriculum Type was significantly correlated with all three dependent measures. A similar pattern emerged when controlling for OTL, suggesting a relationship between OTL and %FRL. In contrast, controlling for Technology & Collaboration weakened the correlations between Curriculum Type and all three assessments. Recommendations related to key issues in data collection, data reduction, and coherence that are prerequisite to the productive modeling of student achievement data in comparative curriculum studies are provided. (Contains 13 tables, 5 figures, and 4 footnotes.) [This paper is based on research conducted as part of the Comparing Options in Secondary Mathematics: Investigating Curriculum (COSMIC) project, a research study supported by the National Science Foundation.]

The thesis of this paper is that, in evaluating most innovative educational projects that are conducted in schools, tenets of formal experimental design and associated inferential data analysis methods should be given limited emphasis. The basis of this thesis lies in the problems and difficulties that undermine the design and implementation of the typical project. Sections of the paper include discussions on design problems and difficulties, data collection, data analysis and reporting, statistical proportions as an evaluative measure, and graphs. These are followed by a presentation of the considerations that may be made in an evaluation report in the face of the problems and difficulties. (Contains 14 references.) (RR)

This paper reviews foreign and domestic sexism research and practice in language testing and reveals that China lags behind in this sociolinguistics perspective in both theoretical study and practice. The paper indicates that sexism is represented in the listening comprehension section in National Matriculation English Test (NMET) after a case study into the listening comprehension part in NMET 2000-2005. It is hoped this tentative research will trigger more studies, ameliorate language testing practice in the perspective of gender equality and reduce DIF [differential item functioning] (from which validity will suffer) in especially large scale language testing programs. (Contains 1 endnote.)

The evaluation methodology delineated in this paper is a work in progress. The purpose is to develop a method for evaluating social studies curriculum materials that encompasses the materials' comprehensibility and the perspective offered. The paper draws from past evaluations that focused on either comprehensibility or perspective but have not typically encompassed both types of evaluation. It offers examples of studies for comprehensibility such as those done by I. L. Beck and M. G. McKeown (1991), Bonnie B. Armbruster and Thomas H. Anderson (1984), Jere Brophy (1990) and Jane J. White (1988). It offers examples of studies for perspective as well, such as those by Jean Anyon (1978, 1983), Ray Allen Billington (1966), Joel Taxel (1984), Linda S. Levstik (1990), and others. As a general overview, the evaluation methodology determines whether the curriculum materials are age appropriate, educationally significant, and offers the perspective of groups of people important to the topic. Noting that a content goal is necessary for each topic being evaluated, the paper states that this content goal serves as a model for age appropriate and educationally significant information on the particular topic. It explains that, for the first use of the evaluation, three topics from the original textbook series for elementary school social studies by Hanna and Rugg: (1) pre-industrial Pacific Island societies, specifically Samoa and New Guinea; (2) Dutch colonial New York; and (3) the beginnings of industrialization in the 18th century English textile industry. It notes that the aim is to use this evaluation in the future with other materials, modern and historical fiction and non-fiction tradebooks, videos and computer resources as well as textbooks and materials at grade levels other than elementary. (Contains 22 references.) (BT)

This paper examines the discourses that dominate thinking about assessment in mathematics education; that is, to analyze the sets of constructs, assumptions, and values that underpin research, curriculum development, and teacher education with regard to assessment. Dominant discourses within mathematics education are analyzed and the social functions that assessment fulfills within the classroom and in the broader society are discussed. (Contains 34 references.) (KHR)

Early Head Start (EHS) is a comprehensive, two-generation program that includes intensive services that begin before the child is born and concentrate on enhancing the child's development and supporting the family during the critical first 3 years of a child's life. This paper discusses approaches to measuring father involvement in their children's lives and examines specific measurement methods used in the Father Studies of the Early Head Start Evaluation Project. In addition, the paper highlights lessons from the field that have emerged as father involvement is measured in the ongoing EHS project. Father involvement is described as a multidimensional, continually evolving concept. Time use measures of father involvement are described, and various models of involvement are presented. The paper notes limitations of existing instruments, including the questionable validity of self-report, the often interchangeable use of generic fathering versus child-specific fathers, and the limited generalization of findings from middle-class, European American groups to other groups. The paper describes the EHS program, summarizing the 17 EHS program sites participating in the national evaluation and local research and representing a diversity of locations, populations, culture, ethnicity, and urban-rural settings. Four strands of father studies planned are detailed: (1) interviews with fathers of 24- and 36-month-olds; (2) study of mothers and fathers of newborns; (3) study of strategies used by programs to engage fathers and father figures; and (4) local research studies focused on fatherhood issues significant to local populations. Measurement instruments are then detailed, including questionnaires and videotaped father-child interactions that will yield both qualitative and quantitative data. Finally, the paper explores some of the challenges in assessing father involvement among low-income men, including developing coding systems, obtaining reliability in coding father-child interactions, operationalizing specific constructs, retaining participants, and studying social fathers. The paper concludes by pointing out advances that have been made in the measurement of father involvement. (Contains 23 references.) (KB)

This paper discusses the development and implementation of the system-wide assessment of arts learning in primary and secondary schools that took place in Western Australia in September 1996. The paper describes the background to the assessment, the design brief, and the assessment package as a whole, focusing on the visual arts tests. It discusses issues that arose during the development of the test and outcomes highlighted in the data analysis. (BT)

This paper details the origins of the media comparison study methodology, its current use as an evaluation instrument in distance education, and recommendations for more stringent discrimination between research and evaluation in the field of distance learning. The following topics are discussed: (1) the history of research in instructional technology from 1968 to the present; (2) recent research in distance learning, including the difference between evaluation and research, and weaknesses in many instructional technology studies (e.g., failing to link the study to a robust theory, poor literature review, weak treatment implementation, measurement flaws, inadequate sample size, and poor analyses) that bias the research toward not finding a statistically significant difference; and (3) evaluation versus research in distance education, focusing on appropriate uses of media comparisons for distance program evaluation, as well as alternative methods and exemplary models. (Contains 55 references.) (MES)

This paper describes techniques for creating a simplified form for evaluating the effectiveness of a distance learning event (i.e., a training or educational program occurring within 1 day) for which the development of comprehensive measures is often not feasible because the brevity of the event precludes the justification for a lengthy evaluation. The paper focuses on distance learning events for the National Guard Bureau. Background is provided on evaluation measures, including student reaction, learning, behavioral criteria, and results criteria. Development of the compressed evaluation form--one page of five assessment categories (course, technology, instructor, demographics, and motivation) with a total of 22 items--is discussed. Testing of the compressed evaluation form is described. Eight distance learning events were sampled, representing the primary distance learning events available on a nation-wide basis during the evaluation time frame. Training was delivered through a multi-point, one-way video, two-way audio connection through satellite links. A total of 1,306 soldiers participated in the training; the mean rate of returned evaluations was 74%. Two tables present data on student ratings of events and cross tabulation of amount learned by previous training. It is concluded that a wealth of information can be obtained from a simple, one-page evaluation form. (DLS)

Since educational statistics, which are relatively easy to obtain, can only attempt to measure "quality," this paper asks how quality in higher education is assessed and how educational rankings, which are defined as benchmarks or attempts to measure, contribute to this process. The paper notes that while attempts to rank institutions of higher education date to the late nineteenth century, two seminal studies published in the 1980s and 1990s generated immense interest in the academic community and prompted much analysis. Since then there has been a proliferation of annual publications by the media, but the paper cautions that many are not published on a timely basis nor are their data comparable. Some discussion is devoted to the four major ranking methodologies--reputational rankings, citation analysis, faculty productivity, and statistical rankings--and five articles, which are considered classic studies, are noted as are efforts at international rankings. The paper then offers eight requirements for perfect educational rankings, including standardized reporting methods, peer review, multidimensional techniques, and focus on interdisciplinary programs and emerging disciplines. The paper concludes that while statistics-based rankings are based on hard facts, consumers need to be aware of the shortcomings of opinion-based rankings. (Contains 34 reference notes.)(CH)

This paper shows how the concept of general objectivity can be used to improve behavioral science measurement, particularly as it applies to the Lexile Framework, a tool for objectively measuring reading comprehension. It begins with a dialogue between a physicist and a psychometrician that details some of the differences between physical science and behavioral science measurement. Building on these distinctions, a definition of measurement is offered that describes what goes on in the physical sciences and represents an attainable ideal of what should go on in the behavioral sciences. This definition of measurement is formalized in an equation that turns out to be the Rasch model, with the important difference that indicant calibrations are obtained via theory, not data. Through the use of theory-based calibrations, a generally objective estimation of the measure parameter in the Rasch model is achieved. The paper then examines the differences between local objectivity obtained with the Rasch model and general objectivity obtained with a theory-enhanced version of that model. Next, it reports on a 10-year study of reading comprehension measurement that implemented the concept of general objectivity through the development of the Lexile Framework. Finally, it summarizes several of the benefits of objective measurement and general objectivity as they might be realized in the measurement of constructs other than reading comprehension. Contains 52 references, and 4 tables and a figure of data. (EF)

The Tell-Me-A-Story Apperceptive Test (TEMAS) is an individual apperceptive test, similar to the Thematic Apperception Test (TAT) of personality assessment. The TEMAS is intended for use with children and adolescents ages 5-18 years old, and is marketed as a personality assessment tool for minority, primarily Hispanic and African-American, and non-minority girls and boys. This paper contains a "Practical Evaluation" section, which: (1) describes the physical composition of a TEMAS; (2) discusses possible confounding factors intrinsic to the administration and interpretation of the TEMAS; and (3) describes how TEMAS stories are scored. The paper's "Technical Evaluation" section examines a sample of 642 children and discusses TEMAS' internal consistency, test-retest reliability, interrater reliability, content validity, construct validity, criterion validity, and concurrent validity as obtained from the data collected. The conclusion is drawn that the TEMAS is an excellent attempt at balancing out the disparity in multi-cultural assessment, while its primary value currently seems to be as a clinical instrument. Further studies of the African-American population's experiences with this instrument need to be conducted, and work needs to be done in general to ascertain whether the TEMAS will be suitable for suburban minority youth. (TS)

A literature search of the ERIC system was conducted with regard to papers focused on evaluation models in science education. The purposes were to identify models, their common and unique features, and to examine the guidance they could provide for evaluating science education programs. The search yielded a small number of entries pertinent to the areas of interest which were subsequently classified into three categories--micro (formative through the completion and summative evaluation of programs and products) and macro (contextual) levels and the interface between them. The categories were discussed along with some of the more general evaluation literature dealing with evaluation models. Conclusions were then drawn about why only a limited literature base existed, general models versus a more "situation" imbedded approach to understanding evaluation, the value of model type thinking, and implications for the future needs of evaluation in science education. (Author)

Despite current interest with the concept of the "New American School" model discussed in "America 2000," school systems continue to approach educational reform and restructuring by tinkering with key organizational components in isolation. The total school organization requires assessment and profiling to determine which key components are drags on efficient and effective increases in the goal of student achievement. Arguments, fighting, interest-group power-plays, and politics do not accomplish discovery and application of already-known ways to improve organizational effectiveness. The path to positive restructuring and reform is through use of the latest available technology and qualitative/quantitative evaluation research procedures. To accomplish organizational assessment from the ground up requires the innovative, validated, and holistic model described in this paper. The process, procedures, and outcomes involved with readiness assessment, profiling, and improvement planning correspond to exhibits A through F and include: (1) adopting an assessment model; (2) identifying key organizational factors; (3) organizational profiling; (4) reporting results of organizational research; (5) improvement planning; and (6) prioritizing factors. Every component and function of educational organizations must be held accountable for the goals of student learning and development. Exhibits A through F are included. (17 references) (MLH)