Search

This digest describes the three most prominent approaches to essay scoring by computer: (1) Project Essay Grade (PEG), introduced by E. Page in 1966; (2) Intelligent Essay Assessor (IEA), introduced for essay grading in 1997 by T. Landauer and P. Foltz; and (3) e-rater, used by the Educational Testing Service and developed by J. Burstein. PEG grades essays primarily on the basis of writing quality. The underlying theory is that there are intrinsic qualities to a person's writing style called "trins" that need to be measured, analogous to true scores in measurement theory. PEG uses approximations of these variables, "proxes," to measure these traits. Research of more than 30 years shows consistently high correlations between PEG and human raters. IEA identifies which of several calibration documents is most similar to the new document. For essays, the average grade on the most similar calibration documents is assigned as the computer generated score. A similarity score is calculated to the essay column vector score relative to each column of the rubric matrix. As with PEG, high correlation has been found with human raters. E-rater is a "Hybrid Feature Technology" that uses syntactic variety, discourse structure (like PEG), and content analysis (like IEA). Several studies have reported favorably on these approaches. Compared to IEA and e-rater, PEG has the advantage of being conceptually simpler and less taxing on computer resources. All of these systems are proprietary and details of the exact process are not generally available. One should not expect perfect accuracy from any automated scoring approaches, but these technologies might be validation tools with each essay scored by one human and the computer. (SLD)

This document consists of articles 23 through 26 published in the electronic journal "Practical Assessment, Research & Evaluation" in 2001: (23) "Effects of Removing the Time Limit on First and Second Language Intelligence Test Performance" (Jennifer Mullane and Stuart J. McKelvie); (24) "Consequences of (Mis)use of the Texas Assessment of Academic Skills (TAAS) for High-Stakes Decisions: A Commentary on Haney and the Texas Miracle in Education" (J. Thomas Kellow and Victor L. Willson); (25) "Designing Scoring Rubrics for Your Classroom" (Craig A. Mertler); and (26) "An Overview of Three Approaches to Scoring Written Essays by Computer" (Lawrence Rudner and Phil Gagne). (SLD)

This digest presents an overview of accommodations for students of limited English proficiency (LEP) and an overview of inclusion practices on statewide assessments, with emphasis on the accommodation known as linguistic simplification. The inclusion of LEP students in statewide testing programs over the last decade has been uneven. In the mid-1990s, 44 of the 48 states with state assessment programs permitted English language learners (ELLs) to be excused from one or more assessments. A study of state inclusion policies and accommodations for ELLs in the 1998-1999 school year showed that most states were trying various accommodations for ELLs. Appropriate test accommodations level the playing field and help ensure the validity of the test for all students by eliminating irrelevant obstacles. It is important to identify appropriate, valid, and reliable accommodations so that students who receive them do not receive a demonstrable advantage over those who do not. Two recent studies that explored the effects of linguistic simplification as an accommodation illustrate the complexity of the issue. The result of the process of linguistic simplification must be to make the item accessible to ELLs without altering the difficulty of the content. This is not always possible. Further studies are necessary to address the usefulness of linguistic simplification for LEP students taking formal and high stakes assessments. (Contains 11 references.) (SLD)

This digest provides an overview of the concept of alignment and the role it plays in assessment and accountability systems. It also discusses methodological issues affecting the study of alignment and explores the relationship between alignment and test score interpretation. Alignment refers to the degree of match between test content and subject area content identified through state academic standards. It is highly unlikely that a single test can achieve a desirable degree of match, and this fact provides the rationale for using multiple accountability measures and points to the need for studying the degree of match at the test level and the system level. The two overarching dimensions of alignment are content match and depth match. Alignment can best be achieved through sound standards and assessment development activities. The analysis of alignment requires a systematic review of standards and a systematic review of test items/tasks. Alignment is not only a methodological requirement, but also an ethical requirement, as it would be a disservice to students and schools to judge academic achievement based on a poorly aligned system of assessment. The systematic study of alignment is time consuming and can be costly, but the benefits outweigh the costs. (SLD)

This digest makes the case that careful planning of replication can enhance the interpretability of applied research. When results are consistent across several studies, there is a stronger basis for observed relationships than the support that is available within each study by itself, since results that have been replicated are considered more likely to generalize (continue to be observed). It is also possible to compare the studies to identify constructs that interact with, or moderate, relationships. The analysis of the results of several studies should also be addressed. Meta-analysis is an attractive vehicle for combining or synthesizing a series of research replications. This digest discusses pertinent features of meta-analysis, and then provides two examples of cases in which multiple replications of a basic field design have been analyzed using meta-analysis to strengthen the evidence available. The basic designs of the two examples differ markedly. Finally, some design approaches for applied researchers thinking about using replications are discussed. (Contains 12 references.) (SLD)

This digest investigates the methodological practice of relying on self-reported Scholastic Assessment Test (SAT) and grade point average (GPA) scores on educational-psychological research, explores the differential reliability of self-reported SAT and GPA values, and examines trends of deviation in a sample of midwestern teacher education students. Eighty-nine undergraduates at a mid-sized university reported their current cumulative GPAs and the scores they had received on the SAT. Each participant's reported value was categorized as an overestimation, an underestimation, or as accurate in relation to official records. Participants had highly reliable self-reported GPA levels, suggesting that such self-reports are sufficiently adequate for research use. The overall accuracy of self-reported SAT scores was lower than that of GPAs, but the average accuracy was still within reasonable guidelines. The accuracy of self-reported scores was dependent on the independent variable of performance level. The ease of acquiring self-reported SAT and GPA data makes this practice an attractive alternative to the more laborious practice of accessing official student records. However, these results do not support the use of self-reported GPA and SAT scores for policy decisions, especially if the students are able to determine the intent of score collection. (SLD)

This digest provides an introduction to content analysis, a systematic, replicable technique for compressing many words of a text into fewer content categories based on explicit rules of coding. This allows inferences to be made that can then be corroborated using other methods of data collection. In a technical sense, content analysis is not restricted to the domain of text, but in order to allow for replication, the technique can only be applied to data that are durable in nature. Content analysis can be a powerful tool for determining authorship, and it is useful for examining trends and patterns in documents. Content analysis also can provide an empirical basis for monitoring shifts in public opinion. K. Krippendorff has identified six questions regarding content analysis that must be addressed to clarify the data being analyzed and the target of the inferences to be drawn. Content analysis is more than a word-frequency count, although such counts may be a useful starting point. Content analysis is particularly rich because of its reliance on coding and categorizing data. Ways of defining coding units are discussed. There are also issues of reliability and validity that must be addressed in content analysis, but when used properly, it is a powerful data reduction technique. (SLD)

This digest describes a program used by two New Jersey educators to help teachers improve instruction through a deeper understanding of state standards and test specifications. Understanding how standards-based tests are constructed is the first step in being able to use them to guide and improve instruction. This digest recommends, as the route to understanding, a three-part process of delineation, alignment, and calibration for developing an understanding of state assessments and using that understanding to improve instruction. Delineation is the process of identifying all aspects or dimensions of a particular subject domain. It involves using state documents that describe the assessment to analyze its characteristics and components. Examples are given from the New Jersey standards and test specification manuals. During the alignment phase, administrators and teachers work to identify, analyze, generalize, and describe the links between the various elements associated with the subject area previously delineated and the sample questions selected for activities to assess student progress. After completing the delineation, and beginning the alignment processes, calibration begins. Calibration is the act of conducting communications and interactions with teaching staff based on the information from the other two phases. Score reports should be used to improve calibration, and teachers can reflect on their level of calibration accuracy by attempting to predict student results on state assessments. There is a distinct difference between traditional notions of test preparation and aligning and calibrating instruction and assessments with the state assessments, specifications, and standards. Delineation, alignment, and calibration are academic endeavors that demand unending commitment and ample time and resources to avoid falling into mere test preparation. (SLD)

This digest discusses sources of error in testing, several approaches to estimating reliability, and several ways to increase test reliability. Reliability has been defined in different ways by different authors, but the best way to look at reliability may be the extent to which measurements resulting from a test are characteristics of those being measured. In a technical sense, the theoretical definition of reliability is the proportion of score variance that is caused by systematic variation in the population of test takers. Reliability is a joint characteristic of a test and examinee group, not just a characteristic of the test. There are three major sources of error: factors in the test itself, factors in the students taking the test, and scoring factors. It is impossible to calculate a reliability coefficient that conforms to the theoretical definition, but there are several statistics, or coefficients, commonly used to estimate the stability of a set of test scores for a group of examinees. The most commonly used statistics are: (1) test-retest reliability; (2) split-half reliability; (3) measures of internal consistency; and (4) alternate form reliability. When one considers how high reliability should be, the points to consider are the consequences of the test and whether the group used to compute the reported reliability is similar to the group of interest. If the consequences are high, then the internal consistency reliability should be high (at least above 0.90, and preferably above 0.95). Classroom tests seldom need to have exceptionally high reliability coefficients, since teachers have access to other sources of information about students. Developing tests with less error is better than simply documenting the amount of error. It is also commented that, especially in the elementary grades, a poor test result resulting from careless mistakes should not dramatically lower a student's grade for the semester. Teachers should ensure that the semester grade reflects what the student has achieved. (SLD)

This document consists of papers published in the electronic journal "Practical Assessment, Research & Evaluation" during 2000-2001: (1) "Advantages of Hierarchical Linear Modeling" (Jason W. Osborne); (2) "Prediction in Multiple Regression" (Jason W. Osborne); (3) Scoring Rubrics: What, When, and How?" (Barbara M. Moskal); (4) "Organizational Issues Related to Portfolio Assessment Implementation in the Classroom" (Renee Forgette-Giroux and Marielle Simon); (5) "Summarizing Change in Test Scores: Shortcomings of Three Common Methods" (Michael Russell); (6) "Using Expected Growth Size To Summarize Test Score Changes" (Michael Russell); (7) "Using Critical Thinking To Conduct Effective Searches of Online Resources" (Sarah K. Brem and Andrea J. Boyes); (8) "Fundamental Assessment Principles for Teachers and School Administrators" (James H. McMillan); (9) "A Framework for Developing an Effective Instructional Program for Limited English Proficient Students with Limited Formal Schooling" (Angelo Alcala); (10) "Scoring Rubric Development: Validity and Reliability" (Barbara M. Moskal and Jon A. Leyden); (11) "Assessments and Accountability" (Condensed Verstion) (Robert L. Linn); (12) "Self-ReportedGPA and SAT: A Methodological Note (Jerrell C. Cassady); (13) "Using State Standards and Tests To Improve Instruction" (Christopher Tienken and Michael Wilson); (14) "Computing the Expected Proportions of Misclassified Examinees" (Lawrence M. Rudner); (15) "Replication in Field Research" (Willoiam D. Schafer); (16) "Profile Analysis: Multidimensional Scaling Approach" (Cody S. Ding); (17) "An Overview of Content Analysis" (Steve Stemler); (18) "A Rubic for Scoring Postsecondary Academic Skills" (Marielle Simon and Renee Forgette-Giroux); (19) "Conducting Web-based Surveys" (David J. Solomon); (20) "The Stability of Undergraduate Students' Cognitive Test Anxiety Levels" (Jerrell C. Cassady); (21) "Alignment of Standards and Assessments as an Accountability Criterion" (Paul M. La Marca); and (22) "A Confirmatory Analysis of the Wechsler Adult Intelligence Scale-Third Edition: Is the Verbal/Performance Discrepancy Justified?" (Gordon E. Tabu). (SLD)

This document consists of the first 10 articles of volume 8 of the electronic journal "Practical Assessment, Research & Evaluation" published in 2002-2003: (1) "Using Electronic Surveys: Advice from Survey Professionals" (David M. Shannon, Todd E. Johnson, Shelby Searcy, and Alan Lott); (2) "Four Assumptions of Multiple Regression That Researchers Should Always Test" (Jason W. Osbourne and Elaine Waters); (3) "Analyzing Online Discussions: Ethics, Data, and Interpretation" (Sarah K. Brem); (4) "Language Ability Assessment of Spanish-English Bilinguals Future Directions" (Ellen Stubbe Kester and Elizabeth D. Pena); (5) "Male and Female Differences in Self-report Cheating" (James A. Athanasou and Olabisi Olasehinde); (6) "Notes on the Use of Data Transformations" (Jason W. Osbourne); (7) "Linguistic Simplification: A Promising Test Accommodation for LEP Students?" (Charles W. Stansfield); (8) "Evaluating Classroom Communication: In Support of Emergent and Authentic Frameworks in Second Language Assessment" (Miguel Mantero); (9) "The Concept of Formative Assessment" (Carol Boston); and (10) "Vertical Equating for State Assessments: Issues and Solutions in Determination of Adequate Yearly Progress and School Accountability" (Robert W. Lissitz and Huynh Huynh). (SLD)

This document consists of articles 1 through 14 of volume 6 of "Practical Assessment, Research & Evaluation": (1) "Seven Myths about Literacy in the United States" (Jeff McQuillan); (2) "Implementing Performance Assessment in the Classroom" (Amy Brualdi); (3) "Some Evaluation Questions" (William Shadish); (4) "Item Banking" (Lawrence Rudner); (5) "Teacher Comments on Report Cards" (Amy Brualdi); (6) "Classroom Questions" (Amy Brualdi); (7) "Communicating Educational Research Data to General, Nonresearcher Audiences" (Gail S. MacColl and Kathleen D. White); (8) "Helping Children Master the Tricks and Avoid the Traps of Standardized Tests" (Lucy Calkins, Kate Montgomery, and Donna Santman); (9) "Strategies for Improving the Process of Educational Assessment" (M. Kevin Matter); (10)"Performance Assessment Links in Science" (Edys Quellmalz, Patricia Schank, Thomas Hinojosa, and Christine Padilla); (11) "The Nature of Evaluation Part I: Relation to Psychology" (Michael Scriven); (12) "The Nature of Evaluation Part II: Training" (Michael Scriven); (13) "How To Write a Scholarly Report" (Lawrence M. Rudner and William D. Schafer); and (14) "A Process for Evaluating Student Records Management Software" (Lisa Vecchioli). (SLD)

Assessment and accountability have played prominent roles in many reform efforts during the past 50 years. This digest reviews features of assessment and accountability that influence the trustworthiness of information provided and impacts educational practices and student learning. It discusses reform based on assessment and accountability, describes salient features in reform efforts, and makes suggestions for assessment and accountability systems. (SM)

This digest presents 11 basic principles to guide the assessment training and professional development of teachers and administrators. These are: (1) assessment is inherently a process of professional judgment; (2) assessment is based on separate but related principles of measurement evidence and evaluation; (3) assessment decision-making is influenced by a series of tensions; (4) assessment influences student motivation and learning; (5) assessment contains error; (6) good assessment enhances instruction; (7) good assessment is valid; (8) good assessment is fair and ethical; (9) good assessments use multiple methods; (10) good assessment is efficient and feasible; and (11) good assessment appropriately incorporates technology. What is most essential about assessment is understanding how general, fundamental assessment principles and ideas can be used to enhance student learning and teacher effectiveness. (Contains 13 references.) (SLD)

This digest introduces ways of responding to the call for criterion-referenced information using Bayes' Theorem, a method that was coupled with criterion-referenced testing in the early 1970s (see R. Hambleton and M. Novick, 1973). To illustrate Bayes' Theorem, an example is given in which the goal is to classify an examinee as being a master or nonmaster. Responses to previously piloted items are used to determine the probabilities of mastery and nonmastery, and then the examinee is classified based on those probabilities. The Bayesian network defined in this example is a simple diverging graph in which the master/nonmaster state is causally connected to the set of item responses. When applied to decision support systems and other expert systems, Bayesian networks are typically much more complex, but the computations for basic applications, as illustrated, are quite simple. The framework outlined is applicable to a range of settings and to multidimensional items and tests. Bayesian networks have been used as the basis for computer adaptive tests. The framework can be embedded in an intelligent tutoring system to determine mastery after each instructional unit. One can experiment with simple Bayesian networks using a variety of free, readily available software packages. (SLD)

This digest introduces hierarchical data structure, describes how hierarchical models work, and presents three approaches to analyzing hierarchical data. Hierarchical, or nested data, present several problems for analysis. People or creatures that exist within hierarchies tend to be more similar to each other than people randomly sampled from the entire population; for this reason, observations based on these individuals are not fully independent. Hierarchical linear modeling can address problems caused by this situation. The basic concept is similar to that of ordinary least squares regression. On a base level (usually the individual), an outcome variable is predicted as a function of a linear combination of one or more level 1 variables. On subsequent levels, the level 1 slope (or slopes) and intercept become dependent variables being predicted from level 2 variables. Through this process, the effects of level 1 variables on the outcome are accurately modeled, and the effects of level 2 variables are also modeled on the outcome. Cross-level interactions can be modeled. Data from the National Education Longitudinal Survey of 1988 are used to illustrate disaggregated, aggregated, and hierarchical analyses. These analyses reveal the need for multilevel analysis of multilevel data. (SLD)

This digest complements guidelines addressing the mechanics of online searching by considering how treating information searches as exercises in critical thinking can improve our use of online resources. It addresses the use and application of metacognition, hypothesis testing, and augmentation. Improving metacognition means improving the ability to monitor what we know and how we know it. Some ways to accomplish this include putting the project aside for a brief time, talking it out, and developing content knowledge. When these strategies have been exhausted, it is useful to find someone who may know more about it. Searching the literature should be an exercise in hypothesis testing, an exercise that can be improved by actively pursuing alternatives and taking an evaluative position. Once one adopts an evaluativist position, augmentation strategies help carry out the evaluation. Considering the structure and reliability of a source and remembering that even reputable sources are fallible are important. Using systematic analysis for a comprehensive, though time-consuming evaluation, is important. Heuristics are useful when it is necessary to make quick decisions, when there is not enough information for systematic analysis, or to complement systematic approaches. (Contains 11 references.) (SLD)

This Digest introduces the advantages and disadvantages of three commonly used methods of reporting test score changes: (1) change in percentile rank; (2) scale or raw score change; and (3) percent change. The change in percentile rank method focuses on the increase or decrease of the mean percentile ranking for a group of students. This method has two main problems. The first is that calculating the mean percentile rank based on an individual's percentile ranks can provide an inaccurate estimate of a group's mean performance. The second is that, because of unequal intervals separating percentile ranks, changes in percentile ranks represent different amounts of growth at each point on the scale. A second method is scale or raw score change. The main drawback to this methods is that when raw scores are used to determine change, it is difficult to compare change across tests with different score ranges. A third approach, that of reporting percent change, causes further distortion. resulting in a statistic that is difficult to interpret and misleading. All of these methods should be avoided when summarizing change in test scores. A separate Digest suggests better ways to summarize changes. (SLD)

This Digest provides a process for developing scoring rubrics. The first step is to identify clearly the qualities that need to be displayed in a student's work to demonstrate proficient performance. These qualities will be the top level or levels of scoring criteria for the scoring rubric. Then the decision must be made whether the information desired from the evaluation can best be acquired through an analytic or holistic scoring rubric. The evaluator may then to choose to define the criteria for the lowest level of performance, and if desired, comparisons can be made between the criteria for each existing score point. Each score category should be defined using descriptions of the work rather than judgments about the work. To determine whether a scoring rubric provides adequate descriptions, another teacher may be asked to use the scoring rubric to evaluate a subset of student responses. Differences between the scores assigned by the rubric developer and a second scorer will show how the rubric may be clarified further. There are many resources available to teachers who wish to use scoring rubrics in their classrooms. Some of these are listed, including the ERIC Clearinghouse on Assessment and Evaluation. (Contains 12 references.) (SLD)

This Digest describes different types of scoring rubrics and explains why they are useful. Scoring rubrics are descriptive scoring schemes that are developed by teachers or other evaluators to guide the analysis of the products or processes of students' efforts. Scoring rubrics are generally used when a judgment of quality is required, and they may be used to evaluate a broad range of activities. The evaluation of writing is a common use of scoring rubrics. Several types of scoring rubrics are available. The first distinction is between analytic and holistic scoring rubrics. An analytic scoring rubric allows for the separate evaluation of each factor considered in the evaluation. A holistic scoring rubric may be preferable when there is an overlap between the criteria set for the evaluation of the different factors. Rubrics are also divided into general versus task-specific rubrics, and may be designed to contain both general and task-specific components. (Contains 12 references.) (SLD)

A special subgroup of Limited English Speaking students is often referred to as students with limited formal schooling (LFS) or "preliterates" because they have not yet had the opportunity to learn to read. This digest explores important aspects of the LFS student population, defining LFS students and discussing their impact on schools, individualized language development plans, classroom instruction, and the assessment of the LFS student. The LFS student is usually older, aged 12 to 21, and in most cases the LFS student possesses less than 2 years of formal education. The impact of LFS students on a school can be significant because they require special services and teacher professional development. It is recommended that an individualized language development plan be prepared for LFS students to ensure that their special needs are met. The LFS student requires a warm and supportive classroom where the goal is not only to teach literacy skills in the native language, but also to teach meaningful communication in and functional use of the English language. Assessing the progress of such students requires the use of a variety of assessment tools. (Contains 16 references.) (SLD)

An earlier Digest described the shortcomings of three methods commonly used to summarize changes in test scores. This Digest describes two less commonly used approaches for examining changes in test scores, those of Standardized Growth Estimates and Effect Sizes. Aspects of these two approaches are combined and applied to the Iowa Test of Basic Skills to demonstrate the usefulness of a third method, termed Expected Growth Size, to examine change in test scores. An expected growth size is more difficult to calculate than the other methods, but it offers three advantages. By expressing change in relation to the standard deviation, growth rates for different tests and different grade levels can be compared directly. Once expected growth sizes are calculated for a given test, they can be transformed easily to more common measurement scales. And once expected growth sizes are transformed to a Normal Curve Equivalent scale, changes in an individual's or a group's mean score can be reported in relation to expected growth. How to calculate expected growth size is illustrated. (SLD)