Search

The usefulness of evaluation is critical. Evaluation use occurs when, from its results or process, decisions are made about the program, it changes people's mindsets, or persuasive or legitimation actions happen (instrumental, conceptual, and symbolic uses respectively). Few quantitative evaluation use studies have been conducted in recent years. Most existing evaluation use scales are outdated due to theoretical advances and were project or program specific, not designed for evaluation systems. This study was designed to develop the Evaluation Use Scale for Evaluation Systems (EUS-ES) and gather initial validity evidence. We collected and analyzed data from Brazilian professors and program coordinators who were consultants of the Brazilian Graduate Evaluation System. Validity evidence was collected based on instrument content and internal structure. CFA results supported a three-factor-correlated model with adequate score reliability estimates. The EUS-ES is valuable for diagnostic purposes to identify low/high evaluation uses in an evaluation system setting. [ABSTRACT FROM AUTHOR]

Full-information item analysis provides item developers and reviewers comprehensive empirical evidence of item quality, including option response frequency, point-biserial index (PBI) for distractors, mean-scores of respondents selecting each option, and option trace lines. The multi-serial index (MSI) is introduced as a more informative item-total correlation, accounting for variable distractor performance. The overall item PBI is empirically compared to the MSI. For items from an operational mathematics and reading test, poorly performing distractors are systematically removed to recompute the MSI, indicating improvements in item quality. Case studies for specific items with different characteristics are described to illustrate a variety of outcomes, focused on improving item discrimination. Full-information item analyses are presented for each case study item, providing clear examples of interpretation and use of item analyses. A summary of recommendations for item analysts is provided. [ABSTRACT FROM AUTHOR]

Researchers often use self-reported instruments to collect data from students when investigating the causes and effects of bullying. When completing the instrument, students may skip items despite anonymous data collection. To interpret data accurately, researchers must identify the causes of nonresponse. This study examined the impact of item (types of bullying and position) and student (sex and grade level) factors on nonresponse in bullying items. The sample included 10,681 students from grades 8, 9, and 11 who skipped at least one bullying item in the Minnesota Student Survey. Results showed that students were more likely to skip the items presented toward the end of the survey. The type of bullying had no impact on nonresponse. Male students and students at higher grade levels were more likely to skip the items. To alleviate the nonresponse issue, researchers are recommended to consider both survey length and student factors when administering bullying instruments. [ABSTRACT FROM AUTHOR]

The evidence is mounting regarding the guidance to employ more three-option multiple-choice items. From theoretical analyses, empirical results, and practical considerations, such items are of equal or higher quality than four- or five-option items, and more items can be administered to improve content coverage. This study looks at 58 tests, including state achievement, college readiness, and credentialing tests. The evidence here supports previous assertions. The article also clarifies distractor functioning criteria and offers a typology of items via distractor functioning. [ABSTRACT FROM AUTHOR]

Spanish speakers in the United States are a steadily increasing population, up by 233% since 1980. Given the growing population of dual language learners (DLLs) and the large numbers of Spanish-speaking children enrolled in pre-kindergarten programs, addressing the educational needs of preschool-aged DLLs has become a national imperative. Specifically, the intersection of this growing population and the dearth of appropriate assessment tools to evaluate DLLs early language and literacy skills creates a need for assessments that accurately measure preschool performance. This manuscript reports on the iterative design process of a measure of Spanish phonological awareness for preschool-aged DLLs: Spanish Individual Growth and Development Indicators (S-IGDI) Primeros Sonidos. We employed measure design framework to develop the measure and tested item function within a study of 970, 4-5 year old DLLs. Results, including item level analyses and evidence regarding construct and criterion validity are reported. [ABSTRACT FROM AUTHOR]

Subscores are of increasing interest in educational and psychological testing due to their diagnostic function for evaluating examinees' strengths and weaknesses within particular domains of knowledge. Previous studies about the utility of subscores have mostly focused on the overall reliability of individual subscores and ignored the fact that subscores should be distinct and have added value over the total score. This study introduces a profile reliability approach that partitions the overall subscore reliability into within-person and between-person subscore reliability. The estimation of between-person reliability and within-person reliability coefficients is demonstrated using subscores from number-correct scoring, unidimensional and multidimensional item response theory scoring, and augmented scoring approaches via a simulation study and a real data study. The effects of various testing conditions, such as subtest length, correlations among subscores, and the number of subtests, are examined. Results indicate that there is a substantial trade-off between within-person and between-person reliability of subscores. Profile reliability coefficients can be useful in determining the extent to which subscores provide distinct and reliable information under various testing conditions. [ABSTRACT FROM AUTHOR]

Because of the unique nature of the students eligible for alternate assessments based on modified academic achievement standards, their varied access to the general education curriculum, and their unique learning needs, innovative psychometric thinking and practice is needed to assure high technical quality of alternate assessments. Indeed, we at least must marshal state-of-the-art procedures to secure strong psychometric evidence to support appropriate and meaningful design and use of these important assessments. The authors contributing work to this special issue, Alternate Assessments Based on Modified Academic Achievement Standards, address important issues and provide guidance to policymakers, test developers, and educators. They also each raise important technical quality issues. This article offers a brief review of such psychometric considerations, in light of the work and comments of the special issue authors. [ABSTRACT FROM AUTHOR]

The primary goal of the Words Work (WW) early literacy initiative was to prepare diverse students to become successful learners in school. Four groups of students were followed over 6 years: students from traditional Head Start (HS) centers, students from augmented HS centers (WW students), students eligible for HS but not enrolled (waitlist or WL), and a random sample of non-Head Start students (NHS). Students received school readiness instruction at approximately 4 years of age, and their performance on standardized tests was followed from 1st grade through 5th grade. WW students significantly outscored HS, WL, and NHS students of similar backgrounds in reading and outscored WL and NHS students in mathematics on 2nd-grade standardized assessments. In addition, WW students were able to maintain their lead through 5th grade in both reading and mathematics. Based on these results, WW is an effective HS enhancement for minimizing the achievement gap often observed between students who may be at risk for academic difficulty and students who are at low risk. [ABSTRACT FROM AUTHOR]

Evaluates the relationship between assessment practices and achievement and the mediating roles of student self-efficacy and effort. Framework proposed by Brookhart (1997); Use of the U.S. portion of the Third International Math and Science Study to estimate the relationships; Student and classroom level characteristics that were important explanatory variables regarding variation in mathematics achievement.

A taxonomy of 31 multiple-choice item-writing guidelines was validated through a logical process that included two sources of evidence: the consensus achieved from reviewing what was found in 27 textbooks on educational testing and the results of 27 research studies and reviews published since 1990. This taxonomy is mainly intended for classroom assessment. Because textbooks have potential to educate teachers and future teachers, textbook writers are encouraged to consider these findings in future editions of their textbooks. This taxonomy may also have usefulness for developing test items for large-scale assessments. Finally, research on multiple-choice item writing is discussed both from substantive and methodological viewpoints. [ABSTRACT FROM AUTHOR]