Search

Introduction: Individuals living with Crohn's disease (CD) experience burdensome symptoms. As such, it is important to measure CD symptom severity in clinical research. The goal of this study was to evaluate the content validity, psychometric performance, and score interpretability of a new patient-reported instrument, the Crohn's Symptom Severity (CSS) questionnaire, among adolescents and adults with moderately to severely active CD. Methods: Cognitive debriefing interviews (N = 30; n = 20 adults, n = 10 adolescents) were conducted to evaluate the content validity of the CSS. Additionally, the CSS scores were evaluated for reliability and validity using data from a phase 3 randomized clinical trial of risankizumab (NCT03105128; N = 850). Meaningful within-patient change (MWPC) thresholds were estimated using anchor-based methods. Results: All interview participants (n = 30/30, 100.00%) reported the CSS was easy to complete and most participants (n = 28/29, 96.55%) reported that the CSS was relevant to their experience of CD. Among the clinical trial subjects (N = 850) the following was found for the CSS: mostly acceptable item–total correlations (0.26–0.79); weak to moderate inter-item correlations (r = 0.07–0.57), good internal consistency (Cronbach's α = 0.76–0.87); intraclass correlation coefficients ranged from 0.48 to 0.70, not consistently exceeding the acceptable range for test–retest reliability (0.70); acceptable convergent validity and known-groups results; and demonstrated sensitivity to change. Analyses supported an MWPC estimate of 6–11 points. Conclusions: This study supports use of the CSS for measuring CD symptoms and sleep impact among adolescents and adults aged 16 and older with moderately to severely active CD in clinical research. Trial Registration: NCT03105128 (registration date 4 April 2017). [ABSTRACT FROM AUTHOR]

Diagnostic classification models (DCMs) are psychometric models that yield probabilistic classifications of respondents according to a set of discrete latent variables. The current study examines the recently introduced one-parameter log-linear cognitive diagnosismodel (1-PLCDM), which has increased interpretability compared with general DCMs due to useful measurement properties like sum score sufficiency and invariance properties. We demonstrate its equivalence with the Latent Class/Rasch Model and discuss interpretational consequences. The model is further examined in a DCM framework. We demonstrate the sum score sufficiency property and we derive an expression for the cut score for mastery classification. It is shown by means of a simulation study that the 1-PLCDM is fairly robust to model constraint violations in terms of classification accuracy and reliability. This robustness in combination with useful measurement properties and ease of interpretation can make the model attractive for stakeholders to apply in various assessment settings. [ABSTRACT FROM AUTHOR]

Diagnostic classification models (DCMs) are psychometric models that yield probabilistic classifications of respondents according to a set of discrete latent variables. The current study examines the recently introduced one-parameter log-linear cognitive diagnosis model (1-PLCDM), which has increased interpretability compared with general DCMs due to useful measurement properties like sum score sufficiency and invariance properties. We demonstrate its equivalence with the Latent Class/Rasch Model and discuss interpretational consequences. The model is further examined in a DCM framework. We demonstrate the sum score sufficiency property and we derive an expression for the cut score for mastery classification. It is shown by means of a simulation study that the 1-PLCDM is fairly robust to model constraint violations in terms of classification accuracy and reliability. This robustness in combination with useful measurement properties and ease of interpretation can make the model attractive for stakeholders to apply in various assessment settings.

Research in validity suggests that stakeholders' interpretation and use of test results should be an aspect of validity. Claims about the meaningfulness of test score interpretations and consequences of test use should be backed by evidence that stakeholders understand the definition of the construct assessed and the score report information. The current study explored stakeholders' uses and interpretations of the score report of a workplace English language proficiency test, the TOEIC® Listening and Reading (TOEIC L&R) test. Online surveys were administered to TOEIC L&R test takers and institutional and corporate score users in Taiwan to collect data about their uses and interpretations of the test score report. Eleven survey respondents participated in follow‐up interviews to further elaborate on their uses of the different score reporting information within the stakeholders' respective contexts. Results indicated that the participants used the TOEIC L&R test scores largely as intended by the test developer although some elements of the score report appeared to be less useful and could be confusing for stakeholders. Findings from this study highlight the importance of providing score reporting information with clarity and ease to enhance appropriate use and interpretation. Report Number: RR‐23‐10 [ABSTRACT FROM AUTHOR]

In between-item multidimensional item response models, it is often desirable to compare individual latent trait estimates across dimensions. These comparisons are only justified if the model dimensions are scaled relative to each other. Traditionally, this scaling is done using approaches such as standardization-fixing the latent mean and standard deviation to 0 and 1 for all dimensions. However, approaches such as standardization do not guarantee that Rasch model properties hold across dimensions. Specifically, for between-item multidimensional Rasch family models, the unique ordering of items holds within dimensions, but not across dimensions. Previously, Feuerstahler and Wilson described the concept of scale alignment, which aims to enforce the unique ordering of items across dimensions by linearly transforming item parameters within dimensions. In this article, we extend the concept of scale alignment to the between-item multidimensional partial credit model and to models fit using incomplete data. We illustrate this method in the context of the Kindergarten Individual Development Survey (KIDS), a multidimensional survey of kindergarten readiness used in the state of Illinois. We also present simulation results that demonstrate the effectiveness of scale alignment in the context of polytomous item response models and missing data.

Classroom language assessment aims to gather various types of information related to student language learning and use and use it to inform a range of formative-summative decisions, including how to provide feedback to help students improve their learning efficacy, what teaching adjustments are needed, and whether students have demonstrated their learning attainment defined by a given syllabus (Turner, 2012). This article presents five golden rules based on the contributions of JD Brown. These golden rules address a need for a practical and accessible assessment approach that helps language teachers effectively assess students' learning achievement in the language classroom. This article is, therefore, written for language teachers and pre-service TESOL (Teaching English to Speakers of Other Languages) students. The five golden rules presented in this article are: (1) choose the suitable types of language assessment tasks; (2) recognise the two types of assessment interpretations; (3) consider and apply basic classical test theory for analysis of the assessment of learning; (4) develop well-defined assessment criteria and rubrics for judging students' performance; and (5) use assessment of language learning as a source of feedback to improve assessment tasks and student learning. [ABSTRACT FROM AUTHOR]

We address two persistent challenges in large‐scale assessments of the Next Generation Science Standards: (a) the validity of score interpretations that target the standards broadly and (b) how to structure claims for assessments of this complex domain. The NGSS pose a particular challenge for specifying claims about students that evidence from summative assessments can support. As a solution, we propose tiered claims, which explicitly distinguish between claims about what students have done or can do on test items—which are typically easier to support under current test designs—and claims about what students could do in the broader domain of performances described by the standards, for which novel evidence is likely required. We discuss the positive implications of tiered claims for test construction, validation, and reporting of results. [ABSTRACT FROM AUTHOR]

Understanding the response process used by test takers when responding to multiple-choice questions (MCQs) is particularly important in evaluating the validity of score interpretations. Previous authors have recommended eye-tracking technology as a useful approach for collecting data on the processes test taker's use to respond to test questions. This study proposes a new method for evaluating alternative score interpretations by using eye-tracking data and machine learning. We collect eye-tracking data from 26 students responding to clinical MCQs. Analysis is performed by providing 119 eye-tracking features as input for a machine-learning model aiming to classify correct and incorrect responses. The predictive power of various combinations of features within the model is evaluated to understand how different feature interactions contribute to the predictions. The emerging eye-movement patterns indicate that incorrect responses are associated with working from the options to the stem. By contrast, correct responses are associated with working from the stem to the options, spending more time on reading the problem carefully, and a more decisive selection of a response option. The results suggest that the behaviours associated with correct responses are aligned with the real-world model used for score interpretation, while those associated with incorrect responses are not. To the best of our knowledge, this is the first study to perform data-driven, machine-learning experiments with eye-tracking data for the purpose of evaluating score interpretation validity. [ABSTRACT FROM AUTHOR]

In this research report, we describe a study to map the scores of the TOEFL® Essentials™ test to the Canadian Language Benchmarks (CLB). The TOEFL Essentials test is a four‐skills assessment of foundational English language skills and communication abilities in academic and general (daily life) contexts. At the time of writing this report, the test was the most recent addition to the TOEFL® Family of Assessments. TOEFL Essentials test scores are intended to provide academic programs and other users with reliable information regarding the test taker's ability to understand and use English. Mapping of scores to widely used language frameworks such as the CLB provides additional support for interpreting test results and for making inferences regarding test‐taker abilities. The score mapping process consisted of the following steps, as recommended in the literature: (a) establishing construct congruence between the test content and the performance descriptors of the CLB; (b) establishing recommended minimum test scores (cut scores) required to classify language learners into CLB levels, based on the judgments of local experts; and (c) providing evidence of procedural, internal, and external validation of the recommended cut scores. [ABSTRACT FROM AUTHOR]

Large-scale language testing uses statistical information to account for the quality of an assessment system. In this reflection article, I explain how basic statistics can be used meaningfully in the context of classroom language assessment. The paper explores a series of statistical calculations that can be used to examine test scores and assessment decisions in the language classroom. Therefore, interpretations for criterion-referenced assessment underlie the paper. Finally, I discuss limitations and include recommendations for teachers to use statistics.

The redesigned four‐skills TOEIC Bridge® tests were designed to measure the listening, reading, speaking, and writing proficiency of beginning to low–intermediate English learners in the context of everyday life. In this paper, I describe two studies that were conducted to investigate claims about the meaningfulness of redesigned TOEIC Bridge test scores by comparing them to self‐assessments of listening, reading, speaking, and writing proficiency. In Study 1, test takers who participated in the redesigned TOEIC Bridge field test in Japan and Taiwan completed an online self‐assessment survey of their speaking and writing proficiency. In Study 2, test takers who participated in pretesting for the redesigned TOEIC Bridge Listening and Reading test in Japan and Taiwan completed a paper‐based self‐assessment survey of their listening and reading proficiency. Both studies were designed with the same research questions in mind in order to evaluate claims about the meaning of redesigned TOEIC Bridge test scores and to enhance the interpretation of test scores by relating them to test takers' self‐assessed ability to perform a variety of related tasks. The results indicated that self‐assessments had medium correlations with redesigned TOEIC Bridge Listening test scores (r =.55), Reading test scores (r =.54), Speaking test scores (r =.51), and Writing test scores (r =.46). These results compare favorably with the results of similar studies of the relationship between test scores and self‐assessments of language proficiency and provide evidence to support claims about the meaningfulness of TOEIC Bridge test scores. In addition, the pattern of results across TOEIC Bridge test proficiency levels and can‐do statements taken from language proficiency standards suggests that the tests meaningfully differentiate test‐takers' proficiency levels, an important consideration of test design. [ABSTRACT FROM AUTHOR]

Large-scale language testing uses statistical information to account for the quality of an assessment system. In this reflection article, I explain how basic statistics can be used meaningfully in the context of classroom language assessment. The paper explores a series of statistical calculations that can be used to examine test scores and assessment decisions in the language classroom. Therefore, interpretations for criterion-referenced assessment underlie the paper. Finally, I discuss limitations and include recommendations for teachers to use statistics. La evaluación de lenguas estandarizada utiliza datos estadísticos para evaluar la calidad de un sistema de evaluación. En este artículo de reflexión, explico cómo se puede usar la estadística de manera significativa en la evaluación en el aula de idiomas. El artículo explora una serie de cálculos estadísticos que pueden usarse para estudiar las notas y decisiones provenientes de instrumentos de evaluación en la clase de idiomas. Por ello, la evaluación criterial es la que utilizo para las interpretaciones en el artículo. Finalmente, discuto unas limitaciones, y hago recomendaciones para que los docentes de idiomas puedan usar la estadística. [ABSTRACT FROM AUTHOR]

The past decade has seen an emerging interest in mapping (aligning or linking) test scores to language proficiency levels of external performance scales or frameworks, such as the Common European Framework of Reference (CEFR), as well as locally developed frameworks, such as China's Standards of English Language Ability (CSE). Such alignment is ultimately a claim about the interpretation of test scores in relation to external levels of language proficiency. To support such a claim, established procedures should be carefully implemented and multiple sources of evidence should be collected. In this research report, we demonstrate the application of a series of steps in building an argument for aligning the scores of the TOEFL iBT® test, an international, large‐scale language proficiency test of English as a foreign language (EFL), to the levels of the CSE. The alignment process comprised the following steps: (a) establishing construct congruence between the TOEFL iBT test and the CSE; (b) establishing recommended minimum test scores (cut scores), set by local experts, to classify language learners into the local proficiency levels; (c) collection of scores by test takers (N = 1,326) and evaluations of the test takers' proficiency levels by their teachers, based on the local framework; and (d) consideration of the results of other alignment studies in the local context as well as the link between the CEFR and the CSE levels. We conclude with a discussion of the contextual issues that should be considered when interpreting test scores in relation to external proficiency levels. These contextual issues are important considerations because they have the potential to impact score‐based decisions on individuals and institutions. We also discuss the implications for similar alignment research. Report Number: TOEFL–RR‐89 Report Number: ETS Research Report No. RR–19‐44 [ABSTRACT FROM AUTHOR]

One of the substantive changes in the 2014 Standards for Educational and Psychological Testing was the elevation of fairness in testing as a foundational element of practice in addition to validity and reliability. Previous research indicates that testing practices often do not align with professional standards and guidelines. Therefore, to raise awareness of fairness concepts and principles from the 2014 Standards, this study aligned those standards with fairness practices, as documented in test manuals and on websites of 18 intelligence and achievement tests from different test publishers. A content analysis indicated that just under half of the fairness standards are frequently or occasionally practiced and those occurrences differed somewhat across tests but did not differ between intelligence and achievement tests or across publishers. To inform and encourage improvements in the future practice of the fairness standards, an evaluative framework along with example practices and related methodological scholarship is discussed. [ABSTRACT FROM AUTHOR]

Although much research has been conducted on the psychometric properties of cognitive diagnostic models, they are only recently being used in operational settings to provide results to examinees and other stakeholders. Using this newer class of models in practice comes with a fresh challenge for diagnostic assessment developers: effectively reporting results and supporting end users to accurately interpret results. Achieving the goal of communicating results in a way that leads users of the assessment to make accurate interpretations requires a prerequisite step that cannot be taken for granted. The assessment developers must first accurately interpret results from a psychometric, or measurement, standpoint. Through this article, we seek to begin a discussion about reasonable interpretations of the results that classification‐based models provide about examinees. Interpretations from published research and ongoing practice show different—and sometimes conflicting—ways to interpret these results. This article seeks to formalize a comparison, critique, and discussion among the interpretations. Before beginning this discussion, we first present background on the results provided by classification‐based models regarding the examinees. We then structure our discussion around key questions an assessment development team needs to answer themselves prior to constructing reports and interpretative guides for end users of the assessment. [ABSTRACT FROM AUTHOR]

Clinical cognitive ability assessment—and its corollary, score interpretation—are in a state of disarray. Many current instruments are designed to provide a bevy of scores to appeal to a variety of school psychologists. These scores are not all grounded in the attribute's theory or developed from sound measurement or psychometric theory. Thus, for a given instrument, there can be substantial variation between school psychologists when interpreting scores from the same instrument. This is contrary to the very purpose of psychological assessment. As a contrast, we provide a sketch of theoretically driven test development and score interpretation. In addition, we provide examples of how this could be implemented using two theories of intelligence (Spearman's two-factor and Cattell and Horn's Gf-Gc) and measurement theory about the nature of psychological test scores. While different from what is often implemented by school psychologists, it is consistent with the guiding principles of evidence-based psychological assessment. [ABSTRACT FROM AUTHOR]

In item response theory (IRT), item response probabilities are a function of item characteristics and latent trait scores. Within an IRT framework, trait score misestimation results from (a) random error, (b) the trait score estimation method, (c) errors in item parameter estimation, and (d) model misspecification. This study investigated the relative effects of these error sources on the bias and confidence interval coverage rates for trait scores. Our results showed that overall, bias values were close to 0, and coverage rates were fairly accurate for central trait scores and trait estimation methods that did not use a strong Bayesian prior. However, certain types of model misspecifications were found to produce severely biased trait estimates with poor coverage rates, especially at extremes of the latent trait continuum. It is demonstrated that biased trait estimates result from estimated item response functions (IRFs) that exhibit systematic conditional bias, and that these conditionally biased IRFs may not be detected by model or item fit indices. One consequence of these results is that certain types of model misspecifications can lead to estimated trait scores that are nonlinearly related to the data-generating latent trait. Implications for item and trait score estimation and interpretation are discussed. [ABSTRACT FROM AUTHOR]

This study provides a comprehensive evaluation of the effects of differential item functioning (DIF) on proficiency classification. Using Monte Carlo simulation, item- and test-level DIF magnitudes were varied systematically to investigate their impact on proficiency classification at multiple decision points. Findings from this study clearly show that the presence of DIF affects proficiency classification not by lowering the overall correct classification rates but by affecting classification error rates differently for reference and focal group members. The study also reveals that multiple items with low levels of DIF can be particularly problematic. They can do similar damage to proficiency classification as high-level DIF items with the same cumulative magnitudes but are much harder to detect with current DIF and differential bundle functioning (DBF) techniques. Finally, how DIF affects proficiency classification errors at multiple cut scores is fully described and discussed. [ABSTRACT FROM AUTHOR]

Purpose: Clinical outcome assessments (COAs) require evidence not only of reliability, validity, and ability to detect change, but also a definition of what constitutes a meaningful change on the instrument. The responder definition specifies the amount of change on the COA that may be interpreted as a treatment benefit and is critical for interpreting what constitutes a meaningful change on the COA scores. However, the literature that describes methods for developing and applying responder definitions can be difficult to navigate. Clear and concise guidelines regarding which methods to apply under what circumstances and how to interpret the results are lacking. This article provides a guide to the variety of available methods and issues that should be considered when establishing responder definitions for interpreting meaningful changes in COA scores.Methods: An overview is provided for selecting anchors, developing study designs, planning psychometric analyses, using psychometric results to set responder thresholds, and applying responder thresholds in demonstrating treatment efficacy.Results: There are a variety of anchor-based methods for consideration, but they all rely on a preference for strongly related and easily interpretable anchors. The benefits of applying multiple anchors and multiple analytic methods are discussed. The process of triangulation can synthesize results across multiple sources to gain confidence in a proposed responder definition. Though a link to meaningfulness from the patient's perspective is absent, distribution-based methods provide lower bound estimates of score precision and have a role in triangulation. Responder definitions are typically required within regulatory review, but their application may differ across clinical trial programs.Conclusions: By careful planning of anchor selection, study design, and psychometric methods, COA researchers can establish defensible responder thresholds that ultimately aid patients and clinicians in making informed treatment decisions. [ABSTRACT FROM AUTHOR]

Introduction The International Index of Erectile Function (IIEF) is the gold standard validated instrument for defining erectile function (EF) and its response to treatment. The EF domain (EFD) contains six questions and is a sensitive and specific measurement of treatment-related changes in EF. The EFD score has been widely used as a primary assessment end point for clinical trials of EF recovery after radical prostatectomy (RP). Various EFD scores have been used to define functional erections. Recently, an EFD score of at least 22 has been used as a threshold in major post-RP penile rehabilitation studies. Aim To define the EFD score that optimally defines “functional” erections after RP. Methods We assessed men 24 months after RP using the IIEF and specifically analyzed the scores of the EFD and intercourse satisfaction domain (ISD). Main Outcome Measures We used two questions on satisfaction (score = 0–5) and enjoyment (score = 0–5) from the ISD to classify IS (score = 0–10). We tested the following intercourse satisfaction classifications: ISD score equal to 10, ISD score of at least 8, and a score of at least 4 for the ISD questions on satisfaction and enjoyment. We used the classification that produced the largest area under the curve (AUC) using a receiver operating characteristic (ROC) curve. Then, we used a three-step process to determine the optimal EFD score cutoff using sensitivity and specificity analysis. Results One hundred seventy-eight men had an average age at RP of 58 ± 7 years and a 24-month EFD score of 20 ± 9. Sixty-four percent had complete nerve-sparing surgery, 35% had partial nerve-sparing surgery, and 1% had the nerves fully resected. Thirty-three percent had laparoscopic RP and 67% had open RP. The ROC curves produced AUCs of 0.80 (ISD score = 10), 0.85 (ISD score ≥ 8), and 0.86 (ISD scores for satisfaction and enjoyment ≥ 4; P < .001 for all comparisons). Using the IS criterion of ISD scores for satisfaction and enjoyment of at least 4 (largest AUC), the sensitivity and specificity values were 0.89 and 0.66 for an ESD score equal to 22, 0.78 and 0.71 for a score equal to 23, 0.78 and 0.80 for a score equal to 24, 0.77 and 0.82 for a score equal to 25, and 0.73 and 0.85 for a score equal to 26. The scores of 24 and 25 met the criteria outlined in the first two steps of analysis. The score of 24 was selected as the cutoff using face valid judgment and the previous literature. Conclusion These data support an EFD score of 24 as a valid cutoff defining “functional” erection in men with erectile dysfunction after RP. These data are important for clinicians in counseling patients and to researchers to define inclusion criteria and treatment end points for trials of erectile dysfunction after RP. Terrier JE, Muhall JP, Nelson CJ. Exploring the Optimal Erectile Function Domain Score Cutoff That Defines Sexual Satisfaction After Radical Prostatectomy. J Sex Med 2017;14:804–809. [ABSTRACT FROM AUTHOR]

The aim of this study was to develop, for the benefit of both test takers and test score users, enhanced TOEFL ITP® test score reports that go beyond the simple numerical scores that are currently reported. To do so, we applied traditional scale anchoring (proficiency scaling) to item difficulty data in order to develop performance descriptors for multiple levels of each of the three sections of the TOEFL ITP. A (novel) constraint was that these levels should correspond to those established in an earlier study that mapped (i.e., aligned) TOEFL ITP scores to a widely accepted framework for describing language proficiency - the Common European Framework of Reference (CEFR). The data used in the present study came from administrations of five current operational forms of the recently revised TOEFL ITP test. The outcome of the effort is a set of performance descriptors for each of several levels of TOEFL ITP scores for each of the three sections of the test. The contribution, we believe, constitutes (1) an enhancement of the interpretation of scores for one widely used assessment of English language proficiency and (2) a modest contribution to the literature on developing proficiency descriptors - an approach that combines elements of both scale anchoring and test score mapping. [ABSTRACT FROM AUTHOR]

Introduction This report describes a post hoc analysis of data from a randomized, double-blinded, placebo-controlled, flexible-dose, sildenafil trial in men with erectile dysfunction. Aims To simplify interpretation of erectile function (EF) domain scores of the International Index of Erectile Function (IIEF). Methods Men at least 18 years old with erectile dysfunction were randomized to receive sildenafil or placebo for 12 weeks. Men taking nitrates or nitric oxide donors were excluded. Responses for each IIEF EF domain question (questions 1–5 and 15) were combined into two broad categories (“success” for responses of the two most favorable categories of a question and “no success” for other responses). Each question was expressed in a logistic regression model (sildenafil and placebo groups combined) as a function of overall EF domain score. Main Outcome Measures IIEF EF domain score and items. Results A four-point increase in the IIEF EF domain score was associated with an odds ratio of success of 6.1 for getting an erection, 29.2 for having a firm erection, 10.0 for able to penetrate,12.8 for maintaining erection, 4.0 for maintaining erection to completion, and 3.7 for erection confidence. An EF domain score of 22 was associated with a probability of success of 81% for getting an erection, 86% for having a firm erection, 89% for able to penetrate, 67% for maintaining an erection, 70% for maintaining an erection to completion, and 32% for erection confidence. For an EF domain score of 16, the corresponding probabilities of success were 22%, 4%, 20%, 4%, 22%, and 6%, respectively. Conclusion These results provide stakeholders with a simplified and meaningful interpretation of IIEF EF domain scores based on six key aspects of EF. [ABSTRACT FROM AUTHOR]

This paper describes ways to approach the conceptual and practical challenges associated with interpreting the clinical meaning of scores produced by patient reported outcome (PRO) questionnaires, particularly when used to inform efficacy decisions for regulatory approval for oncology products. Score interpretation estimates are not inherent to PRO questionnaires per se, instead, vary dependent upon sample and study design characteristics. Scores from PRO measures can be interpreted at the individual and group level, and each carries its own set of statistics for evaluating differences. Oncology researchers have a variety of methods and data analytic strategies available to support their score interpretation needs, which should be considered in the context of their a priori knowledge of the target patient population, the hypothesized effects of treatment, the study design and assessment schedule, and the inferences and decisions to be made from the PRO data. [ABSTRACT FROM AUTHOR]

The authors report final-year ward simulation data from the University of Dundee Medical School. Faculty who designed this assessment intend for the final score to represent an individual senior medical student's level of clinical performance. The results are included in each student's portfolio as one source of evidence of the student's capability as a practitioner, professional, and scholar. Our purpose in conducting this study was to illustrate how assessment designers who are creating assessments to evaluate clinical performance might develop propositions and then collect and examine various sources of evidence to construct and evaluate a validity argument. The data were from all 154 medical students who were in their final year of study at the University of Dundee Medical School in the 2010-2011 academic year. To the best of our knowledge, this is the first report on an analysis of senior medical students' clinical performance while they were taking responsibility for the management of a simulated ward. Using multi-facet Rasch measurement and a generalizability theory approach, we examined various sources of validity evidence that the medical school faculty have gathered for a set of six propositions needed to support their use of scores as measures of students' clinical ability. Based on our analysis of the evidence, we would conclude that, by and large, the propositions appear to be sound, and the evidence seems to support their proposed score interpretation. Given the body of evidence collected thus far, their intended interpretation seems defensible. [ABSTRACT FROM AUTHOR]

The purpose of this study was to investigate the ways in which universities use the GRE® General Test scores to award merit-based fellowships to first-year graduate students in doctoral programs. While GRE use in fellowship award decisions is a common practice, there is very little validity evidence to support its use in this context. This exploratory qualitative study uses an argument-based validity approach as a conceptual framework to document GRE use systematically in award decisions. The findings from this study indicate that universities use merit-based fellowships to recruit the best and the brightest graduate students. The GRE is used as part of the nominee selection process in departments and is also used in the final selection process by university-wide committees. The GRE scores are primarily used to infer whether students have the baseline knowledge and skills needed to succeed in graduate school. These academic competencies were one of the characteristics the decision makers identified as required for successful doctoral students. The participants stated that the factors for success in graduate school are nebulous; however, the students with high GRE scores and other noteworthy supporting materials are expected to perform strongly in graduate school and, therefore, are worthy of the investment of a merit-based fellowship. [ABSTRACT FROM AUTHOR]

Score reports have one or more intended audiences: the people who use the reports to make decisions about test takers, including teachers, administrators, parents and test takers. Attention to audience when designing a score report supports assessment validity by increasing the likelihood that score users will interpret and use assessment results appropriately. Although most design guidelines focus on making score reports understandable to people who are not testing professionals, audiences should be defined by more than just their lack of statistical knowledge. This paper introduces an approach to identifying important audience characteristics for designing computer-based, interactive score reports. Through three examples, we demonstrate how an audience analysis suggests adesign pattern, which guides the overall design of a report, as well as design details, such as data representations and scaffolding. We conclude with a research agenda for furthering the use of audience analysis in the design of interactive score reports. [ABSTRACT FROM PUBLISHER]