Search

Student engagement is a central construct in education research, as it is a strong predictor and/or mediator of both cognitive and non-cognitive student outcomes. However, there remain questions among researchers about how to properly measure student engagement during lessons. Our study contributes to this discussion by presenting the development and validation of a theoretically grounded questionnaire of students' lesson engagement, which is based on an experience sampling methodology. To validate this questionnaire, we issued it to 410 fifth- and sixth-grade students (10–12 years old). The results of both an exploratory and multilevel confirmatory factor analysis approved construct validity of the measure. Adequate within-person variability across time points was also found, which makes the questionnaire suitable for capturing fluctuations over time. Furthermore, partial scalar measurement invariance for questionnaire administration across time points and between fifth- and sixth-grade students was found. [ABSTRACT FROM AUTHOR]

The core content for a medical specialty outlines the scope of the discipline as well as the categories of knowledge considered essential to practice in the field. It provides a template for the development of curricula for medical school, graduate, and postgraduate education, as well as for creating certification standards. Venous and Lymphatic Medicine (VLM) is a specialty that has benefitted from contributions from specialists from several medical disciplines. Optimally, the societies, boards, and residency review committees representing these disciplines would uniformly recognize the scope of VLM to develop education and assessment standards to allow training and identification of qualified practitioners. In order to inform the standard setting bodies and other stakeholders of the current scope of VLM, a task force of VLM experts from cardiology, dermatology, emergency medicine, general surgery, interventional radiology, vascular medicine, and vascular surgery was formed to revise a 2014 consensus document defining the core content of the specialty of VLM. [ABSTRACT FROM AUTHOR]

Background: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace‐based assessments (WBAs). Single‐institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi‐institutional generalisability study to determine the influence of specific factors on reliability of WBAs. Methods: The authors analysed WBA data obtained for clerkship‐level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross‐classified random‐effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. Results: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. Conclusion: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high‐stakes summative entrustment decisions. Through examination of 53,000 workplace‐based assessments at 7 schools, no factors were found to be consistently associated with reliability, leading to the conclusion that these assessments are best used for formative feedback only. [ABSTRACT FROM AUTHOR]

This was a non‐blinded, single‐centre, randomized, controlled clinical trial that compared the effectiveness of direct observation of procedural skills (DOPSs)with traditional assessment methods in pressure injury (PI) care skills. The study population included 82 nursing professionals randomly assigned to the study group (n = 41) and the control group (n = 41). Both groups of nurses underwent a 6‐month training in PI care skills and were subsequently evaluated. The main outcome variables were the PI skill operation scores and theoretical scores. Secondary outcome variables included satisfaction and critical thinking abilities. Independent sample t‐tests and chi‐square tests were used to assess differences between the two groups of nurses. The results showed no statistically significant difference in PI skill operation scores between the two groups of nurses (p > 0.05). When comparing the PI theoretical scores, the study group scored higher than the control group, and this difference was statistically significant (p < 0.05). In terms of satisfaction assessment, the study group and the control group showed differences in improving self‐directed learning, enhancing communication skills with patients, improving learning outcomes and increasing flexibility in clinical application (p < 0.05). When comparing critical thinking abilities between the two groups of nurses, there was no statistically significant difference at the beginning of the training, but after 3 months following the training, there was a statistically significant difference between the two groups (p < 0.01).The results indicated that the DOPS was effective in improving PI theoretical scores, increasing nurse satisfaction with the training and enhancing critical thinking abilities among nurses. [ABSTRACT FROM AUTHOR]

OBJECTIVE: The COVID-19 pandemic resulted in training programs restructuring their curricula. Fellowship programs are required to monitor each fellow's training progress through a combination of formal evaluations, competency tracking, and measures of knowledge acquisition. The American Board of Pediatrics administers subspecialty in-training examinations (SITE) to pediatric fellowship trainees annually and board certification exams at the completion of the fellowship. The objective of this study was to compare SITE scores and certification exam passing rates before and during the pandemic. METHODS: In this retrospective observational study, we collected summative data on SITE scores and certification exam passing rates for all pediatric subspecialties from 2018 to 2022. Trends over time were assessed using analysis of variance (ANOVA) analysis to test for trends across years within one group and t-test analysis to compare groups before and during the pandemic. RESULTS: Data were obtained from 14 pediatric subspecialties. Comparing prepandemic to pandemic scores, Infectious Diseases, Cardiology, and Critical Care Medicine saw statistically significant decreases in SITE scores. Conversely, Child Abuse and Emergency Medicine saw increases in SITE scores. Emergency Medicine saw a statistically significant increase in certification exam passing rates, while Gastroenterology and Pulmonology saw decreases in exam passing rates. CONCLUSIONS: The COVID-19 pandemic resulted in restructuring didactics and clinical care based on the needs of the hospital. There were also societal changes affecting patients and trainees. Subspecialty programs with declining scores and certification exam passing rates may need to assess their educational and clinical programs and adapt to the needs of trainees' learning edges. [ABSTRACT FROM AUTHOR]

Background: Nurse educators at a public university expanded their standardized examination program in Spring 2020 to include more specialty examinations, required remediation, and retesting for students who did not attain benchmark scores. This study conducted a remediation-focused program evaluation. Method: A mixed-methods design was used. Aggregated deidentified Health Education Systems, Inc. (HESI) specialty examination scores (n = 907) were analyzed. Feedback from nursing students and recent alumni (n = 70) was collected using a Qualtrics survey. Results: For students scoring under benchmark, higher remediation time requirements led to greater increases in second-attempt specialty examination scores. More alumni than current students reported remediation was helpful (p =.029). Participants' perspectives regarding remediation barriers, helpful strategies, and motivating policies were identified. Five themes emerged from open-ended survey comments, adding depth to the quantitative findings. Conclusion: Recommendations for standardized examination remediation strategies and policies are offered. [J Nurs Educ. 2023;62(9):519–522.] [ABSTRACT FROM AUTHOR]

The ratings that judges or examiners use for determining pass marks and students' performance on OSCEs serve a number of essential functions in medical education assessment, and their validity is a pivotal issue. However, some types of errors often occur in ratings that require special efforts to minimise. Rater characteristics (e.g. generosity error, severity error, central tendency error or halo error) may present a source of performance irrelevant variance. Prior literature shows the fundamental problems in student performance measurement attached to judges' or examiners' errors. It also indicates that the control of such errors supports a robust and credible pass mark and thus, accurate student marks. Therefore, for a standard-setter who identifies the pass mark and an examiner who rates student performance in OSCEs, proper, user-friendly feedback on their standard-setting and ratings is essential for reducing bias. This feedback provides useful avenues for understanding why performance ratings may be irregular and how to improve the quality of ratings. This AMEE Guide discusses various methods of feedback to support examiners' understanding of the performance of students and the standard-setting process with an effort to make inferences from assessments fair, valid and reliable. [ABSTRACT FROM AUTHOR]

The goal of the present study was to evaluate whether children as standardized patients (SPs) for a summative OSCE station fulfills the Ottawa criteria for good assessment. Data for participating students (n = 124), raters (n = 11), and children (n = 127) were collected using questionnaires (students, raters), focus groups (raters), and interviews (children, raters). The psychometrics (including reproducibility) of this OSCE station were also analyzed according to quality indices. Validity, reproducibility, feasibility, educational effect, and acceptability were attained for this OSCE station. Students and raters accepted the new approach and evaluated it as fair and acceptable, with realism and validity. This OSCE station had a perceived positive educational effect for ∼30% of the students, who expected child SPs. Raters stated it as feasible to involve children. The children enjoyed their participation, which was acceptable and feasible. Statistical measures were in the recommended ranges and showed reproducibility. Children as SPs for a summative OSCE station was perceived as fair, acceptable, realistic, and feasible by all stakeholders, and with perceived positive educational impact. This pediatric OSCE station fulfilled five of the seven Ottawa criteria for good assessment. Further studies are needed to investigate different pediatric stations. [ABSTRACT FROM AUTHOR]

Many institutions require candidates to achieve a minimum number of OSCE stations passed (MNSP) in addition to the aggregate pass mark. The stated rationale is usually that this conjunctive standard prevents excessive degrees of compensation across an assessment. However, there is a lack of consideration and discussion of this practice in the medical education literature. We consider the motivations for the adoption of the MNSP from the assessment designer perspective, outlining potential concerns about the complexity of what the OSCE is trying to achieve, particularly around the blueprinting process and the limitations of scoring instruments. We also introduce four potential methods for setting an examinee-centred MNSP standard, and highlight briefly the theoretical advantages and disadvantages of these approaches. There are psychometric arguments for and against the limiting of compensation in OSCEs, but it is clear that many stakeholders value the application of an MNSP standard. This paper adds to the limited literature on this important topic and notes that current MNSP practices are often problematic in high stakes settings. More empirical work is needed to develop understanding of the impact on pass/fail decision-making of the proposed standard setting methods developed in this paper. [ABSTRACT FROM AUTHOR]

Introduction: Accurate recording of operative cases is essential during training to demonstrate experience. However, indicative numbers delineating minimum desirable experience may incentivise exaggeration or misrepresentation of experience. This study aimed to determine perceptions of real-world eLogbook use among UK surgeons in training.Material and Methods: An anonymous online questionnaire was disseminated electronically using a pre-planned yield-maximisation strategy, incorporating regional champions, email and social media. Evaluation employed mixed methods in a combined interpretation of quantitative and qualitative data from the questionnaire. Recommendations for development of the eLogbook were itemised from respondents' free text items and a modified Delphi process, conducted within the Council of the UK national trainee representative body, the Association of Surgeons in Training, determined the strength of each recommendation.Results: Analysis included 906 complete responses from training-grade surgeons (34.8% female) from all UK recognised specialties and all grades of training. More than two-thirds (68.5%) believed that overstatement or misrepresentation of case involvement occurs. A fifth (20.8%) reported witnessing trainees logging cases they had not actually participated in and almost a third (32.7%) had witnessed overstatement, yet few (15.1%) had raised such an issue with a supervisor. Most (85.2%) respondents had few or no eLogbook entries validated. More than a quarter of respondents felt pressure to overstate their involvement in cases (28.6%) and the number recorded (28.1%). Almost a third (31.5%) felt the required case number for completion of training was not achievable. Female trainees were less likely to feel well supervised (p = 0.022) and to perceive targets for completion of training were achievable (p = 0.005). Thematic analysis identified four key themes to explain logbook misuse: Pressure to achieve training milestones; eLogbook functionality issues; training deficiencies and probity.Conclusions: Inaccurate operative recording was widely reported, primarily in response to perceived pressure to achieve targets for career progression. Operative logbooks may not be as accurate as intended. Consensus recommendations are made for improvement in the eLogbook and its use. [ABSTRACT FROM AUTHOR]

Objective: To use the Messick validity framework for a simulation-based assessment of vaginal hysterectomy skills.Methods: Video recordings of physicians at different levels of training and experience performing vaginal hysterectomy on a high-fidelity vaginal surgery model were objectively assessed using a modified 10-item Vaginal Surgical Skills Index, a one-item global scale of overall performance, and a pass-fail criterion. Participants included obstetrics and gynecology trainees and faculty from five institutions. Video recordings were independently assessed by expert surgeons blinded to the identities of the study participants.Results: Fifty surgeons (11 faculty, 39 trainees) were assessed. Experience level correlated strongly with both the modified Vaginal Surgical Skills Index and global scale score, with more experienced participants receiving higher scores (Pearson r=0.81, P<.001; Pearson r=0.74, P<.001). Likewise, surgical experience was also moderately correlated with the modified Vaginal Surgical Skills Index and global scale score (Pearson r=0.55, P<.001; Pearson r=0.58, P<.001). The internal consistency of the modified Vaginal Surgical Skills Index was excellent (Cronbach's alpha=0.97). Interrater reliability of the modified Vaginal Surgical Skills Index and global scale score, as measured by the intraclass correlation coefficient, was moderate to good (0.49-0.95; 0.50-0.87). Using the receiver operating characteristic curve and the pass-fail criterion, a modified Vaginal Surgical Skills Index cutoff score of 27 was found to most accurately (area under the curve 0.951, 95% CI 0.917-0.983) differentiate competent from noncompetent surgeons.Conclusion: We demonstrated validity evidence for using a high-fidelity vaginal surgery model with the modified Vaginal Surgical Skills Index or global scale score to assess vaginal hysterectomy skills. [ABSTRACT FROM AUTHOR]

Competency frameworks serve various roles including outlining characteristics of a competent workforce, facilitating mobility, and analysing or assessing expertise. Given these roles and their relevance in the health professions, we sought to understand the methods and strategies used in the development of existing competency frameworks. We applied the Arksey and O'Malley framework to undertake this scoping review. We searched six electronic databases (MEDLINE, CINAHL, PsycINFO, EMBASE, Scopus, and ERIC) and three grey literature sources (greylit.org, Trove and Google Scholar) using keywords related to competency frameworks. We screened studies for inclusion by title and abstract, and we included studies of any type that described the development of a competency framework in a healthcare profession. Two reviewers independently extracted data including study characteristics. Data synthesis was both quantitative and qualitative. Among 5710 citations, we selected 190 for analysis. The majority of studies were conducted in medicine and nursing professions. Literature reviews and group techniques were conducted in 116 studies each (61%), and 85 (45%) outlined some form of stakeholder deliberation. We observed a significant degree of diversity in methodological strategies, inconsistent adherence to existing guidance on the selection of methods, who was involved, and based on the variation we observed in timeframes, combination, function, application and reporting of methods and strategies, there is no apparent gold standard or standardised approach to competency framework development. We observed significant variation within the conduct and reporting of the competency framework development process. While some variation can be expected given the differences across and within professions, our results suggest there is some difficulty in determining whether methods were fit-for-purpose, and therefore in making determinations regarding the appropriateness of the development process. This uncertainty may unwillingly create and legitimise uncertain or artificial outcomes. There is a need for improved guidance in the process for developing and reporting competency frameworks. [ABSTRACT FROM AUTHOR]

The array of different philosophical positions underlying contemporary views on competence, assessment strategies and justification have led to advances in assessment science. Challenges may arise when these philosophical positions are not considered in assessment design. These can include (a) a logical incompatibility leading to varied or difficult interpretations of assessment results, (b) an "anything goes" approach, and (c) uncertainty regarding when and in what context various philosophical positions are appropriate. We propose a compatibility principle that recognizes that different philosophical positions commit assessors/assessment researchers to particular ideas, assumptions and commitments, and applies ta logic of philosophically-informed, assessment-based inquiry. Assessment is optimized when its underlying philosophical position produces congruent, aligned and coherent views on constructs, assessment strategies, justification and their interpretations. As a way forward we argue that (a) there can and should be variability in the philosophical positions used in assessment, and these should be clearly articulated to promote understanding of assumptions and make sense of justifications; (b) we focus on developing the merits, boundaries and relationships within and/or between philosophical positions in assessment; (c) we examine a core set of principles related to the role and relevance of philosophical positions; (d) we elaborate strategies and criteria to delineate compatible from incompatible; and (f) we articulate a need to broaden knowledge/competencies related to these issues. The broadened use of philosophical positions in assessment in the health professions affect the "state of play" and can undermine assessment programs. This may be overcome with attention to the alignment between underlying assumptions/commitments. [ABSTRACT FROM AUTHOR]

Background: The Accreditation Council for Graduate Medical Education requires milestone reporting of the Six General Core Competencies. Additionally, Graduate Medical Education (GME) is transitioning to adopt competency-based education methodologies including entrustable professional activities (EPAs) for objective, observable, and measurable milestone progression. The College of American Pathologists published 19 EPAs, including one for transfusion-related adverse events. This survey study includes developing EPAs for transfusion reaction evaluation and assessing residents before and after implementing these EPAs.Study Design and Methods: Three transfusion reaction EPAs were developed and implemented in July 2018 for the Postgraduate Year (PGY) 2 pathology residents. An online, anonymous survey was sent to all 21 pathology trainees before and one year after EPA implementation. In July 2018 and August 2019, each survey included the same six multiple-choice, single-response, confidence questions, with a rating scale of extremely, very, slightly, or not at all confident. This study was approved by the hospital's Institutional Review Board for Health Sciences Research and GME Committee.Results: Analysis was performed on PGY2-4 residents. In 2018, 13 of 20 participants were analyzed. In 2019, 15 of 19 participants were analyzed. Number and percentage of responses were reported. The results showed an increase in trainee confidence, with the greatest improvement among the first class to use the EPAs.Conclusion: EPAs provide an effective framework for objective and measurable progression of trainees. One year after the implementation of transfusion reaction EPAs at our site, the trainees showed enhanced confidence levels in handling Blood Bank and Transfusion Medicine Services coverage. [ABSTRACT FROM AUTHOR]

Background: Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.Methods: This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence.Results: In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence.Conclusion: This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool. [ABSTRACT FROM AUTHOR]

The expectations for clinically ready graduates have increased over the years, paralleling changes in the NCLEX exam with increasingly cognitively difficult items in preparation for professional nursing practice. Yet, it is widely recognized that nursing program exit standardized exams have come under increased scrutiny in several public cases. Several articles have frequently been cited in the opposing argument against use of the HESI Exit Exam or other standardized nursing exit exams. We review two of these for logical coherence and standard psychometric adherence given their recurrent use in arguing against nursing program exit standardized testing. In light of patient safety implications and school accreditation consequences, it's more important that schools use exit standardized testing and assess minimum competency of students based on an exit program standardized exam. Measurement standards are critical to these analyses. The use of a program exit standardized exam also alerts students that faculty are holding them accountable for their time in the program. [ABSTRACT FROM AUTHOR]

The article presents a study on response options for a range of total assessment questions to determine the reliability of the test scores. Topics discussed include factors that influence reliability of test scores, the effect of lengthening the number of questions and increasing the number of options per question on the expected reliability, and the reliability of four- and five-options per question.

Context: Standard setting is critically important to assessment decisions in medical education. Recent research has demonstrated variations between medical schools in the standards set for shared items. Despite the centrality of judgement to criterion‐referenced standard setting methods, little is known about the individual or group processes that underpin them. This study aimed to explore the operation and interaction of these processes in order to illuminate potential sources of variability. Methods: Using qualitative research, we purposively sampled across UK medical schools that set a low, medium or high standard on nationally shared items, collecting data by observation of graduation‐level standard‐setting meetings and semi‐structured interviews with standard‐setting judges. Data were analysed using thematic analysis based on the principles of grounded theory. Results: Standard setting occurred through the complex interaction of institutional context, judges' individual perspectives and group interactions. Schools' procedures, panel members and atmosphere produced unique contexts. Individual judges formed varied understandings of the clinical and technical features of each question, relating these to their differing (sometimes contradictory) conceptions of minimally competent students, by balancing information and making suppositions. Conceptions of minimal competence variously comprised: limited attendance; limited knowledge; poor knowledge application; emotional responses to questions; 'test‐savviness', or a strategic focus on safety. Judges experienced tensions trying to situate these abstract conceptions in reality, revealing uncertainty. Groups constructively revised scores through debate, sharing information and often constructing detailed clinical representations of cases. Groups frequently displayed conformity, illustrating a belief that outlying judges were likely to be incorrect. Less frequently, judges resisted change, using emphatic language, bargaining or, rarely, 'polarisation' to influence colleagues. Conclusions: Despite careful conduct through well‐established procedures, standard setting is judgementally complex and involves uncertainty. Understanding whether or how these varied processes produce the previously observed variations in outcomes may offer routes to enhance equivalence of criterion‐referenced standards. How does variability arise when different groups decide upon the appropriate cut score for an examination? The authors shine light on this question by exploring the complex interaction between individual and group processes. [ABSTRACT FROM AUTHOR]

Background and Objectives: New standards announced in 2017 could increase the failure rate for Step 2 Clinical Skills (CS). The purpose of this study was to identify student performance metrics associated with risk of failing.Methods: Data for 1,041 graduates of one medical school from 2014 through 2017 were analyzed, including 30 (2.9%) failures. Metrics included Medical College Admission Test, United States Medical Licensing Examination Step 1, and clerkship National Board of Medical Examiners (NBME) Subject Examination scores; faculty ratings in six clerkships; and scores on an objective structured clinical examination (OSCE). Bivariate statistics and regression were used to estimate risk of failing.Results: Those failing had lower Step 1 scores, NBME scores, faculty ratings, and OSCE scores (P<.02). Students with four or more low ratings were more likely to fail compared to those with fewer low ratings (relative risk [RR], 12.76, P<.0001). Logistic regression revealed other risks: low surgery NBME scores (RR 3.75, P=.02), low pediatrics NBME scores (RR 3.67, P=.02), low ratings in internal medicine (RR 3.42, P=.004), and low OSCE Communication/Interpersonal Skills (RR 2.55, P=.02).Conclusions: Certain medical student performance metrics are associated with risk of failing Step 2 CS. It is important to clarify these and advise students accordingly. [ABSTRACT FROM AUTHOR]

Pressurized Submarine Escape Training (PSET) physically prepares submariners to safely escape a submarine at depth. Failure to complete PSET is not a submarine service disqualification. Serious medical incidents are rare, but the safety record tradeoff has been low throughput. From 2009 to 2015, only 34% of students screened completed PSET. Students may be medically screened out of the training altogether (disqualified), or dropout during the physical training (attrite). Training records from 12,122 U.S. Navy students were used to identify factors contributing to training disqualification and attrition. Multivariate logistic regression model predictors included demographic and screening items. Association to PSET disqualification included cold/congestion/cough (ORadj 12.34), limited duty status (ORadj 4.29), Physical Readiness Test failure (ORadj 3.37), pneumonia or bronchitis in last 2 years (ORadj 3.17) and nervousness or anxiety in tight spaces (ORadj 2.37). Basic Enlisted Submarine School students were more likely to be disqualified and attrite than other submariner groups, and black/African American (ORadj 1.53) students were more likely to attrite than white students. Only cold/congestion/cough (ORadj 1.52), trouble swimming (ORadj 1.53), and screening during cold/flu season (ORadj 1.28), were associated with training attrition. Recommendations to modify screening requirements are listed in conclusions. [ABSTRACT FROM AUTHOR]

Introduction: In 2010, the Ottawa Conference produced a set of consensus criteria for good assessment. These were well received and since then the working group monitored their use. As part of the 2010 report, it was recommended that consideration be given in the future to preparing similar criteria for systems of assessment. Recent developments in the field suggest that it would be timely to undertake that task and so the working group was reconvened, with changes in membership to reflect broad global representation.Methods: Consideration was given to whether the initially proposed criteria continued to be appropriate for single assessments and the group believed that they were. Consequently, we reiterate the criteria that apply to individual assessments and duplicate relevant portions of the 2010 report.Results and discussion: This paper also presents a new set of criteria that apply to systems of assessment and, recognizing the challenges of implementation, offers several issues for further consideration. Among these issues are the increasing diversity of candidates and programs, the importance of legal defensibility in high stakes assessments, globalization and the interest in portable recognition of medical training, and the interest among employers and patients in how medical education is delivered and how progression decisions are made. [ABSTRACT FROM AUTHOR]

Background and Objectives: Family medicine residency programs accredited by the Accreditation Council for Graduate Medical Education and the American Osteopathic Association typically require their residents to take the American Board of Family Medicine's In-Training Examination (ITE) and the American College of Osteopathic Family Physicians' In-Service Examination (ISE). With implementation of the single accreditation system (SAS), is it necessary to administer both examinations? This pilot study assessed whether the degree of similarity for the construct of family medicine knowledge and clinical decision making as measured by both exams is high enough to be considered equivalent and analyzed resident ability distribution on both exams.Methods: A repeated measures design was used to determine how similar and how different the rankings of PGY-3s were with regard to their knowledge of family medicine as measured by the ISE and ITE. Eighteen third-year osteopathic residents participated in the analysis, and the response rate was 100%.Results: The correlation between ISE and ITE rankings was moderately high and significantly different from zero (rs=.76, P<0.05). A Wilcoxon signed rank test indicated that the median ISE score of 62 was not statistically significantly different than the median ITE score of 71 (Z=-0.74, P=0.46, 2-tailed).Conclusions: The lack of a difference on statistical analysis of ISE scores and the ITE scores of the PGY-3 residents suggests that the cohort of osteopathic residents in family residency programs and the cohort of residents in ACGME-accredited programs seem to be of comparable ability, therefore there is no clear justification for administering both examinations. [ABSTRACT FROM AUTHOR]

Background: Scant evidence is found in the literature regarding the unit-level Health Education Systems, Inc. (HESI) examinations. This study aims to (a) compare student Psych HESI examination scores between cohorts after increasing grade allocation from 5% to 15%, and (b) examine student perceptions of the impact of increasing the percentage of course grade allocation. Method: An ex post facto research design analyzed differences in Psych HESI examination scores. HESI examination mean score differences between groups were analyzed using ANOVA. Narrative responses were analyzed using content analysis. Results: Psych HESI examination means increased from 836 in spring 2015 (5% of course grade) to 905 in fall 2015 (15% of course grade). Narrative response themes included (a) Validated Learning; (b) Compelled Participants to Study More; (c) Incongruent With Class Content and Preparation Books; and (d) Stress- Producing. Conclusion: HESI examination scores improved with increasing course grade percentage allocated to the HESI score. Students indicated positive and negative perceptions of the increased grade allocation. [ABSTRACT FROM AUTHOR]

Background: The Membership of the Royal College of Surgeons examination (MRCS, Parts A and B) is one of the largest postgraduate surgical exams in the world, but little is known about the factors that affect candidate performance. We describe the relationship between both parts of MRCS and several independent predictors of MRCS success.Methods: Pearson correlation coefficients were used to examine the linear relationship between MRCS Part A and B and logistic regression analysis to identify potential independent predictors of MRCS success. We included all UK medical graduates who attempted either part of MRCS between 2007 and 2016.Results: 7896 candidates made 11,867 attempts at Part A and 4310 made 5738 attempts at Part B. A positive correlation was found between Part A and B first attempt score (r = 0.41, P < 0.001). Gender (male vs. female, odds ratio (OR) 2.78, 95% confidence interval (CI) 1.83-4.19), ethnicity (white vs. Black Minority and Ethnic, OR 1.70, 95% CI 1.52-1.89), stage of training (e.g. Core Surgical Year 2 trainees vs. Foundation Year 1 doctors, OR 0.50, 95% CI 0.32-0.77) and maturity (young vs. mature graduates, OR 2.60, 95% CI 1.81-3.63) were all found to be independent predictors of Part A success. In addition to ethnicity and stage of training, Part A performance (number of attempts and score) was also identified as an independent predictor for Part B. The odds of passing each part of the MRCS decreased by 14% (OR 0.86, 95% CI 0.80-0.92) for Part A and 30% for Part B (OR 0.70, 95% CI 0.61-0.81) with each additional attempt that was made.Conclusions: Several independent predictors of MRCS success were identified, but only ethnicity and stage of training were found to be common predictors of both Part A and B. [ABSTRACT FROM AUTHOR]

This article contributes to research practices in marital and family therapy, specifically the dyadic and development over time in clinical supervision, and describes and applies methodological strategies to develop measurements congruent with the systemic and developmental principles of the field. This project evaluates the psychometric properties of the dyadic supervision evaluation (DSE) in terms of measurement equivalence and causality. A structural equation analysis is conducted utilizing the actor-partner interdependent model resulting in a goodness of fit. This study presents a longitudinal model for evaluating the supervisory relationship and proposes a model of clinical supervision evaluation. The relationship among latent constructs in the DSE and its limitations related to inference and application are discussed. [ABSTRACT FROM AUTHOR]

Wilderness first responders are trained to provide competent medical care in wilderness settings or until evacuation for more advanced treatment can be obtained. In light of the isolated environments in which they are called upon to respond to illnesses and injuries, their ability to effectively apply their training is crucial. Despite the responsibility assigned to them, there is an absence of research assessing the skill and knowledge retention of wilderness first responders, creating a gap in understanding whether a deficit in their ability to perform exists between certifications. Without such research, it is important to review knowledge and skill retention in related responder groups. The literature over the past 4 decades documents the loss over time of skills and knowledge across an array of trained responders, both professional and laypeople. Although the findings reviewed suggest that WFRs will exhibit a similar pattern of increasing skill loss beginning shortly after certification and a slower, but concurrent, decrease in knowledge, research is needed to document or refute this assumption. [ABSTRACT FROM AUTHOR]

Despite the frequent use of state-of-the-art psychometric models in the field of medical education, there is a growing body of literature that questions their usefulness in the assessment of medical competence. Essentially, a number of authors raised doubt about the appropriateness of psychometric models as a guiding framework to secure and refine current approaches to the assessment of medical competence. In addition, an intriguing phenomenon known as case specificity is specific to the controversy on the use of psychometric models for the assessment of medical competence. Broadly speaking, case specificity is the finding of instability of performances across clinical cases, tasks, or problems. As stability of performances is, generally speaking, a central assumption in psychometric models, case specificity may limit their applicability. This has probably fueled critiques of the field of psychometrics with a substantial amount of potential empirical evidence. This article aimed to explain the fundamental ideas employed in psychometric theory, and how they might be problematic in the context of assessing medical competence. We further aimed to show why and how some critiques do not hold for the field of psychometrics as a whole, but rather only for specific psychometric approaches. Hence, we highlight approaches that, from our perspective, seem to offer promising possibilities when applied in the assessment of medical competence. In conclusion, we advocate for a more differentiated view on psychometric models and their usage. [ABSTRACT FROM AUTHOR]

Medical school admissions interviews are used to assess applicants' nonacademic characteristics as advocated by the Association of American Medical Colleges' Advancing Holistic Review Initiative. The objective of this study is to determine whether academic metrics continue to significantly influence interviewers' scores in holistic processes by blinding interviewers to applicants' undergraduate grade point averages (uGPA) and Medical College Admission Test (MCAT). This study examines academic and demographic predictors of interview scores for two applicant cohorts at the University of Michigan Medical School. In 2012, interviewers were provided applicants' uGPA and MCAT scores; in 2013, these academic metrics were withheld from interviewers' files. Hierarchical regression analysis was conducted to examine the influence of academic and demographic variables on overall cohort interview scores. When interviewers were provided uGPA and MCAT scores, academic metrics explained more variation in interview scores (7.9%) than when interviewers were blinded to these metrics (4.1%). Further analysis showed a statistically significant interaction between cohort and uGPA, indicating that the association between uGPA and interview scores was significantly stronger for the 2012 unblinded cohort compared to the 2013 blinded cohort (β = .573, P < .05). By contrast, MCAT scores had no interactive effects on interviewer scores. While MCAT scores accounted for some variation in interview scores for both cohorts, only access to uGPA significantly influenced interviewers' scores when looking at interaction effects. Withholding academic metrics from interviewers' files may promote assessment of nonacademic characteristics independently from academic metrics. [ABSTRACT FROM AUTHOR]

This paper considers two major concerns centring on the Joint Council for Qualifications (JCQ) regulations for access arrangement and reasonable adjustments and qualifications for teachers who take on the role of Level 7 access arrangements assessor. Thus, the paper is divided into two parts. First, the JCQ 2017-2018 regulations are critically evaluated highlighting the areas of need for which greater clarity and more extensive detail is required in these regulations. The second part of the paper discusses the findings of research on teacher competence in test administration, scoring, and reporting. Drawing on evidence from the first stage of this research, McMurray, O'Callaghan, and McVeigh highlight the extensive formative process required to build a high-level skill set required for competent assessment involving the use of high level tests and also the specialist knowledge required to analyse and accurately report assessment findings to make recommendations for students with specific literacy difficulties. Recommendations are provided both at policy and practice levels regarding the content of courses and the assessment involving the use of high level tests and also the specialist knowledge required to analyse and accurately report assessment findings to make recommendations for students with specific literacy difficulties. The authors provide recommendations for test publishers, course providers, and assessors. [ABSTRACT FROM AUTHOR]

Background: The San Antonio Uniformed Services Health Education Consortium Infectious Disease Fellowship program historically included a monthly short-answer and multiple-choice quiz. The intent was to ensure medical knowledge in relevant content areas that may not be addressed through clinical rotations, such as operationally relevant infectious disease. After completion, it was discussed in a small group with faculty. Over time, faculty noted increasing dissatisfaction with the activity. Spaced interval education is useful in retention of medical knowledge and skills by medical students and residents. Its use in infectious disease fellow education has not been described. To improve the quiz experience, we assessed the introduction of spaced education curriculum in our program.Materials and Methods: A pre-intervention survey was distributed to assess the monthly quiz with Likert scale and open-ended questions. A multiple-choice question spaced education curriculum was created using the Qstream(R) platform in 2011. Faculty development on question writing was conducted. Two questions were delivered every 2 d. Incorrectly and correctly answered questions were repeated after 7 and 13 d, respectively. Questions needed to be answered correctly twice to be retired. Fellow satisfaction was assessed at semi-annual fellowship reviews over 5 yr and by a one-time repeat survey.Results: Pre-intervention survey of six fellows indicated dissatisfaction with the time commitment of the monthly quiz (median Likert score of 2, mean 6.5 h to complete), neutral in perceived utility, but satisfaction with knowledge retention (Likert score 4). Eighteen fellows over 5 yr participated in the spaced education curriculum. Three quizzes with 20, 39, and 48 questions were designed. Seventeen percentage of questions addressed operationally relevant topics. Fifty-nine percentage of questions were answered correctly on first attempt, improving to 93% correct answer rate at the end of the analysis. Questions were attempted 2,999 times. Fellows consistently indicated that the platform was "highly enjoyed," "beneficial," a "fun format," and "completely satisfied." Fellows additionally commented that they desired more questions and considered the platform helpful in board preparation. Formal survey data post-intervention found that the fellows were satisfied with the new approach, found it to be useful in board preparation, overall educational value, and in-line with their personal learning style (median Likert score of 4 for all queries). Fellows were satisfied with time commitment, spending a mean of 47 min on the spaced education curriculum questions per month.Conclusions: Introduction of a spaced education curriculum resulted in a sustained positive learner experience for >5 yr with demonstrated mastery of material. Spaced education learning is a viable addition to augment training experience, especially in areas of curricular gaps such as operational medicine. Correct answer data may also be useful to perform Accreditation Council for Graduate Medical Education-required objective assessment of knowledge. [ABSTRACT FROM AUTHOR]

Background: Comparing outcome of clinical skills assessment is challenging. This study proposes reliable and valid comparison of communication skills (1) assessment as practiced in Objective Structured Clinical Examinations (2). The aim of the present study is to compare CS assessment, as standardized according to the MAAS Global, between stations in a single undergraduate medical year.Methods: An OSCE delivered in an Irish undergraduate curriculum was studied. We chose the MAAS-Global as an internationally recognized and validated instrument to calibrate the OSCE station items. The MAAS-Global proportion is the percentage of station checklist items that can be considered as 'true' CS. The reliability of the OSCE was calculated with G-Theory analysis and nested ANOVA was used to compare mean scores of all years.Results: MAAS-Global scores in psychiatry stations were significantly higher than those in other disciplines (p<0.03) and above the initial pass mark of 50%. The higher students' scores in psychiatry stations were related to higher MAAS-Global proportions when compared to the general practice stations.Conclusion: Comparison of outcome measurements, using the MAAS Global as a standardization instrument, between interdisciplinary station checklists was valid and reliable.Practice Implications: The MAAS-Global was used as a single validated instrument and is suggested as gold standard. [ABSTRACT FROM AUTHOR]

Background: In the latest release of the Standards for Educational and Psychological Testing, Differential Item Functioning (DIF) is considered as validity evidence based on internal structure. However, there are no indications of how to design a DIF study as a validation study. In this paper, we propose relating DIF to all sources of validity evidence, and provide a general conceptual framework for transforming "typical" DIF studies into validation studies. Method: We perform a comprehensive review of the literature and make theoretical and practical proposals. Results: The article provides arguments in favour of addressing DIF detection and interpretation as validation studies, and suggestions for conducting DIF validation studies. Discussion: The combination of quantitative and qualitative data within a mixed methods research perspective, along with planning DIF studies as validation studies, can help improve the validity of test score interpretations. [ABSTRACT FROM AUTHOR]

This study explores how textbooks function in education. It asked whether opportunities provided in math textbooks to engage in tasks demanding different levels of understanding correlate with students’ achievements on tasks demanding equivalent levels of understanding on a standardized exam. The textbooks evaluated were two 8 th grade mathematics textbooks used by students in the Arab community in Israel, showing that Textbook A makes more cognitive demands than Textbook B. The study correlated textbooks’ cognitive demand with the scores of all 8 th grade students in the Arab community who completed the national math test in 2015 and studied in schools using either Textbook A or B (N = 4040), while attending to mediating variables. The findings show that if a textbook provides the opportunity to engage in tasks demanding higher levels of understanding, students using this book will have higher scores. The study shows that gender and SES play an important role in how opportunities provided in textbooks interact with students’ scores. Many factors influence variations in mathematics achievements within and between nations. The findings illuminate textbooks’ ability to provide opportunities to learn mathematics. As a result, they raise new questions about how teachers use textbooks and about the role of textbooks in promoting access and equity in mathematics education. Although the work explored specific textbooks, its findings shed light on how learning opportunities relate to achievements more generally. [ABSTRACT FROM AUTHOR]

Assessment of complex tasks integrating several competencies calls for a programmatic design approach. As single instruments do not provide the information required to reach a robust judgment of integral performance, 73 guidelines for programmatic assessment design were developed. When simultaneously applying these interrelated guidelines, it is challenging to keep a clear overview of all assessment activities. The goal of this study was to provide practical support for applying a programmatic approach to assessment design, not bound to any specific educational paradigm. The guidelines were first applied in a postgraduate medical training setting, and a process analysis was conducted. This resulted in the identification of four steps for programmatic assessment design: evaluation, contextualisation, prioritisation and justification. Firstly, the (re)design process starts with sufficiently detailing the assessment environment and formulating the principal purpose. Key stakeholders with sufficient (assessment) expertise need to be involved in the analysis of strengths and weaknesses and identification of developmental needs. Central governance is essential to balance efforts and stakes with the principal purpose and decide on prioritisation of design decisions and selection of relevant guidelines. Finally, justification of assessment design decisions, quality assurance and external accountability close the loop, to ensure sound underpinning and continuous improvement of the assessment programme. [ABSTRACT FROM AUTHOR]

PROBLEM The purpose of assigning grades at the conclusion of a course is to convey the level of achievement of learners. However, with the presence of grade inflation in nursing education, it poses a great threat to the evaluation process. PURPOSE The purpose of this article is to highlight the factors that influence grade inflation and to explore possible solutions to minimize the incidence of grade inflation in nursing education. FINDINGS A review of the literature revealed that a variety of factors influenced grade inflation and educators were indeed lenient with their grading practices, which hence attenuates the purpose of grading and evaluation. CONCLUSION While it may be an arduous task for nursing faculty to evaluate nursing students, nursing faculty have the obligation to preserve academic integrity and must devise appropriate solutions to address the incidence of grade inflation in nursing education. [ABSTRACT FROM AUTHOR]

Objective: To create, validate, and use a fertility awareness survey based on current U.S.Data: Design: Cross-sectional study.Setting: Not applicable.Patient(s): Phase 1 included U.S. women ages 18-45; phase 2 included female medical students and obstetrics and gynecology trainees at two urban academic programs.Intervention(s): Survey including demographics, the Fertility & Infertility Treatment Knowledge Score (FIT-KS) instrument, and General Nutrition Knowledge Questionnaire.Main Outcome Measure(s): Knowledge of natural fertility and infertility treatments.Result(s): The FIT-KS was validated through detailed item and validity analyses. In phase 1, 127 women participated; their median age was 31 years, and 43.7% had children. Their mean FIT-KS score was 16.2 ± 3.5 (55.9% correct). In phase 2, 118 medical trainees participated; their median age was 25 years, and 12.4% had children. Their mean FIT-KS score was 18.8 ± 2.1 (64.9% correct), with year of training correlating to a higher score (r=0.40). Participant awareness regarding lifestyle factors varied, but it was particularly low regarding the effects of lubricants. The majority underestimated the spontaneous miscarriage rate and overestimated the fecundability of 40-year-old women. There was general overestimation of success rates for assisted reproductive technologies, particularly among medical trainees.Conclusion(s): The FIT-KS is validated to current U.S. data for use in both general and medical populations as a quick assessment of fertility knowledge. The knowledge gaps demonstrated in this study correlate with national trends in delayed childbearing and time to initiate treatment. For medical trainees, these results raise concerns about the quality of fertility counseling they may be able to offer patients. Greater educational outreach must be undertaken to enhance fertility awareness. [ABSTRACT FROM AUTHOR]

Objective: The purpose of this study was to develop a reliable and valid scale that broadly measures knowledge about palliative care among non-healthcare professionals.Method: An initial item pool of 38 true/false questions was developed based on extensive qualitative and quantitative pilot research. The preliminary items were tested with a community sample of 614 adults aged 18-89 years as well as 30 palliative care professionals. The factor structure, reliability, stability, internal consistency, and validity of the 13-item Palliative Care Knowledge Scale (PaCKS) were assessed.Results: The results of our study indicate that the PaCKS meets or exceeds the standards for psychometric scale development.Significance Of Results: Prior to this study, there were no psychometrically evaluated scales with which to assess knowledge of palliative care. Our study developed the PaCKS, which is valid for assessing knowledge about palliative services in the general population. With the successful development of this instrument, new research exploring how knowledge about palliative care influences access and utilization of the service is possible. Prior research in palliative care access and utilization has not assessed knowledge of palliative care, though many studies have suggested that knowledge deficits contribute to underutilization of these services. Creating a scale that measures knowledge about palliative care is a critical first step toward understanding and combating potential barriers to access and utilization of this life-improving service. [ABSTRACT FROM AUTHOR]

In the midst of the debate surrounding the question of whether Australia’s National Assessment Program: Literacy and Numeracy (NAPLAN) test is high-stakes, it is evident that children’s own accounts of their experiences remain sparse. This paper describes the findings of a case study which documented the experiences of 105 children across two Catholic primary schools in Queensland serving different socio-economic status (SES) communities. Analysis of the data revealed that these teachers and principals did not experience NAPLAN as high-stakes. However, the data suggested that the children experienced the tests within a confusing context of contradictions and dissonances emanating from multiple sources; receiving little, if any, clear and consistent information regarding the purpose and significance of NAPLAN. While the children’s responses were varied, many reported NAPLAN as a negative experience, with some constructing the test as high-stakes. These constructions ranged from personal judgement or sense of letting their families down, to failure, and less commonly, grade retention and school exclusion. Some Year 3 children had also constructed good results as vital to future prosperity. These constructions bring into question the assumption that because NAPLAN is designed to be a low-stakes test, that children will necessarily experience it in this way. [ABSTRACT FROM PUBLISHER]

Background: The assessment of dermatology undergraduates is being done through computer assisted objective structured clinical examination at our institution for the last 4 years. We attempted to compare objective structured clinical examination (OSCE) and computer assisted objective structured clinical examination (CA-OSCE) as assessment tools.Aim: To assess the relative effectiveness of CA-OSCE and OSCE as assessment tools for undergraduate dermatology trainees.Methods: Students underwent CA-OSCE as well as OSCE-based evaluation of equal weightage as an end of posting assessment. The attendance as well as the marks in both the examination formats were meticulously recorded and statistically analyzed using SPSS version 20.0. Intercooled Stata V9.0 was used to assess the reliability and internal consistency of the examinations conducted. Feedback from both students and examiners was also recorded.Results: The mean attendance for the study group was 77% ± 12.0%. The average score on CA- OSCE and OSCE was 47.4% ± 19.8% and 53.5% ± 18%, respectively. These scores showed a mutually positive correlation, with Spearman's coefficient being 0.593. Spearman's rank correlation coefficient between attendance scores and assessment score was 0.485 for OSCE and 0.451 for CA-OSCE. The Cronbach's alpha coefficient for all the tests ranged from 0.76 to 0.87 indicating high reliability.Limitations: The comparison was based on a single batch of 139 students. Such an evaluation on more students in larger number of batches over successive years could help throw more light on the subject.Conclusions: Computer assisted objective structured clinical examination was found to be a valid, reliable and effective format for dermatology assessment, being rated as the preferred format by examiners. [ABSTRACT FROM AUTHOR]

A focus on quality assurance of assessment processes in tertiary education within Australia and throughout the world has resulted in a changing landscape of assessment types and grading schemes over the last decade. The use of criteria and standards-based assessment systems are now very commonplace in tertiary education. There are a variety of models now used, but typically they include a criteria sheet and a levelled rubric. An alternative to the traditional matrix-style rubric is the Continua Model of a Guide to Making Judgments (GTMJ). In this paper, we analyse available assessment models and their capacity to guide the marking, grading and moderation of student assessment tasks. We specifically address standards descriptors used to identify the quality expected at each standard. The research was undertaken through a community of practice within the School of Education at a tertiary institution where the collective goal of enhancing assessment grading tools to improve student outcomes was approached through a process of peer review. In our results section, we analyse the efficacy of an internal peer-review model as part of a community of practice and the professional learning about grading tools that occurs. [ABSTRACT FROM PUBLISHER]

Purpose: In this epilogue, we review the 4 response articles and highlight the implications of a multidimensional view of reading for the assessment and instruction of reading comprehension. Method: We reiterate the problems with standardized tests of reading comprehension and discuss the advantages and disadvantages of recently developed authentic tests of reading comprehension. In the "Instruction" section, we review the benefits and limitations of strategy instruction and highlight suggestions from the response articles to improve content and language knowledge. Conclusions: We argue that the only compelling reason to administer a standardized test of reading comprehension is when these tests are necessary to qualify students for special education services. Instruction should be focused on content knowledge, language knowledge, and specific task and learning requirements. This instruction may entail the use of comprehension strategies, particularly those that are specific to the task and focus on integrating new knowledge with prior knowledge. [ABSTRACT FROM AUTHOR]

Few studies have investigated specific learning disabilities (SLD) identification methods based on the identification of patterns of processing strengths and weaknesses (PSW). We investigated the reliability of SLD identification decisions emanating from different achievement test batteries for 1 method to operationalize the PSW approach: the concordance/discordance model (C/DM; Hale & Fiorello, 2004). Two studies examined the level of agreement for SLD identification decisions between 2 different simulated, highly correlated achievement test batteries. Study 1 simulated achievement and cognitive data across a wide range of potential latent correlations between an achievement deficit, a cognitive strength and a cognitive weakness. Latent correlations permitted simulation of case-level data at specified reliabilities for cognitive abilities and 2 achievement observations. C/DM criteria were applied and resulting SLD classifications from the 2 achievement test batteries were compared for agreement. Overall agreement and negative agreement were high, but positive agreement was low (0.33-0.59) across all conditions. Study 2 isolated the effects of reduced test reliability on agreement for SLD identification decisions resulting from different test batteries. Reductions in reliability of the 2 achievement tests resulted in average decreases in positive agreement of 0.13. Conversely, reductions in reliability of cognitive measures resulted in small average increases in positive agreement (0.0-0.06). Findings from both studies are consistent with prior research demonstrating the inherent instability of classifications based on C/DM criteria. Within complex ipsative SLD identification models like the C/DM, small variations in test selection can have deleterious effects on classification reliability. (PsycINFO Database Record [ABSTRACT FROM AUTHOR]

Predicting item difficulty is highly important in education for both teachers and item writers. Despite identifying a large number of explanatory variables, predicting item difficulty remains a challenge in educational assessment with empirical attempts rarely exceeding 25% of variance explained. This paper analyses 216 science items of key stage 2 tests which are national sampling assessments administered to 11 year olds in England. Potential predictors (topic, subtopic, concept, question type, nature of stimulus, depth of knowledge and linguistic variables) were considered in the analysis. Coding frameworks employed in similar studies were adapted and employed by two coders to independently rate items. Linguistic demands were gauged using a computational linguistic facility. The stepwise regression models predicted 23% of the variance with extended constructed questions and photos being the main predictors of item difficulty. While a substantial part of unexplained variance could be attributed to the unpredictable interaction of variables, we argue that progress in this area requires improvement in the theories and the methods employed. Future research needs to be centred on improving coding frameworks as well as developing systematic training protocols for coders. These technical advances would pave the way to improved task design and reduced development costs of assessments. [ABSTRACT FROM PUBLISHER]

Bayesian dynamic item response models have been successfully used for educational testing data; these models are especially useful for individually varying and irregularly spaced longitudinal testing data. However, because of the complexity of the models and the large size of the data sets, computation time is excessive for carrying out full data analyses in practice. Here, we introduce a parallel Markov chain Monte Carlo method to speed the implementation of these Bayesian models. Using both simulation data and real educational testing data for reading ability, we demonstrate that computation time is greatly reduced for our parallel computing method versus full data analyses. The estimated error of our method is shown to be small, using common distance metrics. Our parallel computing approach can be used for other models in the Educational and Psychometric fields, including Bayesian item response theory models. Copyright © 2017 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Computer adaptive testing (CAT) is a kind of tailored testing, in that it is a form of computer-based testing that is adaptive to each test-taker’s ability level. In this review, the impacts of CAT are discussed from different perspectives in order to illustrate crucial points to keep in mind during the development and implementation of CAT. Test developers and psychometricians often emphasize the efficiency and accuracy of CAT in comparison to traditional linear tests. However, many test-takers report feeling discouraged after taking CATs, and this feeling can reduce learning self-efficacy and motivation. A trade-off must be made between the psychological experiences of test-takers and measurement efficiency. From the perspective of educators and subject matter experts, nonstatistical specifications, such as content coverage, content balance, and form length are major concerns. Thus, accreditation bodies may be faced with a discrepancy between the perspectives of psychometricians and those of subject matter experts. In order to improve test-takers’ impressions of CAT, the author proposes increasing the target probability of answering correctly in the item selection algorithm even if doing so consequently decreases measurement efficiency. Two different methods, CAT with a shadow test approach and computerized multistage testing, have been developed in order to ensure the satisfaction of subject matter experts. In the shadow test approach, a full-length test is assembled that meets the constraints and provides maximum information at the current ability estimate, while computerized multistage testing gives subject matter experts an opportunity to review all test forms prior to administration. [ABSTRACT FROM AUTHOR]

Multisource assessment (MSA) uses multiple assessors to provide feedback. Little is known about the validity of using MSA feedback for improving students' ability to self-assess in a preclinical environment. Therefore, the aim of this study was to measure the validity of using a defined reflective process involving an MSA tool for building skill in dental students' self-evaluation of caries excavation on extracted teeth. As part of this process, 104 first-year students at one U.S. dental school used a self-generated study plan (SGSP) for structured reflection on MSA feedback during the 2013-14 academic year. Interrater agreement, determined through calculation of percentage-agreements in scoring, was measured among three assessor groups (self-, peer, and expert assessors) in formative assessment and between two assessor groups (self- and expert assessors) in summative assessment two weeks apart, allowing for reflective practice and completion of an SGSP between assessments. Validity for improving self-assessment was determined by measuring significance in positive shifts of agreement between self- and expert assessors. The results showed that interrater agreement between the self- and expert assessors increased significantly: from a 28% agreement in formative assessment to a 60% agreement in summative assessment. Significance in percentage shifts between assessments was demonstrated with a McNemar score of 0.26 (p<0.001). These results suggest that the described MSA tool and reflective process in an SGSP may be valid methods for improving skill in student self-evaluation of competence in caries excavation on extracted teeth. [ABSTRACT FROM AUTHOR]

Introduction: It is known that test-centered methods for setting standards in knowledge tests (e.g. Angoff or Ebel) are problematic, with expert judges not able to consistently predict the difficulty of individual items. A different approach is the Cohen method, which benchmarks the difficulty of the test based on the performance of the top candidates. Methods: This paper investigates the extent to which Ebel (and also Cohen) produces a consistent standard in a knowledge test when comparing between adjacent cohorts. The two tests are linked using common anchor items and Rasch analysis to put all items and all candidates on the same scale. Results: The two tests are of a similar standard, but the two cohorts are different in their average abilities. The Ebel method is entirely consistent across the two years, but the Cohen method looks less so, whilst the Rasch equating itself has complications – for example, with evidence of overall misfit to the Rasch model and change in difficulty for some anchor items. Conclusion: Based on our findings, we advocate a pluralistic and pragmatic approach to standard setting in such contexts, and recommend the use of multiple sources of information to inform the decision about the correct standard. [ABSTRACT FROM AUTHOR]

Objectives: Endotracheal intubation (ETI) is an important skill for all emergency providers; our ability to train and assess our learners is integral to providing optimal patient care. The primary aim of this study was to assess the inter-rater reliability (IRR) and discriminant validity of a novel field ETI assessment tool using a checklist-derived performance score (PS) and critical failure (CF) rate.Methods: Forty-three participants (18 paramedic students, 11 paramedics, and 14 emergency physicians [EPs]) performed ETI during a simulated trauma scenario on a pseudo-ventilated cadaver. Each participant was assessed by two experienced raters. IRR was calculated using the intraclass correlation coefficient. Regarding discriminant validity, a Kruskal-Wallis test was used to analyze PSs and a χ2 test was used for CFs. Mean global rating scale (GRS) scores were compared using an analysis of variance.Results: The ETI assessment tool had excellent IRR, with an intraclass correlation coefficient of 0.94. There was a significant difference in PSs, CFs, and GRSs (p < 0.05) between cohorts.Conclusion: The novel field ETI assessment tool has excellent reliability among trained raters and discriminates between experienced ETI providers (EPs) and less experienced ETI performers using PSs, CFs, and GRSs on a fresh cadaveric model. [ABSTRACT FROM AUTHOR]

Background: Rapid effective trauma resuscitations (TRs) decrease patient morbidity and mortality. Few studies have evaluated TR care times. Effective time goals and superior human patient simulator (HPS) training can improve patient survivability.Objectives: The purpose of this study was to compare live TR to HPS resuscitation times to determine mean incremental resuscitation times and ascertain if simulation was educationally equivalent. The study was conducted at San Antonio Military Medical Center, Department of Defense Level I trauma center.Design and Methods: This was a prospective observational study measuring incremental step times by trauma teams during trauma and simulation patient resuscitations. Trauma and simulation patient arms had 60 patients for statistical significance. Participants included Emergency Medicine residents and Physician Assistant residents as the trauma team leader.Results: The trauma patient arm revealed a mean evaluation time of 10:33 and simulation arm 10:23. Comparable time characteristics in the airway, intravenous access, blood sample collection, and blood pressure data subsets were seen.Conclusions: TR mean times were similar to the HPS arm subsets demonstrating simulation as an effective educational tool. Effective stepwise approaches, incremental time goals, and superior HPS training can improve patient survivability and improved departmental productivity using TR teams. [ABSTRACT FROM AUTHOR]