Search

To determine diagnoses and image features that are associated with difficult prepubescent female genital image interpretations.This was a mixed-methods study conducted at a tertiary care pediatric center using images from a previously developed education platform.Participants comprised 107 medical students, residents, fellows, and attendings who interpreted 158 cases to derive case difficulty estimates.This was a planned secondary analysis of participant performance data obtained from a prospective multi-center cross-sectional study. An expert panel also performed a descriptive review of images with the highest frequency of diagnostic error.We derived the proportion of participants who interpreted an image correctly, and features that were common in images with the most frequent diagnostic errors.We obtained 16,906 image interpretations. The mean proportion correct scores for each diagnosis were as follows: normal/normal variants 0.84 (95% confidence interval [CI] 0.82, 0.87); infectious/dermatology pathology 0.59 (95% CI 0.45, 0.73); anatomic pathology 0.61 (95% CI 0.41, 0.81); and, traumatic pathology 0.64 (95% CI 0.49, 0.79). The mean proportion correct scores varied by diagnosis (P .001). The descriptive review demonstrated that poor image quality, infant genitalia, normal variant anatomy, external material (eg, diaper cream) in the genital area, and nonspecific erythema were common features in images with lower accuracy scores.A quantitative and qualitative examination of prepubescent female genital examination image interpretations provided insight into diagnostic challenges for this complex examination. These data can be used to inform the design of teaching interventions to improve skill in this area.

OBJECTIVES: To complement bedside learning of point‐of‐care ultrasound (POCUS), we developed an online learning assessment platform for the visual interpretation component of this skill. This study examined the amount and rate of skill acquisition in POCUS image interpretation in a cohort of pediatric emergency medicine (PEM) physician learners. METHODS: This was a multicenter prospective cohort study. PEM physicians learned POCUS using a computer‐based image repository and learning assessment system that allowed participants to deliberately practice image interpretation of 400 images from four pediatric POCUS applications (soft tissue, lung, cardiac, and focused assessment sonography for trauma [FAST]). Participants completed at least one application (100 cases) over a 4‐week period. RESULTS: We enrolled 172 PEM physicians (114 attendings, 65 fellows). The increase in accuracy from the initial to final 25 cases was 11.6%, 9.8%, 7.4%, and 8.6% for soft tissue, lung, cardiac, and FAST, respectively. For all applications, the average learners (50th percentile) required 0 to 45, 25 to 97, 66 to 175, and 141 to 290 cases to reach 80, 85, 90, and 95% accuracy, respectively. The least efficient (95th percentile) learners required 60 to 288, 109 to 456, 160 to 666, and 243 to 1040 cases to reach these same accuracy benchmarks. Generally, the soft tissue application required participants to complete the least number of cases to reach a given proficiency level, while the cardiac application required the most. CONCLUSIONS: Deliberate practice of pediatric POCUS image cases using an online learning and assessment platform may lead to skill improvement in POCUS image interpretation. Importantly, there was a highly variable rate of achievement across learners and applications. These data inform our understanding of POCUS image interpretation skill development and could complement bedside learning and performance assessments.

OBJECTIVES: Using a sample of pediatric chest radiographs (pCXR) taken to rule out pneumonia, we obtained diagnostic interpretations from physicians and used learning analytics to determine the radiographic variables and participant review processes that predicted for an incorrect diagnostic interpretation. METHODS: This was a prospective cross‐sectional study. A convenience sample of frontline physicians with a range of experience levels interpreted 200 pCXR presented using a customized online radiograph presentation platform. Participants were asked to determine absence or presence (with respective location) of pneumonia. The pCXR were categorized for specific image‐based variables potentially associated with interpretation difficulty. We also generated heat maps displaying the locations of diagnostic error among normal pCXR. Finally, we compared image review processes in participants with higher versus lower levels of clinical experience. RESULTS: We enrolled 83 participants (20 medical students, 40 postgraduate trainees, and 23 faculty) and obtained 12,178 case interpretations. Variables that predicted for increased pCXR interpretation difficulty were pneumonia versus no pneumonia (β = 8.7, 95% confidence interval [CI] = 7.4 to 10.0), low versus higher visibility of pneumonia (β = –2.2, 95% CI = –2.7 to –1.7), nonspecific lung pathology (β = 0.9, 95% CI = 0.40 to 1.5), localized versus multifocal pneumonia (β = –0.5, 95% CI = –0.8 to –0.1), and one versus two views (β = 0.9, 95% CI = 0.01 to 1.9). A review of diagnostic errors identified that bony structures, vessels in the perihilar region, peribronchial thickening, and thymus were often mistaken for pneumonia. Participants with lower experience were less accurate when they reviewed one of two available views (p

Background Correct interpretation of the prepubescent female genital examination is a critical skill; however, physician skill in this area is limited. Objective To complement the bedside learning of this examination, we developed a learning platform for the visual diagnosis of the prepubescent female genital examination and examined the amount and rate of skill acquisition. Participants and Setting Medical students, residents, and fellows and attendings participated in an on-line learning platform. Methods This was a multicenter prospective cross-sectional study. Study participants deliberately practiced 158 prepubescent female genital examination cases hosted on a computer-based learning and assessment platform. Participants assigned the case normal or abnormal; if abnormal, they identified the location of the abnormality and the specific diagnosis. Participants received feedback after every case. Results We enrolled 107 participants (26 students, 31 residents, 24 fellows and 26 attendings). Accuracy (95 % CI) increased by 10.3 % (7.8, 12.8), Cohen’s d-effect size of 1.17 (1.14, 1.19). The change in specificity was +16.8 (14.1, 19.5) and sensitivity +2.4 (-0.9, 5.6). It took a mean (SD) 46.3 (32.2) minutes to complete cases. There was no difference between learner types with respect to initial (p = 0.2) or final accuracy (p = 0.4) scores. Conclusions This study’s learning intervention led to effective and feasible skill improvement. However, while participants improved significantly with normal cases, which has relevance in reducing unnecessary referrals to child protection teams, learning gains were not as evident in abnormal cases. All levels of learners demonstrated a similar performance, emphasizing the need for this education even among experienced clinicians.

Objectives: Forgetting curves plot skill decay over time. After exposure to a simulation-based radiograph interpretation learning system, we determined the rate of learning decay and how this was i...

Collecting and analyzing large amounts of process data for the purposes of education can be considered a big data/learning analytics (BD/LA) approach to improving learning. However, in the education of health care professionals, the application of BD/LA is limited to date. The authors discuss the potential advantages of the BD/LA approach for the process of learning via cognitive simulations. Using the lens of a cognitive model of radiograph interpretation with four phases (orientation, searching/scanning, feature detection, and decision making), they reanalyzed process data from a cognitive simulation of pediatric ankle radiography where 46 practitioners from three expertise levels classified 234 cases online. To illustrate the big data component, they highlight the data available in a digital environment (time-stamped, click-level process data). Learning analytics were illustrated using algorithmic computer-enabled approaches to process-level feedback.For each phase, the authors were able to identify examples of potentially useful BD/LA measures. For orientation, the trackable behavior of re-reviewing the clinical history was associated with increased diagnostic accuracy. For searching/scanning, evidence of skipping views was associated with an increased false-negative rate. For feature detection, heat maps overlaid on the radiograph can provide a metacognitive visualization of common novice errors. For decision making, the measured influence of sequence effects can reflect susceptibility to bias, whereas computer-generated path maps can provide insights into learners' diagnostic strategies.In conclusion, the augmented collection and dynamic analysis of learning process data within a cognitive simulation can improve feedback and prompt more precise reflection on a novice clinician's skill development.

Sequential context effects, the psychological interactions occurring between the events of successive trials when a sequence of similar stimuli are judged, have interested psychologists for decades. It has been well established that individuals exhibit sequential context effects in psychophysical experiments involving unidimensional stimuli. Recent evidence shows that these effects generalize to quantitative judgments of more complex multidimensional stimuli such as images of faces, chairs, and shoes. In this article, we test for the presence of sequential context effects by re-examining previously published data on categorical judgments of 234 complex radiographic images made by 20 experienced physicians and 20 medical students engaged in an online training task. We found that medical students, but not experienced physicians, displayed evidence of sequential context effects. We also found evidence suggesting that as the students learned over blocks of trials, they tended to shift from relative comparisons between consecutive images toward more independent comparisons of each image against (strengthening) internalized standards.

Background: Radiograph teaching files are usually dominated by abnormal cases, implying that normal radiographs are easier to interpret. Our main objective was to compare the interpretation difficulty of normal versus abnormal radiographs of a set of common pediatric radiographs.Methods: We developed a 234-item digital case bank of pediatric ankle radiographs, recruited a convenience sample of participants, and presented the cases to each participant who then classified the cases as normal or abnormal. We determined and contrasted the interpretation difficulty of the normal and abnormal x-rays items using Rasch Measurement Theory. We also identified case features that were associated with item difficulty.Results: 139 participants (86 medical students, 7 residents, 29 fellows, 5 emergency physicians, and 3 radiologists) rated a minimum of 50 cases each, which resulted in 16,535 total ratings. Abnormal cases were more difficult (+0.99 logits) than were normal ones (-0.58 logits), difference 1.57 logits (95% CI 1.2, 2.0), but there was considerable overlap in difficulty scores. Patient variables associated with a more difficult normal radiograph included younger patient age (β = -0.16, 95% CI -0.22, -0.10), history of distal fibular tenderness (β = 0.55, 95% CI 0.17, 0.93), and presence of a secondary ossification centre (β = 0.84, 95% CI 0.27, 1.41).Conclusions: While abnormal images were more difficult to interpret, normal images did show a range of interpretation difficulties. Including a significant proportion of normal cases may be of benefit to learners.

Models for diagnostic reasoning in radiology have been based on the observed behaviors of experienced radiologists but have not directly focused on the thought processes of novices as they improve their accuracy of image interpretation. By collecting think-aloud verbal reports, the current study was designed to investigate differences in specific thought processes between medical students (novices) as they learn and radiologists (experts), so that we can better design future instructional environments. Seven medical students and four physicians with radiology training were asked to interpret and diagnose pediatric elbow radiographs where fracture is suspected. After reporting their diagnosis of a case, they were given immediate feedback. Participants were asked to verbalize their thoughts while completing the diagnosis and while they reflected on the provided feedback. The protocol analysis of their verbalizations showed that participants used some combination of four processes to interpret the case: gestalt interpretation, purposeful search, rule application, and reasoning from a prior case. All types of processes except reasoning from a prior case were applied significantly more frequently by experts. Further, gestalt interpretation was used with higher frequency in abnormal cases while purposeful search was used more often for normal cases. Our assessment of processes could help guide the design of instructional environments with well-curated image banks and analytics to facilitate the novice’s journey to expertise in image interpretation.

Studies in the learning sciences suggest that drawingmay prove useful for reection, as a learning diagnostic,and as a cognitive tool.1 A study has been designed todetermine if drawing at dierent times while learning withan application on radiograph interpretation (Figure 1) willimprove learning and cognition. The research is ongoing,and new qualitative data reveals various uses for drawingdependent on the timing of the drawing activity.

Medical Education 2012: 46: 289–298 Objectives Using a large image bank, we systematically examined how the use of different ratios of abnormal to normal cases affects trainee learning. Methods This was a prospective, double-blind, randomised, three-arm education trial conducted in six academic training programmes for emergency medicine and paediatric residents in post-licensure years 2–5. We developed a paediatric ankle trauma radiograph case bank. From this bank, we constructed three different 50-case training sets, which varied in their proportions of abnormal cases (30%, 50%, 70%). Levels of difficulty and diagnoses were similar across sets. We randomly assigned residents to complete one of the training sets. Users classified each case as normal or abnormal, specifying the locations of any abnormalities. They received immediate feedback. All participants completed the same 20-case post-test in which 40% of cases were abnormal. We determined participant sensitivity, specificity, likelihood ratio and signal detection parameters. Results A total of 100 residents completed the study. The groups did not differ in accuracy on the post-test (p = 0.20). However, they showed considerable variation in their sensitivity–specificity trade-off. The group that received a training set with a high proportion of abnormal cases achieved the best sensitivity (0.69, standard deviation [SD] = 0.24), whereas the groups that received training sets with medium and low proportions of abnormal cases demonstrated sensitivities of 0.63 (SD = 0.21) and 0.51 (SD = 0.24), respectively (p

To demonstrate how learning curves can describe proficiency improvements associated with deliberate practice of radiograph interpretation.This was a prospective, cross-sectional study of pediatric residents in two tertiary care programs. A 234-item digital case bank of pediatric ankle radiographs was developed. The authors gave participants a brief clinical summary of each case and asked them to consider three radiograph views of the ankle. Participants classified each case as either normal or abnormal and, if applicable, specified the location of the abnormality. They received immediate feedback and a radiologist's dictated report. The authors reviewed longitudinal learning curves, which were generated based on calculated test characteristics (e.g., accuracy, sensitivity, specificity).Eighteen participants (56.3% of those eligible) completed all 234 cases. The form of the participants' learning curves was similar across all test characteristics. The curves showed a period of "noise" until the participants completed an average of 20 cases. The slope of the learning curve was maximal from 21 to 50 cases during which cumulative sensitivity (95% CI) increased from 0.50 (0.45, 0.57) to 0.54 (0.47, 0.58). Then, the curves reached an inflection point after which learning slowed but did not stop even after 234 cases. The final cumulative sensitivity was 0.60 (0.54, 0.63). Applying a reference criterion, the authors classified learners into formative categories.Learning curves describing deliberate practice of radiograph interpretation allow medical educators to define at which point(s) practice is most efficient and how much practice is required to achieve a defined level of mastery.

ACADEMIC EMERGENCY MEDICINE 2010; 17:709–717 © 2010 by the Society for Academic Emergency Medicine Abstract Objectives: The Low Risk Ankle Rule (LRAR) is a validated clinical decision rule (CDR) about the indications for ankle radiographs in children with acute blunt ankle trauma. Although application of the LRAR has the potential to safely reduce the rate of ankle radiography by 60%, current x-ray rates in most emergency departments (EDs) in the United States and Canada remain unnecessarily high (85%–100%). To evaluate this gap between knowledge and practice, physicians who treat pediatric ankle injuries in EDs were surveyed to determine physician awareness and use of the LRAR, acceptability of the LRAR as measured by the Ottawa Acceptability for Decision Rules Scale (OADRS), and perceived barriers to the use of a validated pediatric ankle x-ray rule. Methods: An on-line survey of members of two national pediatric emergency medicine (PEM) physician associations in the United States and Canada was conducted using a modified Dillman technique. Results: Response rates were 75.6% (149/197) in Canada and 45.7% (352/770) in the United States, yielding an aggregate rate of 51.8%. Only 119 of 478 respondents (24.9%) had heard of the LRAR, and 53 of 432 (12.3%) were sufficiently familiar with the LRAR to apply it. The LRAR scored a mean (± standard deviation [SD]) OADRS score of 4.28 out of 6 (±0.67), comparable to published OADRS scores for two well-known CDRs used in adults. Of the respondents, 434 of 471 (92.1%) at least “slightly agreed” that ankle x-ray CDRs would be useful in their practice, with no significant differences between the two sides of the border (p = 0.28). Ankle x-ray rules were felt to save time by 342 (72.6%) of the participants, and the pediatric ankle exam was considered easy enough to apply a CDR by 306 (65.0%). The most common barriers reported for use of any ankle x-ray rule included perceived reduction in family satisfaction without imaging in 380 (80.7%), nurse-initiated x-ray protocols not based on ankle x-ray rules in 285 (60.5%), concerns about missing a significant fracture in 248 (52.7%), and a preference for own clinical judgment in 246 (52.2%). Conclusions: Although the LRAR had a high acceptability score among respondents in this survey, this validated CDR is not widely known and is even less frequently applied by PEM physicians in the United States and Canada. Barriers were identified that will guide efforts to improve the knowledge translation of the LRAR into pediatric EDs.

Signal detection theory (SDT) parameters can describe a learner's ability to discriminate (d') normal from abnormal and the learner's criterion (λ) to under or overcall abnormalities. To examine the serial changes in SDT parameters with serial exposure to radiological cases. 46 participants were recruited for this study: 20 medical students (MED), 6 residents (RES), 12 fellows (FEL), 5 staff pediatric emergency physicians (PEM), and 3 staff radiologists (RAD). Each participant was presented with 234 randomly assigned ankle radiographs using a web-based application. Participants were given a clinical scenario and considered 3 views of the ankle. They classified each case as normal or abnormal. For abnormal cases, they specified the location of the abnormality. Immediate feedback included highlighting on the images and the official radiologist's report. The low experience group (MED, RES, FEL) showed steady improvement in discrimination ability with each case, while the high experience group (PEM, RAD) had higher and stable discrimination ability throughout the exercise. There was also a difference in the way the high and low experience groups balanced sensitivity and specificity (λ) with the low experience group tending to make more errors calling positive radiographs negative. This tendency was progressively less evident with each increase in expertise level. SDT metrics provide valuable insight on changes associated with learning radiograph interpretation, and may be used to design more effective instructional strategies for a given learner group.

Learning curves are a useful way of representing the rate of learning over time. Features include an index of baseline performance (y-intercept), the efficiency of learning over time (slope parameter) and the maximal theoretical performance achievable (upper asymptote). Each of these parameters can be statistically modelled on an individual and group basis with the resulting estimates being useful to both learners and educators for feedback and educational quality improvement. In this primer, we review various descriptive and modelling techniques appropriate to learning curves including smoothing, regression modelling and application of the Thurstone model. Using an example dataset we demonstrate each technique as it specifically applies to learning curves and point out limitations.

Summary Decades of research on perception and prediction of randomness led us to speculate that the various response tendencies observed in these studies might manifest in multi-trial discrimination tasks used in medical education. By re-analyzing data from a previously published study in which 46 physicians and medical trainees judged 234 pediatric ankle radiographs, we show that (i) response tendencies can be differentially induced when individuals receive uniquely ordered sequences and (ii) response patterns consistent with win-stay/lose-shift and win-shift/lose-stay heuristics can be predicted from stimulus alternation rates and marginal distributions. Our results illustrate the importance of carefully arranging trials when studying discrimination and when using discrimination tasks to teach or to assess learners' skill levels. We call into question the wisdom of designing studies that present uniquely ordered stimulus sequences and discuss alternatives. Copyright © 2013 John Wiley & Sons, Ltd.

Context We examined whether a ‘hint’ manoeuvre increases the time novice medical learners spend on reviewing a radiograph, thereby potentially increasing their interpretation accuracy. Methods Senior year medical students were recruited into a randomised control, three-arm, multicentre trial. Students reviewed an online 50-case learning set that varied in degree of ‘hint’ intervention. The ‘hint’ was a dialogue box that appeared after a student submitted an answer, encouraging the student to re-evaluate their interpretation. The students in the control group received no hints. In the weak intervention group, students received ‘hints’ with 66% of their incorrect interpretations and 33% of those that were correct. In the strong intervention group, the incorrect interpretation hint frequency was 80%, whereas for correct responses it was 20%. All students completed a 20-case post-test immediately and 2 weeks after the 50 cases. The primary outcome was student performance on the immediate post-test, measured as the ability to discriminate between normal and abnormal films (dPrime). Secondary outcomes included the probability of considering the hint, time spent on learning cases and knowledge retention at 2 weeks. Results We enrolled 117 medical students from three sites into the three study groups: control (36), weak intervention (40) and strong intervention (41) groups. The mean (standard deviation) dPrime in the control, weak and strong groups were 0.4 (1.1), 0.7 (1.1) and 0.4 (0.9), respectively (P = 0.4). In the weak and strong groups, participants reconsidered answers in 556 of 1944 (28.6%) hinting opportunities, and those who reconsidered their answers spent a mean (95% confidence interval) of 13.9 (11.9, 16.0) seconds longer on each case. There were no significant differences in knowledge retention at 2 weeks between the groups (P = 0.2). Conclusions Although the implemented hinting strategy did result in students spending more time considering a proportion of the cases, overall it was not effective in improving student performance.

Deliberate practice is an important skill-training strategy in emergency medicine (EM) education. Learning curves display the relationship between practice and proficiency. Forgetting curves show the opposite, and demonstrate how skill decays over time when it is not reinforced. Using examples of published studies of deliberate practice in EM we list the properties of learning and forgetting curves and suggest how they can be combined to create experience curves: a longitudinal representation of the relationship between practice, skill acquisition, and decay over time. This framework makes explicit the need to avoid a piecemeal, episodic approach to skill practice and assessment in favor of more emphasis on what can be done to improve durability of competence over time. The authors highlight the implications for both educators and education researchers.

The Low Risk Ankle Rule (LRAR) is a validated clinical decision rule (CDR) about the indications for ankle radiographs in children with acute blunt ankle trauma. Although application of the LRAR has the potential to safely reduce the rate of ankle radiography by 60%, current x-ray rates in most emergency departments (EDs) in the United States and Canada remain unnecessarily high (85%-100%). To evaluate this gap between knowledge and practice, physicians who treat pediatric ankle injuries in EDs were surveyed to determine physician awareness and use of the LRAR, acceptability of the LRAR as measured by the Ottawa Acceptability for Decision Rules Scale (OADRS), and perceived barriers to the use of a validated pediatric ankle x-ray rule.An on-line survey of members of two national pediatric emergency medicine (PEM) physician associations in the United States and Canada was conducted using a modified Dillman technique.Response rates were 75.6% (149/197) in Canada and 45.7% (352/770) in the United States, yielding an aggregate rate of 51.8%. Only 119 of 478 respondents (24.9%) had heard of the LRAR, and 53 of 432 (12.3%) were sufficiently familiar with the LRAR to apply it. The LRAR scored a mean (+/- standard deviation [SD]) OADRS score of 4.28 out of 6 (+/-0.67), comparable to published OADRS scores for two well-known CDRs used in adults. Of the respondents, 434 of 471 (92.1%) at least "slightly agreed" that ankle x-ray CDRs would be useful in their practice, with no significant differences between the two sides of the border (p = 0.28). Ankle x-ray rules were felt to save time by 342 (72.6%) of the participants, and the pediatric ankle exam was considered easy enough to apply a CDR by 306 (65.0%). The most common barriers reported for use of any ankle x-ray rule included perceived reduction in family satisfaction without imaging in 380 (80.7%), nurse-initiated x-ray protocols not based on ankle x-ray rules in 285 (60.5%), concerns about missing a significant fracture in 248 (52.7%), and a preference for own clinical judgment in 246 (52.2%).Although the LRAR had a high acceptability score among respondents in this survey, this validated CDR is not widely known and is even less frequently applied by PEM physicians in the United States and Canada. Barriers were identified that will guide efforts to improve the knowledge translation of the LRAR into pediatric EDs.

Context The unbiased selection of images representing a spectrum of diagnostic difficulty is an important first step in designing effective assessment and teaching interventions for X-ray interpretation. Objectives This study aimed to develop a scale that would reliably differentiate more difficult X-rays from those that are easier to interpret. Methods After pilot testing, an X-ray difficulty scale (XRDS) was developed. Raters of different learner levels from two universities were presented with 20 chest X-rays (CXRs) and asked to read them and then to answer the scale questions that would help to differentiate the level of difficulty of interpretation of each film. Reliability of the scale was evaluated. Face validity of the scale was assessed and the construct validity of two hypotheses was tested. Results The final scale consisted of five questions in which a given X-ray could score from − 10 (most difficult) to + 10 (easiest to interpret) by a single rater. Raters included 53 medical students, 10 paediatric residents and 10 emergency staff. The scale demonstrated excellent internal consistency (r = 0.94), inter-rater reliability (r = 0.95) and overall reliability (r = 0.90) in medical students. Construct validity testing demonstrated good correlation (r = 0.72) between diagnostic accuracy and mean XRDS score. Mean scores on the scale were significantly lower (indicating that CXRs were more difficult to interpret) for students than for resident and staff doctors (P