Search

Context Conducting research in real life settings (effectiveness studies) can introduce many confounding factors. Efficacy studies seek to control for researcher bias and data quality rather than considering how the efficacy of an intervention is changed by the contexts in which it is used. Relatively little is known about the impact of context on educational interventions, in particular on multimedia learning. Methods An effectiveness study to understand implementation variance of online educational modules in surgery clerkships was conducted in six US medical schools participating in an efficacy trial of different multimedia designs. Student and teacher experiences were captured through focus groups and one-to-one interviews with trial participants and their teachers. Audio-recordings of these sessions were transcribed and analysed using grounded theory techniques. Results Differences were identified in student and teacher perceptions of how the educational intervention had been implemented and how its uptake had been influenced by context-dependent factors: (i) the intervention was implemented in different ways to suit different educational contexts and this influenced how students and teachers responded to it; (ii) the ways students and teachers interacted with, and behaved around, the intervention influenced its uptake; (iii) the way the intervention was perceived by students and teachers influenced its uptake; and (iv) the medium and design of the intervention had a directing influence on its uptake. Conclusions It was observed that each institutional context formed a complex educational ecology. The intervention became interwoven with different educational ecologies so that it could no longer be considered a stable variable across the study. We suggest that researchers should conduct implementation-profiling studies in advance of any intervention-based research to account for the constructing nature of educational ecologies on their interventions and in doing so to more clearly differentiate between efficacy and effectiveness studies.

Medical Education 2012: 46: 289–298 Objectives Using a large image bank, we systematically examined how the use of different ratios of abnormal to normal cases affects trainee learning. Methods This was a prospective, double-blind, randomised, three-arm education trial conducted in six academic training programmes for emergency medicine and paediatric residents in post-licensure years 2–5. We developed a paediatric ankle trauma radiograph case bank. From this bank, we constructed three different 50-case training sets, which varied in their proportions of abnormal cases (30%, 50%, 70%). Levels of difficulty and diagnoses were similar across sets. We randomly assigned residents to complete one of the training sets. Users classified each case as normal or abnormal, specifying the locations of any abnormalities. They received immediate feedback. All participants completed the same 20-case post-test in which 40% of cases were abnormal. We determined participant sensitivity, specificity, likelihood ratio and signal detection parameters. Results A total of 100 residents completed the study. The groups did not differ in accuracy on the post-test (p = 0.20). However, they showed considerable variation in their sensitivity–specificity trade-off. The group that received a training set with a high proportion of abnormal cases achieved the best sensitivity (0.69, standard deviation [SD] = 0.24), whereas the groups that received training sets with medium and low proportions of abnormal cases demonstrated sensitivities of 0.63 (SD = 0.21) and 0.51 (SD = 0.24), respectively (p

Objectives Most medical procedures have a time element. It is uncommon, however, to explicitly use chronometry, the measurement of time, in the learning of these procedures. This study considered whether instructional designs that include chronometry could improve deliberate practice and be used in meaningful formative assessments. Methods A selective review of the medical education literature was undertaken to identify how chronometry was used in a broad sampling of medical education research in the learning of medical procedures. We identified prior publications in which time measurement was used either directly as a pedagogic intervention or as an assessment method in a medical school programme. Results Our review suggests a number of desirable features of chronometry. For the individual learner, procedural time measurements can demonstrate both improving ability and increasing consistency. Chronometry can enhance instructional designs involving deliberate practice by facilitating overlearning (i.e. learning that goes beyond minimum competence), increasing the challenge level and enhancing self-regulation of learning (e.g. self-competition). Breaking down chronometric data into meaningful interval or split times might further inform instructional designs. Conclusions Chronometry has the potential to contribute to instructional designs and assessment methods in medical procedures training. However, more research is needed to elucidate its full potential and describe possible negative consequences of this widely available but underutilised educational tool.

Context We examined whether a ‘hint’ manoeuvre increases the time novice medical learners spend on reviewing a radiograph, thereby potentially increasing their interpretation accuracy. Methods Senior year medical students were recruited into a randomised control, three-arm, multicentre trial. Students reviewed an online 50-case learning set that varied in degree of ‘hint’ intervention. The ‘hint’ was a dialogue box that appeared after a student submitted an answer, encouraging the student to re-evaluate their interpretation. The students in the control group received no hints. In the weak intervention group, students received ‘hints’ with 66% of their incorrect interpretations and 33% of those that were correct. In the strong intervention group, the incorrect interpretation hint frequency was 80%, whereas for correct responses it was 20%. All students completed a 20-case post-test immediately and 2 weeks after the 50 cases. The primary outcome was student performance on the immediate post-test, measured as the ability to discriminate between normal and abnormal films (dPrime). Secondary outcomes included the probability of considering the hint, time spent on learning cases and knowledge retention at 2 weeks. Results We enrolled 117 medical students from three sites into the three study groups: control (36), weak intervention (40) and strong intervention (41) groups. The mean (standard deviation) dPrime in the control, weak and strong groups were 0.4 (1.1), 0.7 (1.1) and 0.4 (0.9), respectively (P = 0.4). In the weak and strong groups, participants reconsidered answers in 556 of 1944 (28.6%) hinting opportunities, and those who reconsidered their answers spent a mean (95% confidence interval) of 13.9 (11.9, 16.0) seconds longer on each case. There were no significant differences in knowledge retention at 2 weeks between the groups (P = 0.2). Conclusions Although the implemented hinting strategy did result in students spending more time considering a proportion of the cases, overall it was not effective in improving student performance.

Context The unbiased selection of images representing a spectrum of diagnostic difficulty is an important first step in designing effective assessment and teaching interventions for X-ray interpretation. Objectives This study aimed to develop a scale that would reliably differentiate more difficult X-rays from those that are easier to interpret. Methods After pilot testing, an X-ray difficulty scale (XRDS) was developed. Raters of different learner levels from two universities were presented with 20 chest X-rays (CXRs) and asked to read them and then to answer the scale questions that would help to differentiate the level of difficulty of interpretation of each film. Reliability of the scale was evaluated. Face validity of the scale was assessed and the construct validity of two hypotheses was tested. Results The final scale consisted of five questions in which a given X-ray could score from − 10 (most difficult) to + 10 (easiest to interpret) by a single rater. Raters included 53 medical students, 10 paediatric residents and 10 emergency staff. The scale demonstrated excellent internal consistency (r = 0.94), inter-rater reliability (r = 0.95) and overall reliability (r = 0.90) in medical students. Construct validity testing demonstrated good correlation (r = 0.72) between diagnostic accuracy and mean XRDS score. Mean scores on the scale were significantly lower (indicating that CXRs were more difficult to interpret) for students than for resident and staff doctors (P