Search

Purpose To describe how the authors developed an objective structured clinical examination (OSCE) station to assess aspects of collaborative practice competency and how they then assessed validity using Kane's framework. Method After piloting the collaborative practice OSCE station in 2015 and 2016, this was introduced at the Cumming School of Medicine in 2017. One hundred fifty-five students from the class of 2017 and 22 students from the class of 2018 participated. To create a validity argument, the authors used Kane's framework that views the argument for validity as 4 sequential inferences on the validity of scoring, generalization, extrapolation, and implications, RESULTS: Scoring validity is supported by psychometric analysis of checklist items and the fact that the contribution of rater specificity to students' ratings was similar to OSCE stations assessing clinical skills alone. The claim of validity of generalization is backed by structural equation modeling and confirmatory factor analysis that identified 5 latent variables, including 3 related to collaborative practice ("provides an effective handover," "provides mutual support," and "shares their mental model"). Validity of extrapolation is argued based upon the correlation between the rating for "shares their mental model" and the rating on in-training evaluations for "relationship with other members of the health care team," in addition to the association between performance on the collaborative practice OSCE station and the subsequent rating of performance during residency. Finally, validity of implications is supported by the fact that pass/fail decisions on the collaborative practice station were similar to other stations and by the observation that ratings on different aspects of collaborative practice associate with pass/fail decisions. Conclusions Based upon the validity argument presented, the authors posit that this tool can be used to assess the collaborative practice competence of graduating medical students and the adequacy of training in collaborative practice.

Context The incidence of medical error, adverse clinical events and poor quality health care is unacceptably high and there are data to suggest that poor coordination of care, or teamwork, contributes to adverse outcomes. So, can we assume that increased collaboration in multidisciplinary teams improves performance and health care outcomes for patients? Methods In this essay, the authors discuss some reasons why we should not presume that collective decision making leads to better decisions and collaborative care results in better health care outcomes. Results Despite an exponential increase in interventions designed to improve teamwork and interprofessional education (IPE), we are still lacking good quality data on whether these interventions improve important outcomes. There are reasons why some of the components of ‘effective teamwork’, such as shared mental models, team orientation and mutual trust, could impair delivery of health care. For example, prior studies have found that brainstorming results in fewer ideas rather than more, and hinders rather than helps productivity. There are several possible explanations for this effect, including ‘social loafing’ and cognitive overload. Similarly, attributes that improve cohesion within groups, such as team orientation and mutual trust, may increase the risk of ‘groupthink’ and group conformity bias, which may lead to poorer decisions. Conclusions In reality, teamwork and IPE are not inherently good, bad or neutral; instead, as with any intervention, their effect is modified by the persons involved, the situation and the interaction between persons and situation. Thus, rather than assume better outcomes with teamwork and IPE interventions, as clinicians and educators we must demonstrate that our interventions improve the delivery of health care.

Clinical decision making requires knowledge, experience and analytical/non-analytical types of decision processes. As clinicians progress from novice to expert, research indicates decision-making becomes less reliant on foundational biomedical knowledge and more on previous experience. In this study, we investigated how knowledge and experience were reflected in terms of differences in neural areas of activation. Novice and expert clinicians diagnosed simple or complex (easy, hard) cases while functional magnetic resonance imaging (fMRI) data were collected. Our results highlight key differences in the neural areas activated in novices and experts during the clinical decision-making process. fMRI data were collected from ten second year medical students (novices) and ten practicing gastroenterologists (experts) while they diagnosed sixteen (eight easy and eight hard) clinical cases via multiple-choice questions. Behavioral data were collected for diagnostic accuracy (correct/incorrect diagnosis) and time taken to assign a clinical diagnosis. Two analyses were performed with the fMRI data. First, data from easy and hard cases were compared within respective groups (easy > hard, hard > easy). Second, neural differences between novices and experts (novice > expert, expert > novice) were assessed. Experts correctly diagnosed more cases than novices and made their diagnoses faster than novices on both easy and hard cases (all p's

After many years in the making, an increasing number of postgraduate medical education (PGME) training programs in North America are now adopting a competency-based medical education (CBME) framework based on entrustable professional activities (EPAs) that, in turn, encompass a larger number of competencies and training milestones. Following the lead of PGME, CBME is now being incorporated into undergraduate medical education (UME) in an attempt to improve integration across the medical education continuum and to facilitate a smooth transition from clerkship to residency by ensuring that all graduates are ready for indirect supervision of required EPAs on day one of residency training. The Association of Faculties of Medicine of Canada recently finalized its list of 12 EPAs, which closely parallels the list of 13 EPAs published earlier by the Association of American Medical Colleges, and defines the "core" EPAs that are an expectation of all medical school graduates.In this article, the authors focus on important, practical considerations for the transition to CBME that they feel have not been adequately addressed in the existing literature. They suggest that the transition to CBME should not threaten diversity in UME or require a major curricular upheaval. However, each UME program must make important decisions that will define its version of CBME, including which terminology to use when describing the construct being evaluated, which rating tools and raters to include in the assessment program, and how to make promotion decisions based on all of the available data on EPAs.

Although the clinical clerkship model is based upon sound pedagogy, including theories of social learning and situated learning, studies evaluating clinical performance of residents suggests that this model may not fully meet the learning needs of students. Here our objective was to design a curriculum to bridge the learning gaps of the existing clerkship model and then evaluate the impact of this on performance on clerkship summative evaluations. We followed Kern’s framework to design our curriculum and then compared performance on the clerkship objective structured clinical examination (OSCE), all summative clerkship multiple choice question (MCQ) examinations, and the Medical Council of Canada Qualifying Examination (MCCQE) Part 1 before and after the introduction of our curriculum. In the 2 years following the introduction of our clinical skills curriculum the mean score on the clerkship OSCE was significantly higher than in the 2 years prior to our curriculum [67.12 (5.3) vs. 62.44 (4.93), p < 0.001, d = 0.91]. With the exception of the surgical clerkship MCQ, performance on all clerkship summative MCQ examinations and MCCQE Part 1 was significantly higher following the introduction of our curriculum. In this study we found a significant improvement in the performance on clerks on summative evaluations of knowledge and clinical skills following the introduction of our clinical skills curriculum. Given the unpredictable nature of clinical rotations, the clerkship will always be a risk of failing to deliver the intended curriculum—so medical schools should continue to explore and evaluate ways of changing the delivery of clerkship training to improve learning outcomes.

BACKGROUND: Although workplace workload assessments exist in different fields, an endoscopy-specific workload assessment tool is lacking.OBJECTIVE: To validate such a workload tool and use it to map the progression of novice trainees in gastroenterology in performing their first endoscopies.METHODS: The National Aeronautics and Space Administration Task Load Index (NASA-TLX) workload assessment tool was completed by eight novice trainees in gastroenterology and 10 practicing gastroenterologists/surgeons. An exploratory factor analysis was performed to construct a streamlined endoscopy-specific task load index, which was subsequently validated. The ‘Endoscopy Task Load Index’ was used to monitor progression of trainee exertion and self-assessed performance over their first 40 procedures.RESULTS: From the factor analysis of the NASA-TLX, two principal components emerged: a measure of exertion and a measure of self-efficacy. These items became the components of the newly validated Endoscopy Task Load Index. There was a steady decline in self-perceived exertion over the training period, which was more rapid for gastroscopy than colonoscopy. The self-efficacy scores for gastroscopy rapidly increased over the first few procedures, reaching a plateau after this period of time. For colonoscopy, there was a progressive increase in reported self-efficacy over the first three quartiles of procedures, followed by a drop in self-efficacy scores over the final quartile.DISCUSSION: The present study validated an Endoscopy Task Load Index that can be completed in

PURPOSE Most diagnostic errors involve faulty diagnostic reasoning. Consequently, the authors assessed the effect of querying initial hypotheses on diagnostic performance. METHOD In 2007, the authors randomly assigned 67 first-year medical students from the University of Calgary to two groups and asked them to diagnose eight common problems. The authors presented the same primary data to both groups and asked students for their initial diagnosis. Then, after presenting secondary data that were either discordant or concordant with the primary data, they asked students for a final diagnosis. The authors noted changes in students' diagnoses and the accuracy of initial and final diagnoses for discordant and concordant cases. RESULTS For concordant cases, students retained 84.2% of their initial diagnoses and were equally likely to move toward a correct as incorrect final diagnosis (6.9% versus 8.9%, P = .3); no difference existed in the accuracy of initial and final diagnoses: 85.9% versus 84.0% (P = .4). By contrast, for discordant cases, students retained only 23.3% of initial diagnoses, change was almost invariably from incorrect to correct (76.3% versus 0.4%, P < .001), and final diagnoses were more accurate than initial diagnoses: 80.7% versus 4.8% (P < .001). Overall, no difference existed in the accuracy of final diagnoses for concordant and discordant cases (P = .18). CONCLUSIONS These data suggest that querying an initial diagnostic hypothesis does not harm a correct diagnosis but instead allows students to rectify an incorrect diagnosis. Whether querying initial diagnoses reduces diagnostic error in clinical practice remains unknown.

Endoscopy instruction has progressed a great deal in recent years, evolving from the age-old dictum of ‘see one, do one’ to the current skillful application of sound educational principles. Some of these educational principles are generic and applicable to the teaching of any content at all levels, while others are quite specific to technical skills training. The present review summarizes these important principles under the following headings: creating a learner-centred curriculum; delivering an achievable learning task; and moving from theory to practice. The present article challenges national gastroenterology organizations to embrace these concepts in structured, outcome-based educational programs.

Context Despite the impartiality implied in its title, the objective structured clinical examination (OSCE) is vulnerable to systematic biases, particularly those affecting raters’ performance. In this study our aim was to examine OSCE ratings for evidence of differential rater function over time (DRIFT), and to explore potential causes of DRIFT. Methods We studied ratings for 14 internal medicine resident doctors over the course of a single formative OSCE, comprising 10 12-minute stations, each with a single rater. We evaluated the association between time-slot and rating for a station. We also explored a possible interaction between time-slot and station difficulty, which would support the hypothesis that rater fatigue causes DRIFT, and considered ‘warm-up’ as an alternative explanation for DRIFT by repeating our analysis after excluding the first two OSCE stations. Results Time-slot was positively associated with rating on a station (regression coefficient 0.88, 95% confidence interval [CI] 0.38–1.38; P = 0.001). There was an interaction between time-slot and station difficulty: for the more difficult stations the regression coefficient for time-slot was 1.24 (95% CI 0.55–1.93; P = 0.001) compared with 0.52 (95% CI − 0.08 to 1.13; P = 0.09) for the less difficult stations. Removing the first two stations from our analyses did not correct DRIFT. Conclusions Systematic biases, such as DRIFT, may compromise internal validity in an OSCE. Further work is needed to confirm this finding and to explore whether DRIFT also affects ratings on summative OSCEs. If confirmed, the factors contributing to DRIFT, and ways to reduce these, should then be explored.

Many people are afflicted with chronic diseases, in which nutrition plays a key role. The need for greater nutrition training among physicians, particularly gastroenterologists, is becoming increasingly evident.To determine the nutritional knowledge and perceived nutrition knowledge of gastroenterology fellows.Thirty-two gastrointestinal (GI) fellows currently enrolled in a GI fellowship program completed a needs assessment evaluating perceived nutrition knowledge and interest in the areas of nutrition support, assessment, obesity, micro/macronutrients, and nutrition in GI diseases. Additionally, an examination evaluating nutrition knowledge specific to gastroenterology fellows was administered.Thirty-two GI fellows completed the needs assessment. Cronbach alpha of the needs assessment instrument was 0.72, indicating satisfactory internal consistency reliability. GI fellows perceived themselves to have the least knowledge in obesity and micro/macronutrients. They indicated a perceived greater knowledge base in nutrition assessment. The mean total test score was 50.04% (SD=7.84%). Fellows had the highest score in the subscale of nutrition assessment (80.64%; SD=19.05%), which was significantly higher than scores obtained in nutrition support (49.45%; SD=11.98%; P0.05), micro/macronutrients (37.84%; SD=16.94%; P0.05), obesity (40.11%; SD=20.00%; P0.05), and nutrition in GI diseases (65.05%; SD=22.09%; P0.05). A backward linear regression including hours of nutrition education received during GI fellowship, hours of nutrition education received during medical school, and year of GI fellowship accounted for 22.7% of the variance in test performance (multiple R=0.477).Gastroenterology fellows think their knowledge of nutrition is suboptimal; objective evaluation of nutrition knowledge in this cohort confirmed this belief. A formal component of nutrition education could be developed in the context of GI fellowship education and continuing medical education as necessary.

As society's expectations of physicians change, so must the objectives of training. Professional organizations involved in training now emphasize multiplicity of roles. But how well do we evaluate these multiple roles?To investigate the principal components of evaluation in the Internal Medicine clerkship rotation at the University of Calgary.We performed factor analysis on all evaluation components in the Internal Medicine clerkship rotation, including the in-training evaluation report (ITER), objective structured clinical examination (OSCE), and multiple choice questions (MCQ) examination.We identified three principal components: information processing, professionalism, and declarative knowledge. Both the OSCE and MCQ loaded on a single factor, declarative knowledge. The nine items on the ITER loaded on two factors-information processing and professionalism.Despite using 11 evaluation items on three tools, we identified only three principal components of evaluation. Both our MCQ and OSCE appeared to measure declarative knowledge. The latter may be due to the fact that we use standardized patients without clinical findings-such that evaluations are primarily based upon the demonstration of examination routines. Reasons for the lack of discriminant validity of our ITER include overlapping attributes and constant errors, including a halo effect and an error of leniency.

Clinical reasoning is dependent upon working memory (WM). More precisely, during the clinical reasoning process stored information within long-term memory is brought into WM to facilitate the internal deliberation that affords a clinician the ability to reason through a case. In the present study, we examined the relationship between clinical reasoning and WM while participants read clinical cases with functional magnetic resonance imaging (fMRI). More specifically, we examined the impact of clinical case difficulty (easy, hard) and clinician level of expertise (2nd year medical students, senior gastroenterologists) on neural activity within regions of cortex associated with WM (i.e., the prefrontal cortex) during the reasoning process. fMRI was used to scan ten second-year medical students and ten practicing gastroenterologists while they reasoned through sixteen clinical cases [eight straight forward (easy) and eight complex (hard)] during a single 1-h scanning session. Within-group analyses contrasted the easy and hard cases which were then subsequently utilized for a between-group analysis to examine effects of expertise (novice > expert, expert > novice). Reading clinical cases evoked multiple neural activations in occipital, prefrontal, parietal, and temporal cortical regions in both groups. Importantly, increased activation in the prefrontal cortex in novices for both easy and hard clinical cases suggests novices utilize WM more so than experts during clinical reasoning. We found that clinician level of expertise elicited differential activation of regions of the human prefrontal cortex associated with WM during clinical reasoning. This suggests there is an important relationship between clinical reasoning and human WM. As such, we suggest future models of clinical reasoning take into account that the use of WM is not consistent throughout all clinical reasoning tasks, and that memory structure may be utilized differently based on level of expertise.

Evolution from novice to expert is associated with the development of expert-type knowledge structure. The objectives of this study were to examine reliability and validity of concept sorting (ConSort©) as a measure of static knowledge structure and to determine the relationship between concepts in static knowledge structure and concepts used during diagnostic reasoning. ConSort© was used to identify static knowledge concepts and analysis of think-aloud protocols was used to identify dynamic knowledge concepts (used during diagnostic reasoning). Intra- and inter-rater reliability, and correlation across cases, were evaluated. Construct validity was evaluated by comparing proportions of nephrologists and students with expert-type knowledge structure. Sensitivity and specificity of static knowledge concepts as a predictor of dynamic knowledge concepts were estimated. Thirteen first-year medical students and 19 nephrologists participated. Intra- and inter-rater agreement for determination of static knowledge concepts were 1.0 and 0.90, respectively. Reliability across cases was 0.45. The proportions of nephrologists and students identified as having expert-type knowledge structure were 82.9% and 55.8%, respectively (p=0.001). Sensitivity and specificity of ConSort© in predicting concepts that were used during diagnostic reasoning were 96.8% and 27.8% for nephrologists and 87.2% and 55.1% for students. ConSort© is a reliable, valid and sensitive tool for studying static knowledge structure. The applicability of tools that evaluate static knowledge structure should be explored as an addition to existing tools that evaluate dynamic tasks such as diagnostic reasoning.

BACKGROUND: Magnetic endoscopic imaging represents a recent advance in colonoscopy training. This technique provides adjunct information to the endoscopist, specifically with regard to colonoscope loop formation.OBJECTIVE: To examine the effect of a magnetic endoscopic imager on novice performance and workload in colonoscopy.METHODS: Twenty complete novices received an introductory teaching session followed by the completion of two procedures on a colonoscopy model. One-half of the participants performed their first procedure with the imager, and the second procedure without, while the other one-half were trained with the inverse sequence. Two main outcome measures were recorded: distance achieved and total workload as measured by the National Aeronautics and Space Administration task load index tool.RESULTS: A significant improvement was noted between the first and second colonoscopies, with the best performance recorded for participants who performed their first procedure with the imager, and their second without. The imager did not significantly change the total workload.DISCUSSION: The study participants paid attention to the magnetic endoscopic imager; however, this did not translate into a measurable increase in novice workload. A delayed learning benefit was conferred to the group exposed to the imager on their first colonoscopy, suggesting that, even at an early training stage, the additional imager information entered working memory and was processed in a useful fashion. The introductory teaching strategy used in the present study was successful as judged by the overall distance achieved and performance improvement seen in all study participants.

Diagnosing is a knowledge-based skill: to diagnose one must retrieve knowledge from long-term memory and then apply this to a new clinical problem. Prior research on expertise found differences in knowledge structure between experts and novices, and it is assumed that the superior diagnostic performance of experts is somehow related to their superior knowledge structure. Here our objective was to study knowledge structure in final year medical students and to examine the association between knowledge structure and diagnostic performance. Ninety-one students participated. We used concept sorting to assess knowledge structures for four clinical problems. We performed qualitative analysis of knowledge structures, categorizing these as either problem-specific, where knowledge was predominantly structured around concepts specific to that clinical problem, or generic, where knowledge was structured around general concepts that could apply to all clinical problems. We evaluated diagnostic performance using problem-solving questions. Knowledge structure varied between different problems, but for each problem most students had problem-specific knowledge structure. These students had better diagnostic performance than those with generic structure (68.5 vs. 55.3%, d = 0.45, P = 0.004). This difference persisted after adjusting for overall medical knowledge (performance on the Medical Council of Canada Part 1 examination) and clinical problem. We found that most students organize their knowledge around problem-specific concepts, and that this type of knowledge was associated with better diagnostic performance. This may be due to easier knowledge retrieval if there is congruence between how knowledge is stored and how it is applied when diagnosing.

The relation between knowledge structure and diagnostic performance is unclear. Similarly, variables affecting knowledge structure are poorly understood.The 1st objective was to examine the relation between concepts in knowledge structure and diagnostic performance. The 2nd objective was to examine the relation between the use of diagnostic schemes by small-group preceptors and knowledge structure of medical students.This was a cross-sectional study of 1st-year medical students in 4 clinical presentations: hyponatremia, hyperkalemia, metabolic acidosis, and metabolic alkalosis. The 1st dependent variable was diagnostic success with the number of expert-type concepts in knowledge structure (determined by concept sorting), diagnostic scheme use by preceptors, and clinical presentation as independent variables. The 2nd dependent variable was the number of expert-type concepts in knowledge structure with diagnostic scheme use by preceptors and clinical presentation as independent variables. Data were analyzed using multiple logistic and linear regression.Thirty 1st-year medical students participated. After adjusting for clinical presentation and scheme use by preceptors, the number of expert-type concepts in knowledge structure was associated with increased odds of diagnostic success (odds ratio 1.18 [1.03, 1.35], p = .016). After adjustment for clinical presentation, scheme use by preceptors was associated with increased number of expert-type concepts in knowledge structure (2.22 vs. 1.86, p = .01, d = 0.23).The number of expert-type concepts in knowledge structure is associated with increased odds of diagnostic success. Scheme use by small-group preceptors is associated with an increased number of expert-type concepts in knowledge structure.

Background To reduce inter-rater variability in evaluations and the demand on physician time, standardized patients (SP) are being used as examiners in OSCEs. There is concern that SP have insufficient training to provide valid evaluation of student competence and/or provide feedback on clinical skills. It is also unknown if SP ratings predict student competence in other areas. The objectives of this study were: to examine student attitudes towards SP examiners; to compare SP and physician evaluations of competence; and to compare predictive validity of these scores, using performance on the multiple choice questions examination (MCQE) as the outcome variable. Methods This was a cross-sectional study of third-year medical students undergoing an OSCE during the Internal Medicine clerkship rotation. Fifty-two students rotated through 8 stations (6 physician, 2 SP examiners). Statistical tests used were Pearson's correlation coefficient, two-sample t-test, effect size calculation, and multiple linear regression. Results Most students reported that SP stations were less stressful, that SP were as good as physicians in giving feedback, and that SP were sufficiently trained to judge clinical skills. SP scored students higher than physicians (mean 90.4% +/- 8.9 vs. 82.2% +/- 3.7, d = 1.5, p < 0.001) and there was a weak correlation between the SP and physician scores (coefficient 0.4, p = 0.003). Physician scores were predictive of summative MCQE scores (regression coefficient = 0.88 [0.15, 1.61], P = 0.019) but there was no relationship between SP scores and summative MCQE scores (regression coefficient = -0.23, P = 0.133). Conclusion These results suggest that SP examiners are acceptable to medical students, SP rate students higher than physicians and, unlike physician scores, SP scores are not related to other measures of competence.

Background The influence of intended and informal curricula on examination preparation has not been extensively studied. This study aims to firstly describe how students utilized components of intended and informal curricula to guide examination preparation, and secondly to study the relationship between examination preparation and performance. Methods Students received a pre-examination questionnaire to identify components from the intended curriculum (objectives and examination blueprint), and informal curriculum (content emphasised during lectures and small groups), used during examination preparation. Multiple logistic regression was used to study the relationship between these variables and student performance (above versus at or below average). Results Eighty-one students participated. There was no difference in the proportions using the examination blueprint, content emphasised during lectures, and content emphasised during small groups (87 – 93%) but fewer students used objectives (35%, p < 0.001). Objectives use was associated with reduced odds of above average examination performance (adjusted odds ratio 0.27 [0.07, 0.97], p = 0.04). Conclusion When preparing for the renal course examination, students were influenced at least as much by the informal as the intended curriculum. Of the two intended curriculum components, the examination blueprint appeared to be more widely used than the course objectives. This decreased use of objectives on examination preparation did not appear to have a detrimental effect on student performance.

Purpose Cognitive psychology research supports the notion that experts use mental frameworks or ‘schemes’, both to organize knowledge in memory and to solve clinical problems. The central purpose of this study was to determine the relationship between problem-solving strategies and the likelihood of diagnostic success. Methods Think-aloud protocols were collected to determine the diagnostic reasoning used by experts and non-experts when attempting to diagnose clinical presentations in gastroenterology. Results Using logistic regression analysis, the study found that there is a relationship between diagnostic reasoning strategy and the likelihood of diagnostic success. Compared to hypothetico-deductive reasoning, the odds of diagnostic success were significantly greater when subjects used the diagnostic strategies of pattern recognition and scheme-inductive reasoning. Two other factors emerged as independent determinants of diagnostic success: expertise and clinical presentation. Not surprisingly, experts outperformed novices, while the content area of the clinical cases in each of the four clinical presentations demonstrated varying degrees of difficulty and thus diagnostic success. Conclusions These findings have significant implications for medical educators. It supports the introduction of ‘schemes’ as a means of enhancing memory organization and improving diagnostic success.

Background Pencil-and-paper examination formats, and specifically the standard, five-option multiple-choice question, have often been questioned as a means for assessing higher-order clinical reasoning or problem solving. This study firstly investigated whether two paper formats with differing number of alternatives (standard five-option and extended-matching questions) can test problem-solving abilities. Secondly, the impact of the alternatives number on psychometrics and problem-solving strategies was examined. Methods Think-aloud protocols were collected to determine the problem-solving strategy used by experts and non-experts in answering Gastroenterology questions, across the two pencil-and-paper formats. Results The two formats demonstrated equal ability in testing problem-solving abilities, while the number of alternatives did not significantly impact psychometrics or problem-solving strategies utilized. Conclusions These results support the notion that well-constructed multiple-choice questions can in fact test higher order clinical reasoning. Furthermore, it can be concluded that in testing clinical reasoning, the question stem, or content, remains more important than the number of alternatives.