Search

Background and Aims: Colonoscopy performance is typically assessed by a supervisor in the clinical setting. There are limitations of this approach, however, because it allows for rater bias and increases supervisor workload demand during the procedure. Video-based assessment of recorded procedures has been proposed as a complementary means by which to assess colonoscopy performance. This study sought to investigate the reliability, validity, and feasibility of video-based assessments of competence in performing colonoscopy compared with live assessment., Methods: Novice (<50 previous colonoscopies), intermediate (50-500), and experienced (>1000) endoscopists from 5 hospitals participated. Two views of each colonoscopy were videotaped: an endoscopic (intraluminal) view and a recording of the endoscopist's hand movements. Recorded procedures were independently assessed by 2 blinded experts using the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT), a validated procedure-specific assessment tool comprising a global rating scale (GRS) and checklist (CL). Live ratings were conducted by a non-blinded expert endoscopist. Outcomes included agreement between live and blinded video-based ratings of clinical colonoscopies, intra-rater reliability, inter-rater reliability and discriminative validity of video-based assessments, and perceived ease of assessment., Results: Forty endoscopists participated (20 novices, 10 intermediates, and 10 experienced). There was good agreement between the live and video-based ratings (total, intra-class correlation [ICC] = 0.847; GRS, ICC = 0.868; CL, ICC = 0.749). Intra-rater reliability was excellent (total, ICC = 0.99; GRS, ICC = 0.99; CL, ICC = 0.98). Inter-rater reliability between the 2 blinded video-based raters was high (total, ICC = 0.91; GRS, ICC = 0.918; CL, ICC = 0.862). GiECAT total, GRS, and CL scores differed significantly among novice, intermediate, and experienced endoscopists (P < .001). Video-based assessments were perceived as "fairly easy," although live assessments were rated as significantly easier (P < .001)., Conclusions: Video-based assessments of colonoscopy procedures using the GiECAT have strong evidence of reliability and validity. In addition, assessments using videos were feasible, although live assessments were easier., (Copyright © 2018 American Society for Gastrointestinal Endoscopy. Published by Elsevier Inc. All rights reserved.)

Background: The validity of high-stakes decisions derived from assessment results is of primary concern to candidates and certifying institutions in the health professions. In the field of orthopaedic manual physical therapy (OMPT), there is a dearth of documented validity evidence to support the certification process particularly for short-answer tests. To address this need, we examined the internal structure of the Case History Assessment Tool (CHAT); this is a new assessment rubric developed to appraise written responses to a short-answer test of clinical reasoning in post-graduate OMPT certification in Canada., Methods: Fourteen physical therapy students (novices) and 16 physical therapists (PT) with minimal and substantial OMPT training respectively completed a mock examination. Four pairs of examiners (n = 8) participated in appraising written responses using the CHAT. We conducted separate generalizability studies (G studies) for all participants and also by level of OMPT training. Internal consistency was calculated for test questions with more than 2 assessment items. Decision studies were also conducted to determine optimal application of the CHAT for OMPT certification., Results: The overall reliability of CHAT scores was found to be moderate; however, reliability estimates for the novice group suggest that the scale was incapable of accommodating for scores of novices. Internal consistency estimates indicate item redundancies for several test questions which will require further investigation., Conclusion: Future validity studies should consider discriminating the clinical reasoning competence of OMPT trainees strictly at the post-graduate level. Although rater variance was low, the large variance attributed to error sources not incorporated in our G studies warrant further investigations into other threats to validity. Future examination of examiner stringency is also warranted.

Objectives: Validated assessment tools are required to support competency-based education. We aimed to assess the reliability and validity of the Gastrointestinal Endoscopy Competency Assessment Tool for Pediatric Colonoscopy (GiECATKIDS), an instrument developed by 41 North American experts using Delphi methodology., Methods: GiECATKIDS consists of a 7-item global rating scale (GRS) and an 18-item checklist (CL). An attending physician assessed 104 colonoscopies performed at 3 North American hospitals by 56 endoscopists, including 25 novices (<50 previous procedures), 21 intermediates (50-250), and 10 advanced endoscopists (>500). Another observer rated procedures to assess interrater reliability using intraclass correlation coefficient (ICC). Test-retest reliability was measured with ICC comparing endoscopists' first and second procedure scores. Discriminative validity was examined by comparing experience level with scores. Concurrent validity was assessed by correlating scores with colonoscopy experience, cecal and terminal ileal intubation rates, and physician global assessment., Results: Interrater reliability of the GiECATKIDS was high (total: ICC = 0.88; GRS: ICC = 0.79; CL: ICC = 0.89). Test-retest reliability was excellent (total: ICC = 0.94; GRS: ICC = 0.94; CL: ICC = 0.84). GiECATKIDS total, GRS, and CL scores differed significantly among novice, intermediate, and advanced endoscopists (P < 0.001). There was a significant positive correlation (P < 0.001) between scores and number of previous colonoscopies (total: ρ = 0.91, GRS: ρ = 0.92, CL: ρ = 0.84), cecal intubation rate (total: ρ = 0.82, GRS: ρ = 0.85, CL: ρ = 0.77), ileal intubation rate (total: ρ = 0.82, GRS: ρ = 0.82, CL: ρ = 0.80), and physician global assessment (total: ρ = 0.95, GRS: ρ = 0.94, CL: ρ = 0.89)., Conclusions: The GiECATKIDS demonstrates strong reliability and validity as a measure of performance of pediatric colonoscopy that can be used to support training and assessment.

Context: In health professions education, tests have traditionally been used to assess the skills and knowledge of learners. More recently, research in psychology and education has shown that tests can also be used to enhance student memory; a phenomenon called the 'testing effect'. Much of the research in this domain has focused on enhancing rote memory of simple facts, and not on the deeper comprehension and application of complex theoretical knowledge necessary to diagnose and manage patients. The purpose of this study was to examine the effects of testing on students' comprehension of the basic science mechanisms and diagnostic accuracy., Methods: Undergraduate dental and dental hygiene students (n = 112) were taught the radiographic features and pathophysiology underlying four intrabony abnormalities. Participants were divided into two groups: the test-enhanced (TE) condition and the study (ST) condition. Following the learning phase, the TE condition group completed an interventional test that tested the basic science mechanisms and the ST condition group was given additional study passages without being tested. Participants in both groups then completed a diagnostic test and a memory test immediately after the learning phase and 1 week later., Results: A main effect of testing condition was found. The participants in the TE condition group outperformed those in the ST condition group on immediate and delayed testing. Unlike the diagnostic test, the memory test showed no difference between the groups., Conclusion: The inclusion of the basic science test appears to have improved the students' understanding of the underlying disease mechanisms learned and also improved their performance on a test of diagnostic accuracy., (© 2014 John Wiley & Sons Ltd.)

Context:   Video-based observational practice can extend simulation-based learning outside the training space. This study explores the value of collaborative feedback provided during observational practice to the acquisition of clinical skills., Methods:   Nursing students viewed a video demonstrating the proper ventrogluteal injection technique before performing a videotaped pre-test trial on a simulator. They were then assigned randomly to one of three observational practice groups: a group that observed the expert demonstration (EO group); a group that viewed the expert demonstration, self-assessed their individual pre-test and contrasted their self-assessments with expert feedback (ESO group), and a group that observed the expert demonstration, self-assessed and contrasted their assessments with those of an expert, and formed a community that engaged in peer-to-peer feedback (ESPO group). The observation of all videos, the provision of assessments and all networking occurred via an Internet-mediated network. After 2 weeks, participants returned for post-tests and transfer tests., Results:   The pre-test-post-test analyses revealed significant interactions (global rating scale: F((2,22)) =4.00 [p =0.033]; checklist: F((2,22)) =4.31 [p =0.026]), which indicated that post-test performance in the ESPO group was significantly better than pre-test performance. The transfer analyses revealed main effects for both the global rating scale (F((2,23)) =6.73; p =0.005) and validated checklist (F((2,23)) =7.04; p =0.004) measures. Participants in the ESPO group performed better on the transfer test than those in the EO group., Conclusions:   The results suggest that video-based observational practice can be effective in extending simulation-based learning, but its effectiveness is mediated by the amount of time the learner spends engaged in the practice and the type of learning activities the learner performs in the observational practice environment. We speculate that increasing collaborative interactivity supports observational learning by increasing the extent to which the educational environment can accommodate learners' specific needs., (© Blackwell Publishing Ltd 2012.)

This pilot study explored the use of tensiometry as a measure of retention of knot tying skills. Medical students learned a one-handed square knot tying technique. Their final performances were video recorded and these videos were uploaded on to a website. Students were divided into two groups: an observational learning group that had access to videos before a retention test, and a control group that did not. After a two-week retention period, all students came back and performed one more trial to test the amount of retention of the skill. Tensiometry was used to measure strengths of the knots before and after the retention period. The scores showed no significant difference between the groups (p&gt;0.308) or tests (p&gt;0.737). We interpret the results to suggest that tensiometry is not sensitive enough to detect degradation in the performance of fundamental clinical skills as they are forgotten after being taught.

Aim: In this paper, we tested the over-arching hypothesis that progressive self-guided learning offers equivalent learning benefit vs. proficiency-based training while limiting the need to set proficiency standards., Background: We have shown that self-guided learning is enhanced when students learn on simulators that progressively increase in fidelity during practice. Proficiency-based training, a current gold-standard training approach, requires achievement of a criterion score before students advance to the next learning level., Methods: Baccalaureate nursing students (n = 15/group) practised intravenous catheterization using simulators that differed in fidelity (i.e. students' perceived realism). Data were collected in 2008. Proficiency-based students advanced from low- to mid- to high-fidelity after achieving a proficiency criterion at each level. Progressive students self-guided their progression from low- to mid- to high-fidelity. Yoked control students followed an experimenter-defined progressive practice schedule. Open-ended students moved freely between the simulators. One week after practice, blinded experts evaluated students' skill transfer on a standardized patient simulation. Group differences were examined using analyses of variance., Results: Proficiency-based students scored highest on the high-fidelity post-test (effect size = 1.22). An interaction effect showed that the Progressive and Open-ended groups maintained their performance from post-test to transfer test, whereas the Proficiency-based and Yoked control groups experienced a significant decrease (P < 0.05). Surprisingly, most Open-ended students (73%) chose the progressive practice schedule., Conclusion: Progressive training and proficiency-based training resulted in equivalent transfer test performance, suggesting that progressive students effectively self-guided when to transition between simulators. Students' preference for the progressive practice schedule indicates that educators should consider this sequence for simulation-based training.

Purpose: To evaluate the effectiveness of a novel, simulation-based educational model rooted in scaffolding theory that capitalizes on a systematic progressive sequence of simulators that increase in realism (i.e., fidelity) and information content., Method: Forty-five medical students were randomly assigned to practice intravenous catheterization using high-fidelity training, low-fidelity training, or progressive training from low to mid to high fidelity. One week later, participants completed a transfer test on a standardized patient simulation. Blinded expert raters assessed participants' global clinical performance, communication, procedure documentation, and technical skills on the transfer test. Participants' management of the resources available during practice was also recorded. Data were analyzed using multivariate analysis of variance. The study was conducted in fall 2008 at the University of Toronto., Results: The high-fidelity group scored higher (P < .05) than the low-fidelity group on all measures except procedure documentation. The progressive group scored higher (P < .05) than other groups for documentation and global clinical performance and was equivalent to the high-fidelity group for communication and technical skills. Total practice time was greatest for the progressive group; however, this group required little practice time on the resource-intensive high-fidelity simulator., Conclusions: Allowing students to progress in their practice on simulators of increasing fidelity led to superior transfer of a broad range of clinical skills. Further, this progressive group was resource-efficient, as participants concentrated on lower fidelity and lower resource-intensive simulators. It is suggested that clinical training curricula incorporate exposure to multiple simulators to maximize educational benefit and potentially save educator time.

Introduction: This study examined the validity of tensiometry as an objective method of assessing the improvements in the quality of knots and technical performance due to practice on suturing and knot-tying skills., Methods: We evaluated the quality of 1,830 knots produced by undergraduate medical students before training (pre-test), after (post-test), and after a week retention (retention-test) using tensiometry. These results were compared to expert-based assessments of products and performance., Results: Tensiometry, as well as expert-based assessments can detect changes in the quality of knots and technical performance from pre to post and retention-test ( p<0.05). However, these three methods may assess different dimensions of the same products and actions., Conclusions: The development of trainee evaluation criteria for the acquisition of surgical skills is a crucial element in surgical education. Based on our patterns of results it is hypothesized that tensiometry can serve as an economic first approximation of the quality of knots and suturing performance.

This study is the first phase of research aimed at developing new educational approaches to enhance occupational therapy students' orthosis fabrication skills. Before the effectiveness of training can be determined, a method for evaluating performance must be established. Using the Delphi method, we developed a global rating scale and checklist for evaluating technical competence when fabricating metacarpophalangeal (MCP) joint-stabilizing orthoses. To determine the reliability and validity of these tools, three evaluators used them to assess and score orthotic fabrication performance by experienced and student occupational therapists. The results suggest that these measurement tools are valid and reliable indicators of the technical skills involved in fabricating an MCP joint-stabilizing orthosis. Future studies should focus on building on these tools to evaluate communication skills, technical skills for making other types of orthoses, and effectiveness of training programs. [ABSTRACT FROM AUTHOR]