Search

INTRODUCTION: Generative artificial intelligence (AI) has proven to be a powerful tool with increasing applications in clinical care and medical education. ChatGPT has performed adequately on many specialty certification and knowledge assessment exams. The objective of this study was to assess the performance of ChatGPT 4 on a multiple-choice exam meant to simulate the Canadian urology board exam. METHODS: Graduating urology residents representing all Canadian training programs gather yearly for a mock exam that simulates their upcoming board-certifying exam. The exam consists of written multiple-choice questions (MCQs) and an oral objective structured clinical examination (OSCE). The 2022 exam was taken by 29 graduating residents and was administered to ChatGPT 4. RESULTS: ChatGPT 4 scored 46% on the MCQ exam, whereas the mean and median scores of graduating urology residents were 62.6%, and 62.7%, respectively. This would place ChatGPT's score 1.8 standard deviations from the median. The percentile rank of ChatGPT would be in the sixth percentile. ChatGPT scores on different topics of the exam were as follows: oncology 35%, andrology/benign prostatic hyperplasia 62%, physiology/anatomy 67%, incontinence/female urology 23%, infections 71%, urolithiasis 57%, and trauma/reconstruction 17%, with ChatGPT 4's oncology performance being significantly below that of postgraduate year 5 residents. CONCLUSIONS: ChatGPT 4 underperforms on an MCQ exam meant to simulate the Canadian board exam. Ongoing assessments of the capability of generative AI is needed as these models evolve and are trained on additional urology content. [ABSTRACT FROM AUTHOR]

Context.--Artificial intelligence algorithms hold the potential to fundamentally change many aspects of society. Application of these tools, including the publicly available ChatGPT, has demonstrated impressive domain-specific knowledge in many areas, including medicine. Objectives.--To understand the level of pathology domain-specific knowledge for ChatGPT using different underlying large language models, GPT-3.5 and the updated GPT-4. Design.--An international group of pathologists (n = 15) was recruited to generate pathology-specific questions at a similar level to those that could be seen on licensing (board) examinations. The questions (n = 15) were answered by GPT-3.5, GPT-4, and a staff pathologist who recently passed their Canadian pathology licensing exams. Participants were instructed to score answers on a 5-point scale and to predict which answer was written by ChatGPT. Results.--GPT-3.5 performed at a similar level to the staff pathologist, while GPT-4 outperformed both. The overall score for both GPT-3.5 and GPT-4 was within the range of meeting expectations for a trainee writing licensing examinations. In all but one question, the reviewers were able to correctly identify the answers generated by GPT-3.5. Conclusions.--By demonstrating the ability of ChatGPT to answer pathology-specific questions at a level similar to (GPT-3.5) or exceeding (GPT-4) a trained pathologist, this study highlights the potential of large language models to be transformative in this space. In the future, more advanced iterations of these algorithms with increased domain-specific knowledge may have the potential to assist pathologists and enhance pathology resident training. [ABSTRACT FROM AUTHOR]

"The only constant is change" -- Heraclitus, Greek Philosopher The objective of this article, Part 1 of a two part series, is to provide a narrative review of the evolution of teaching chiropractic manual skills by four tutors who taught in the technique trenches at the Canadian Memorial Chiropractic College collectively for 120 years. Based on their collective memories, this narrative review describes the evolution of: central demonstrations; inconsistency between tutors with respect to demonstrating and grading chiropractic manual skills; determining course content; policy on students providing high velocity, low amplitude thrusts on each other during class time and testing; quantitative versus qualitative grading; remediation; acknowledgment of risk and; changes to technique class due to Covid. The results of a unique survey evaluating students' perception of these changes is presented. The intent of this article is for faculty at other accredited educational programs to learn from our experiences and potentially strengthen their pedagogical approach to teaching chiropractic manual skills. [ABSTRACT FROM AUTHOR]

Purpose: Learning is optimised when postgraduate trainees engage in clinical tasks in their zone of proximal development (ZPD). However, workplace learning environments impose additional non‐learning goals and additional tasks that may lead to trainees engaging in tasks that do not fall within their ZPD. We do not fully understand how trainees select clinical tasks in the workplace learning environment. If we knew the goals and factors they consider when selecting a task, we could better equip trainees with strategies to select tasks that maximise learning. We explored how postgraduate trainees select clinical tasks using echocardiography interpretation as a model. Methods: Canadian General Cardiology residents and Echocardiography fellows were invited to participate in semi‐structured interviews. Aligning with a theory‐informed study, two independent researchers used a deductive, directed content analysis approach to identify codes and themes. Results: Eleven trainees from seven Canadian universities participated (PGY4 = 4, PGY5 = 3, PGY6 = 1 and echocardiography fellows = 3). Goals included learning content, fulfilling assessment criteria and contributing to clinical demands. Trainees switched between goals throughout the day, as it was too effortful for them to engage in tasks within their ZPD at all times. When trainees had sufficient mental effort available, they selected higher complexity tasks that could advance learning content. When available mental effort was low, trainees selected less complex tasks that fulfilled numerically based assessment goals or contributed to clinical demands. Trainees predominantly used perceived complexity of the echocardiogram as a factor to select tasks to achieve their desired goals. Conclusion: Postgraduate trainees select tasks within their ZPD that enable them to maximise learning when they perceive to have sufficient mental effort available and workplace affordances are adequate. These findings can inform individual and systemic strategies to maximise learning when selecting tasks. Furthering understanding of workplace‐based learning, Blissett et al. demonstrate how trainees take on low complexity tasks to allow mental effort to recharge, thus facilitating maximal engagement with tasks within their Zone of Proximal Development. [ABSTRACT FROM AUTHOR]

Copyright of Physiotherapy Canada is the property of University of Toronto Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

INTRODUCTION: We sought to determine the satisfaction rates of examiners and candidates in a virtual Objective Structured Clinical Exam (OSCE) of graduating Canadian urology residents. METHODS: An annual mock exam, aimed at simulating the licencing urology exam for Canadian graduates, was moved to an online format for the 2020 cohort. This exam consists of an OSCE, and a written multiple-choice exam. The Telemedicine Satisfaction Questionnaire (TSQ), a previously validated tool for clinical encounters with three sub-domains (quality of care provided, similarity to face-to-face encounter, and perception of the interaction) was modified for the purposes of evaluating the OSCE encounter. The TSQ was sent electronically to all examiners and candidates after the exam. RESULTS: There were 14/16 responses from examiners (87.5%) and 24/39 responses from candidates (61.5%). Overall, the format was judged to be a good experience by 13/14 (92.9%) of examiners and 21/24 (87.5%) of candidates; however, when asked specifically if the virtual OSCE was an acceptable way to determine a candidate's competency to practice urology independently, only 8/14 (57.1%) of examiners and 15/24 (62.5%) of candidates agreed. CONCLUSIONS: This study demonstrates an overall good satisfaction rate among both examiners and candidates when using a teleconference format for a mock OSCE. [ABSTRACT FROM AUTHOR]

Canada relies heavily on foreign-trained physicians. As a Federation, with health care being a Provincial jurisdiction, this often translates into varied processes that international medical graduates (IMGs) must undertake to obtain a Canadian medical license. Two decades ago, several government officials and representatives of many physician organizations, including regulatory bodies, met and proposed 6 recommendations to make the processes standardized, simpler, and more transparent to aid internationally trained physicians in their pursuit of Canadian medical licenses. The Medical Council of Canada (MCC) was one of the organizations at the 2002 meeting in Calgary, Alberta. As an organization responsible for the assessment of physicians' knowledge and skills and the issuant of the Licentiate of the MCC (LMCC), a prerequisite for Canadian medical license, the MCC was one of the institutions tasked with implementation of the recommendations. The purpose of this manuscript is to evaluate how well the recommendations were met. To do this, we explored whether the IMGs' journey to obtain Canadian medical licenses in 2022 is more challenging or less challenging than in 2002. The MCC's role in helping to effect changes in the licensing process was highlighted. [ABSTRACT FROM AUTHOR]

Copyright of Canadian Journal of Pathology is the property of Canadian Association of Pathologists and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Background: Nursing students in Canada are typically enrolled in a four-year bachelor degree program that provides students with the necessary skills and knowledge to enter a highly demanding and challenging workforce. Strong critical thinking, clinical reasoning, and clinical judgment skills are essential skills for safe nursing practice. Therefore, educational institutes and their mentors are mandated to teach and assess these skills. In addition, nursing programs operate under an apprenticeship model, which entails the fulfillment of practical experience during which students are expected to develop and refine their skills in critical thinking, clinical reasoning, and clinical judgment. Purpose: The purpose of this scoping review of the literature is to assess the available evidence of how higher-level thinking, including critical thinking, clinical reasoning, and clinical judgment are evaluated in undergraduate nursing students in clinical settings. Methods: The inclusion criteria consisted of quantitative research articles published in the last 10 years. Search databases accessed included CINAHL Plus (EBSCO), Medline, and PubMed. Results: Seven articles that fit the inclusion criteria became the focus of this scoping review. Four tools to evaluate higher-thinking processes in clinical settings were located and scrutinized: Lasater Clinical Judgment Rubric (LCJR), Script Concordance Testing, and Yoon's Critical Thinking Disposition Instrument. Relevance to Practice: The scoping review will provide direction and contextualize future studies that focus on the appraisal of nursing students' critical thinking, clinical reasoning, and clinical judgment in clinical settings. [ABSTRACT FROM AUTHOR]

Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts. After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis. A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation. [ABSTRACT FROM AUTHOR]

Copyright of Physiotherapy Canada is the property of University of Toronto Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Canadian Journal of Dental Hygiene is the property of Canadian Dental Hygienists Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Background: There are currently no reference standards for the development of competence in bronchoscopy.Research Question: The aims of this study were to (1) develop learning curves for bronchoscopy skill development and (2) estimate the number of bronchoscopies required to achieve competence.Study Design and Methods: Trainees from seven North American academic centers were enrolled at the beginning of their pulmonology training. Performance during clinical bronchoscopies was assessed by supervising physicians using the Ontario Bronchoscopy Assessment Tool (OBAT). Group-level learning curves were modeled using a quantile regression growth model, where the dependent variable was the mean OBAT score and the independent variable was the number of bronchoscopies performed at the time the OBAT was completed.Results: A total of 591 OBAT assessments were collected from 31 trainees. The estimated regression quantiles illustrate significantly different learning curves based on trainees' performance percentiles. When competence was defined as the mean OBAT score for all bronchoscopies rated as being completed without need for supervision, the mean OBAT score associated with competence was 4.54 (95% CI, 4.47-4.58). Using this metric, the number of bronchoscopies required to achieve this score varied from seven to 10 for the 90th percentile of trainees and from 109 to 126 for the lowest 10th percentile of trainees. When competence was defined as the mean OBAT score for the first independent bronchoscopy, the mean was 4.40 (95% CI, 4.20-4.60). On the basis of this metric, the number of bronchoscopies required varied from one to 11 for the 90th percentile of trainees and from 83 to 129 for the lowest 10th percentile of trainees.Interpretation: We were able to generate learning curves for bronchoscopy across a range of trainees and centers. Furthermore, we established the average number of bronchoscopies required for the attainment of competence. This information can be used for purposes of curriculum planning and allows a trainee's progress to be compared with an established norm. [ABSTRACT FROM AUTHOR]

This study is to introduce an effective and systematic accreditation and evaluation system of continuing medical education (CME) and continuing professional development (CPD) in Korea, suitable for the future development, and based on the review of the characteristics of well-recognized systems implemented in western countries such as the US and Canada. CME and CPD comprise programs that expand the clinical knowledge and skills required of physicians as well as educational activities aiming to maintain, develop, and increase knowledge, specialized skills, or performance standards for provision of better medical services. These include not only self-directed activities but also official activities, such as leadership activities, professional development, and evaluation of the practitioner's level of knowledge and competence. Recently, continuing education for doctors has emphasized CPD, centered around learners, in a departure from CME, centered around educators. Each CME/CPD program in the US and Canada has unique features. However, they share common features of self-directed learning, competency-based education, and evaluation. Although there are remain problems, the Continuing Medical Education Accreditation and Evaluation Council of the Korean Medical Association, as a unique accreditation authority for CME, is currently developing a new CME/CPD system, which would constitute an idealistic onesystem for Korean physicians oriented toward future development, designed to fulfill standards equivalent to that of the Accreditation Council for Continuing Medical Education or any other internationally well-recognized organization. [ABSTRACT FROM AUTHOR]

Based on the framework of testlet models, the current study suggests the Bayesian random block item response theory (BRB IRT) model to fit forced-choice formats where an item block is composed of three or more items. To account for local dependence among items within a block, the BRB IRT model incorporated a random block effect into the response function and used a Markov Chain Monte Carlo procedure for simultaneous estimation of item and trait parameters. The simulation results demonstrated that the BRB IRT model performed well for the estimation of item and trait parameters and for screening those with relatively low scores on target traits. As found in the literature, the composition of item blocks was crucial for model performance; negatively keyed items were required for item blocks. The empirical application showed the performance of the BRB IRT model was equivalent to that of the Thurstonian IRT model. The potential advantage of the BRB IRT model as a base for more complex measurement models was also demonstrated by incorporating gender as a covariate into the BRB IRT model to explain response probabilities. Recommendations for the adoption of forced-choice formats were provided along with the discussion about using negatively keyed items. [ABSTRACT FROM AUTHOR]

Context: Assessment for and of learning in workplace settings is at the heart of competency‐based medical education. In postgraduate medical education (PGME), entrustable professional activities (EPAs) and entrustment scales are increasingly used to assess competence. However, the educational impacts of these assessment approaches remain unknown. Therefore, this study aimed to explore trainee perceptions regarding the impacts of EPAs and entrustment scales on feedback and learning processes in the clinical setting. Methods: Four focus groups were conducted with postgraduate trainees in anaesthesia, emergency medicine, general internal medicine and nephrology at McMaster University in Hamilton, Ontario, Canada. Data collection and analysis were informed by principles of constructivist grounded theory. Results: Entrustable professional activities representing well‐defined tasks are perceived as potentially effective drivers for feedback and learning. Use of EPAs and entrustment scales, however, may augment existing tensions between developmental (for learning) and decision‐making (of learning) assessment functions. Three key dilemmas seem to influence the impact of EPA‐based assessment approaches on residents' learning: (a) standardisation of outcomes versus flexibility in assessment to align with individual learning experiences; (b) assessment tasks focusing on performance standards versus opportunities for learning, and (c) feedback focusing on numeric entrustment scores versus narrative and dialogue. Use of entrustment as an assessment outcome may impact trainees' motivation and feelings of self‐efficacy, further enhancing tensions between learning and performance. Conclusions: Entrustable professional activities and entrustment scales may support assessment for learning in PGME. However, their successful implementation requires the careful management of dilemmas that arise in EPA‐based assessment in order to support competence development. While it appears that entrustable professional activities and entrustment scales can support assessment for learning in postgraduate medical education, this study shows that key dilemmas need managed to ensure successful implementation. [ABSTRACT FROM AUTHOR]

Construct: The globalization of healthcare has been accentuated by the export of health professional curricula overseas. Yet intact translation of pedagogies and practices devised in one cultural setting may not be possible or necessarily appropriate for alternate environments. Purposeful examination of workplace learning is necessary to understand how the source or "home" program may need adapting in the distributed or "host" setting. Background: Strategies to optimize cross-border medical education partnerships have been largely focused on elements of campus-based learning. Determining how host clinical supervisors approach assessment in experiential settings within a different culture and uphold the standards of home programs is relevant given the influence of context on trainees' demonstrated competencies. In this mixed-methods study, we sought to explore assessor judgments of student workplace-based performance made by preceptors sharing a pharmacy curriculum in Canada and Qatar. Approach: Using modified Delphi consensus technique, we asked clinical supervisors in Canada (n = 18) and in Qatar (n = 14) to categorize trainee performance as described in 16 student vignettes. The proportion of ratings for three levels of expectation (exceeds, meets, or below) was calculated and within-country group consensus achieved if the level of agreement reached 80%. Between-country group comparisons were measured using a chi-square statistic. We then conducted follow-up semi-structured interviews to gain further perspectives and clarify assessor rationale. Transcripts were analyzed using thematic content analysis. Results: The threshold for between-country group differences in assessor impressions was met for only two of the 16 student vignettes. Compared to Canadian clinical supervisors, relatively more preceptors in Qatar judged one described student as meets rather than exceeds expectations and one as meets rather than falls below expectations. Analysis of follow-up interviews exploring how culture may inform variations in assessor judgments identified themes associated with the profession, organization, learner, and supervisor performance theories but not their particular geographic context. Clinical supervisors in both countries were largely aligned in expectations of student knowledge, skills, and behaviors demonstrated in patient care and multidisciplinary team interactions. Conclusions: Our study demonstrated that variation in student assessment was more frequent among clinical supervisors within the same national context than any differences identified between the two countries. In these program settings, national sociocultural norms did not predict global assessor impressions or competency-specific judgments; instead, professional and organizational cultures were more likely to inform student characterizations of performance in workplace-based settings. Further study situated within the specific experiential learning contexts of cross-border health professional curricula is assuredly warranted. [ABSTRACT FROM AUTHOR]

Background: A streamlined academic approach for career advancement is needed that allows practical or enrolled nurses to obtain a Bachelor of Nursing (BN) degree. One strategy in this approach is offering college-prepared Practical Nurses (PNs) the opportunity to transition into a baccalaureate program through a bridging course. Bridging initiatives serve as professional development opportunities for learners with personal growth and financial advantages on degree completion and enhance health and human resources for health care systems within national and international landscapes. Objective and methods: A curricular model and strategies on how such a bridging course can be constructed are discussed in this article. The model integrates teaching and learning strategies as well as course sequencing, structure, and assessment strategies. Results and conclusion: This innovative bridging curriculum offers Canadian and international nurse educators a programmatic guideline to create educational pathways for practical or enrolled nurses to obtain a baccalaureate degree. Completion of the bridging curriculum and BN program allow graduates to assume the RN role following a successful pass on the National Council Licensure Examination (NCLEX). [ABSTRACT FROM AUTHOR]

Copyright of Physiotherapy Canada is the property of University of Toronto Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Objectives: This qualitative study describes the social processes of evidence interpretation employed by Clinical Competency Committees (CCCs), explicating how they interpret, grapple with and weigh assessment data. Methods: Over 8 months, two researchers observed 10 CCC meetings across four postgraduate programmes at a Canadian medical school, spanning over 25 hours and 100 individual decisions. After each CCC meeting, a semi‐structured interview was conducted with one member. Following constructivist grounded theory methodology, data collection and inductive analysis were conducted iteratively. Results: Members of the CCCs held an assumption that they would be presented with high‐quality assessment data that would enable them to make systematic and transparent decisions. This assumption was frequently challenged by the discovery of what we have termed 'problematic evidence' (evidence that CCC members struggled to meaningful interpret) within the catalogue of learner data. When CCCs were confronted with 'problematic evidence', they engaged in lengthy, effortful discussions aided by contextual data in order to make meaning of the evidence in question. This process of effortful discussion enabled CCCs to arrive at progression decisions that were informed by, rather than ignored, problematic evidence. Conclusions: Small groups involved in the review of trainee assessment data should be prepared to encounter evidence that is uncertain, absent, incomplete, or otherwise difficult to interpret, and should openly discuss strategies for addressing these challenges. The answer to the problem of effortful processes of data interpretation and problematic evidence is not as simple as generating more data with strong psychometric properties. Rather, it involves grappling with the discrepancies between our interpretive frameworks and the inescapably subjective nature of assessment data and judgement. The authors of this qualitative study illustrate the processes of evidence interpretation Clinical Competency Committees (CCCs) use to grapple with and weigh assessment data. [ABSTRACT FROM AUTHOR]

Student-led peer-assisted mock objective structured clinical examinations (MOSCEs) have been used in various settings to help students prepare for subsequent higher-stakes, faculty-run OSCEs. MOSCE participants generally valued feedback from peers and reported benefits to learning. Our study investigated whether participation in a peer-assisted MOSCE affected subsequent OSCE performance. To determine whether mean OSCE scores differed depending on whether medical students participated in the MOSCE, we conducted a between-subjects analysis of variance, with cohort (2016 vs. 2017) and MOSCE participation (MOSCE vs. no MOSCE) as independent variables and the mean OSCE score as the dependent variable. Participation in the MOSCE had no influence on mean OSCE scores (P=0.19). There was a significant correlation between mean MOSCE scores and mean OSCE scores (Pearson r=0.52, P<0.001). Although previous studies described self-reported benefits from participation in student-led MOSCEs, it was not associated with objective benefits in this study. [ABSTRACT FROM AUTHOR]

There is nearly a century of educational research that has demonstrated that three option multiple-choice questions (MCQs) are as valid and reliable as four or five option, yet this format continues to be underutilized in educational institutions. This replication study was a quasi-experimental between groups research design conducted at three Canadian schools of nursing to examine the psychometric properties of three option MCQs when compared to the more traditional four option questions. Data analysis revealed that there were no statistically significant differences in the item discrimination, difficulty or mean examination scores when MCQs were administered with three versus four option answer choices. [ABSTRACT FROM AUTHOR]

Copyright of Physiotherapy Canada is the property of University of Toronto Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Introduction: In a script concordance test (SCT), examinees are asked to judge the effect of a new piece of clinical information on a proposed hypothesis. Answers are collected using a Likert-type scale (ranging from −2 to +2, with ‘0’ indicating no effect), and compared with those of a reference panel of ‘experts’. It has been argued, however, that SCT may be susceptible to the influences of gaming and guesswork. This study aims to address some of the mounting concern over the response process validity of SCT scores.Method: Using published datasets from three independent SCTs, we investigated examinee response patterns, and computed the score a hypothetical examinee would obtain on each of the tests if he 1) guessed random answers and 2) deliberately answered ‘0’ on all test items.Results: A simulated random guessing strategy led to scores 2 SDs below mean scores of actual respondents (Z-scores −3.6 to −2.1). A simulated ‘all-0’ strategy led to scores at least 1 SD above those obtained by random guessing (Z-scores −2.2 to −0.7). In one dataset, stepwise exclusion of items with modal panel response ‘0’ to fewer than 10% of the total number of test items yielded hypothetical scores 2 SDs below mean scores of actual respondents.Discussion: Random guessing was not an advantageous response strategy. An ‘all-0’ response strategy, however, demonstrated evidence of artificial score inflation. Our findings pose a significant threat to the SCT’s validity argument. ‘Testwiseness’ is a potential hazard to all testing formats, and appropriate countermeasures must be established. We propose an approach that might be used to mitigate a potentially real and troubling phenomenon in script concordance testing. The impact of this approach on the content validity of SCTs merits further discussion. [ABSTRACT FROM AUTHOR]

The article examines the social/racial biases of Professor Peter Sandiford of the University of Toronto in his intelligence testing of "Oriental" or Asian schoolchildren in the public schools in Vancouver, British Columbia during the 1920s. Topics covered include background of the Putman-Weir Survey of 1925, a survey of the Canadian school system under the direction of George Moir Weir and Harold J. Putman, social racism, and justification of Sandiford's racial prejudice by science.

Given the longstanding role of grades in education, and their increased use for high-stakes decisions including student mobility, admission, selection, and accountability, this paper presents a systematic review of grading policies across all 10 Canadian provinces and 3 territories. In total, 23 policies were inductively analyzed for their articulation of (a) the purposes of grades, (b) the methods used for generating grades, and (c) the relationship between grading and formative assessment. Our analysis revealed significant areas of consistency across Canada while also highlighting important areas of variation. Implications of these findings on the value and use of grades within and across educational systems in Canada are discussed. [ABSTRACT FROM AUTHOR]

Background and Objectives: In our family medicine residency program, we have established a culture of guided self-assessment through a systematic approach of direct observation of residents and documentation of formative feedback. We have observed that our residents have become more accurate in self-assessing their clinical performance. The objective of this study was to examine whether this improved accuracy extended to residents' self-assessment of their medical knowledge and clinical reasoning on the In-Training Examination (ITE).Methods: In November each year, residents in their first (PGY1) and second (PGY2) years of residency take the ITE (240 multiple-choice questions). Immediately before and right after taking the ITE, residents complete a questionnaire, self-assessing their knowledge and predicting their performances, overall and in eight high-level domains. Consented data from residents who took the ITE in 2009-2015 (n=380, 60% participation rate) were used in the Generalized Estimating Equations analyses.Results: PGY2 residents outperformed PGY1 residents; Canadian medical graduates consistently outperformed international medical graduates; urban and rural residents performed similarly overall. Residents' pre-post self-assessments were in line with residents' actual performance on the overall examination and in the domains of Adult Medicine and Care of Surgical Patients. The underperforming residents in this study accurately predicted both pre- and post-ITE that they would perform poorly.Conclusion: Our findings suggest that the ITE operates well in our program. There was a tendency among residents in this study to appropriately adjust their self-assessment of their overall performance after completing the ITE. Irrespective of the residency year, resident self-assessment was less accurate on individual domains. [ABSTRACT FROM AUTHOR]

Context Predictive validity studies on the use of the multiple mini-interview ( MMI) have been primarily in medicine. Objectives This study sought to determine the predictive validity of the MMI for performance within a pharmacy programme and on the Pharmacy Examining Board of Canada ( PEBC) Qualifying Examination for licensure, and to compare the predictive validity of the MMI with that of pre-pharmacy grade point average ( GPA) and Pharmacy College Admission Test ( PCAT) score. Methods Admissions data for 223 graduates of the pharmacy programme at the University of Toronto were matched to programme and licensure outcome measures. Multiple linear regression assessed the predictive ability of the MMI, pre-pharmacy GPA, PCAT and covariates for performance in final-year experiential rotations, cumulative GPA ( cGPA) and PEBC- MCQ (multiple-choice question examination) and PEBC- OSCE (objective structured clinical examination) overall and subcomponent scores. Results The PCAT, pre-pharmacy GPA and age significantly predicted the PEBC- MCQ overall score. The MMI was the only significant predictor of overall score on the PEBC- OSCE (β = 0.17, p = 0.02); it also predicted communication and performance subscores. Scores on the PCAT and female gender predicted the communication subscore. Pre-pharmacy GPA, age and female gender significantly predicted cGPA. The MMI was the only significant predictor of institutional/ambulatory rotation score (β = 0.26, p = 0.00). Conclusions The MMI, designed to measure non-academic attributes including communication, motivation and problem-solving skills, was the only admissions tool with significant predictive validity for performance on the PEBC- OSCE national pharmacy certification examination and in an institutional/ambulatory rotation. These findings, from a single cohort of undergraduates, provide the first report of the predictive validity of the MMI for performance on pharmacy licensure examinations and thereby strengthen the evidence for its use in health professions selection. Prior university academic performance significantly predicted cGPA and performance on the PEBC- MCQ. Performance on the PCAT also predicted PEBC- MCQ results. [ABSTRACT FROM AUTHOR]

We examined how positive and negative affect covary within individuals over time and how patterns of association between affective traits and states relate to academic success across 4 years of university. Participants were 187 full-time first-year students at a large Canadian university who completed questionnaires about recent affective experiences in 6 waves across 4 years. Grade point average for each year of study was provided by the registrar's office. Our analysis identified an adaptive pattern characterized by the maintenance of high positive affect ("chronic happiness") and the cooccurrence of time-limited bouts of negative affect. Our results are consistent with findings showing productive consequences of experiencing positive and negative affect in tandem and the development of emotion regulation capacity across the transition to adulthood. [ABSTRACT FROM AUTHOR]

Introduction: Maintaining the existing FLS test centers requires considerable investment in human and financial resources. It can also be particularly challenging for those outside of North America to become certified due to the limited number of international test centers. Preliminary work suggests that it is possible to reliably score the FLS manual skills component remotely using low-cost videoconferencing technology. Significant work remains to ensure that testing procedures adhere to standards defined by SAGES for this approach to be considered equivalent to standard on-site testing.Objective: To validate the integrity and validity of the FLS manual skills examination administered remotely in a real-world environment according to FLS testing protocols and to evaluate participants' experience with the setting.Methods: Individuals with various levels of training from the University of Toronto completed a pre- and a post-test questionnaire. Participants presented to one of the two FLS testing rooms available for the study, each connected via Skype to a separate room with a FLS proctor who administered and scored the test remotely (RP). An on-site proctor (OP) was present in the room as a control. An invigilator was also present in the testing room to follow directions from the RP and ensure the integrity of test materials.Results: Twenty-one participants were recruited, and 20 completed the test. There was no significant difference between scores by RP and OP. Interrater reliability between the RP and OP was excellent. One critical error was missed by the RP, but this would not have affected the test outcome. Participants reported being highly satisfied.Conclusion: We demonstrate that proctors located remotely can administer the FLS skills test in a secure and reliable fashion, with excellent interrater reliability compared to an on-site proctor. Remote proctoring of the FLS examination could become a strategy to increase certification rates while containing costs. [ABSTRACT FROM AUTHOR]

Although education scholars have examined the globalization effect of the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA) and its impact in several countries, few have explored its effect at the sub-national level. Taking the Canadian federation as its case study, I argue that the PISA, as a universalizing project for education, is being uncritically replicated through the implementation of student assessments at the national level. By drawing on the policy studies and policy sociology literature, I find evidence of policy discursive practices and techniques, which led to the creation and replication of a PISA-modeled assessment sub-nationally in the form of the Pan-Canadian Assessment Program. Three key themes emerge that facilitate the modeling of universalizing educational projects such as the PISA by sub-national entities: (1) a preoccupation with the international benchmarking of student performance, (2) a shift away from a curriculum-based assessment to a competency-based one, and (3) the adoption of organizational systems and processes of assessment aligned with supranational assessment practices. I suggest that domestic conditions in the Canadian federation were conducive to the rapid integration of the PISA sub-nationally despite the decentralized structure of the Canadian elementary and secondary education system. [ABSTRACT FROM AUTHOR]

Background: Simulated office orals (SOOs) are used by the College of Family Physicians of Canada as a method to evaluate family medicine resident readiness for clinical practice. The use of SOOs as a progress test would provide residency programs with useful information to determine resident readiness for challenging the certification exam. The data from a progress test could then be easily manipulated to generate a risk assessment plot. Methods: During a prospective cohort study conducted at the University of Ottawa, the feasibility of using practice SOO sessions, a structured clinical exam, as a progress test was explored. Twenty-three residents participated in all four practice SOO sessions and their results were entered into a risk assessment plot. Results: Repeated measures analysis of the data using ANOVA demonstrated that the residents’ scores at each time point were statistically different from each other, generating anF(3, 66) = 27.52,p < 0.001,η2 = 0.55.and that the relationship over time was linear with anF(1, 22) = 123.80,p < 0.001,η2 = 0.85. At the final time point, a risk assessment resulted in no learners mapping to quadrants III or IV. Conclusions: Our results demonstrate the feasibility of utilizing a SOO exam, a clinical exam, as a progress test. In addition, we propose generating a risk assessment plot, using the data from the Fall 2013 and Spring 2014 practice SOO sessions, as a means of identifying residents at risk. Further studies will be needed to confirm the utility of this analysis. Combined with other measures acquired during in-training evaluation, the utilization of practice SOOs as a progress test will provide program directors with valuable information on resident progression. [ABSTRACT FROM AUTHOR]

Background: Emerging evidence suggests that despite dedicated practice, not all surgical trainees have the ability to reach technical competency in minimally invasive techniques. While selecting residents that have the ability to reach technical competence is important, evidence to guide the incorporation of technical ability into selection processes is limited. Therefore, the purpose of the present study was to evaluate whether background experiences and 2D-3D visual spatial test results are predictive of baseline laparoscopic skill for the novice surgical trainee.Methods: First-year residents were studied. Demographic data and background surgical and non-surgical experiences were obtained using a questionnaire. Visual spatial ability was evaluated using the PicSOr, cube comparison (CC) and card rotation (CR) tests. Technical skill was assessed using the camera navigation (LCN) task and laparoscopic circle cut (LCC) task. Resident performance on these technical tasks was compared and correlated with the questionnaire and visual spatial findings.Results: Previous experience in observing laparoscopic procedures was associated with significantly better LCN performance, and experience in navigating the laparoscopic camera was associated with significantly better LCC task results. Residents who scored higher on the CC test demonstrated a more accurate LCN path length score (r s(PL) = -0.36, p = 0.03) and angle path (r s(AP) = -0.426, p = 0.01) score when completing the LCN task. No other significant correlations were found between the visual spatial tests (PicSOr, CC or CR) and LCC performance.Conclusion: While identifying selection tests for incoming surgical trainees that predict technical skill performance is appealing, the surrogate markers evaluated correlate with specific metrics of surgical performance related to a single task but do not appear to reliably predict technical performance of different laparoscopic tasks. Predicting the acquisition of technical skills will require the development of a series of evidence-based tests that measure a number of innate abilities as well as their inherent interactions. [ABSTRACT FROM AUTHOR]

This paper provides a detailed account of the Queensland education system's engagement with reforming curriculum, pedagogies and assessment. In so doing, it responds to the University College London's Institute of Education report on ‘high-performing’ jurisdictions, of which Queensland, Australia, was identified as one. In this report, nine issues and choices in relation to ‘instructional systems’ confronting a new education minister (or secretary) are considered. This paper details the strengths of the Queensland education system in relation to curriculum, pedagogy and assessment, whilst acknowledging that particular aspects of the global and national Australian contexts, alongside internal Queensland politics, can inhibit the ongoing work of the very processes that led to Queensland being identified in the Institute of Education report as a ‘high-performing jurisdiction’. We conclude by arguing that minsters of education need to take a longer view of educational reform than is currently the case and that quick fixes determined by electoral policy cycles need to be avoided. Learning about and from one's own system as well as from others requires a depth of analysis and time that is often not conducive to such policy cycles. [ABSTRACT FROM PUBLISHER]

The Medical College Admission Test (MCAT) is a quantitative metric used by MD and MD-PhD programs to evaluate applicants for admission. This study assessed the validity of the MCAT in predicting training performance measures and career outcomes for MD-PhD students at a single institution. The study population consisted of 153 graduates of the Vanderbilt Medical Scientist Training Program (combined MD-PhD program) who matriculated between 1963 and 2003 and completed dual-degree training. This population was divided into three cohorts corresponding to the version of the MCAT taken at the time of application. Multivariable regression (logistic for binary outcomes and linear for continuous outcomes) was used to analyze factors associated with outcome measures. The MCAT score and undergraduate GPA (uGPA) were treated as independent variables; medical and graduate school grades, time-to-PhD defense, USMLE scores, publication number, and career outcome were dependent variables. For cohort 1 (1963-1977), MCAT score was not associated with any assessed outcome, although uGPA was associated with medical school preclinical GPA and graduate school GPA (gsGPA). For cohort 2 (1978-1991), MCAT score was associated with USMLE Step II score and inversely correlated with publication number, and uGPA was associated with preclinical GPA (mspGPA) and clinical GPA (mscGPA). For cohort 3 (1992-2003), the MCAT score was associated with mscGPA, and uGPA was associated with gsGPA. Overall, MCAT score and uGPA were inconsistent or weak predictors of training metrics and career outcomes for this population of MD-PhD students. [ABSTRACT FROM AUTHOR]

Background and Objectives: Our objective was to determine the relationship between critical thinking skills and objective measures of academic success in a family medicine residency program.Methods: This prospective observational cohort study was set in a large Canadian family medicine residency program. Intervention was the California Critical Thinking Skills Test (CCTST), administered at three points in residency: upon entry, at mid-point, and at graduation. Results from the CCTST, Canadian Residency Matching Service file, and interview scores were compared to other measures of academic performance (Medical Colleges Admission Test [MCAT] and College of Family Physicians of Canada [CCFP] certification examination results).Results: For participants (n=60), significant positive correlations were found between critical thinking skills and performance on tests of knowledge. For the MCAT, CCTST scores correlated positively with full scores (n=24, r=0.57) as well as with each section score (verbal reasoning: r=0.59; physical sciences: r=0.64; biological sciences: r=0.54). For CCFP examination, CCTST correlated reliably with both sections (n=49, orals: r=0.34; short answer: r=0.47). Additionally, CCTST was a better predictor of performance on the CCFP exam than was the interview score at selection into the residency program (Fisher's r-to-z test, z=2.25).Conclusions: Success on a critical thinking skills exam was found to predict success on family medicine certification examinations. Given that critical thinking skills appear to be stable throughout residency training, including an assessment of critical thinking in the selection process may help identify applicants more likely to be successful on final certification exam. [ABSTRACT FROM AUTHOR]

High-stakes standardized literacy testing is not neutral and continues to build upon the legacy of dominant power relations in the state in its ability to sort, select and rank students and ultimately produce and name some youth as illiterate in contrast to an ideal white, male, literate citizen. I trace the effects of high-stakes standardized testing by using the voices of 16 youth who failed the Ontario Secondary School Literacy Test (OSSLT) to illustrate how the ‘illiterate youth’ revealed to students, schools, and communities by this test is culturally and socially constructed. In an age where multiple literacies are more and more valued, standardized literacy testing acts as a form of social control projected upon the ‘adolescent’ body that has historically been deemed ‘other’ or ‘deficient.’ Just as colonized subjects needed to be ‘civilized,’ so youth now need to acquire a state defined literacy in a competitive and fast paced learning environment. This article helps to demonstrate how power operates on marginalized youth through standardized literacy testing that is being used transnationally. [ABSTRACT FROM PUBLISHER]

Context Reflective writing is a popular tool to support the growth of reflective capacity in undergraduate medical learners. Its popularity stems from research suggesting that reflective capacity may lead to improvements in skills such as empathy, communication, collaboration and professionalism. This has led to assumptions that reflective writing can also serve as a tool for student assessment. However, evidence to support the reliability and validity of reflective writing as a meaningful assessment strategy is lacking. Methods Using a published instrument for measuring 'reflective capacity' (the Reflection Evaluation for Learners' Enhanced Competencies Tool [ REFLECT]), four trained raters independently scored four samples of writing from each of 107 undergraduate medical students to determine the reliability of reflective writing scores. REFLECT scores were then correlated with scores on a Year 4 objective structured clinical examination ( OSCE) and Year 2 multiple-choice question ( MCQ) examinations to examine, respectively, convergent and divergent validity. Results Across four writing samples, four-rater Cronbach's α-values ranged from 0.72 to 0.82, demonstrating reasonable inter-rater reliability with four raters using the REFLECT rubric. However, inter-sample reliability was fairly low (four-sample Cronbach's α = 0.54, single-sample intraclass correlation coefficient: 0.23), which suggests that performance on one reflective writing sample was not strongly indicative of performance on the next. Approximately 14 writing samples are required to achieve reasonable inter-sample reliability. The study found weak, non-significant correlations between reflective writing scores and both OSCE global scores ( r = 0.13) and MCQ examination scores ( r = 0.10), demonstrating a lack of relationship between reflective writing and these measures of performance. Conclusions Our findings suggest that to draw meaningful conclusions about reflective capacity as a stable construct in individuals requires 14 writing samples per student, each assessed by four or five raters. This calls into question the feasibility and utility of using reflective writing rigorously as an assessment tool in undergraduate medical education. [ABSTRACT FROM AUTHOR]

E-learning is becoming more and more popular as it allows the delivery of educational content in a way that is more convenient and self-paced than traditional methods. E-learning is flexible as content is accessible from various electronic devices in the workplace or outside of the work environment. This article describes the history of e-learning as an andragogical approach and offers a guide to the clinician who is seeking to create a module related to vascular access. Examples of three modules developed at the McGill University Health Centre in Montréal, Québec, are used to illustrate key elements that should be included in a fun and effective e-learning tool. Learners’ perceptions of the e-learning modules are described. [ABSTRACT FROM AUTHOR]

This paper presents perspectives and controversies surrounding the use of milestones to assess competency in outcomes-based medical education. Global perspectives (Canada, Europe, and the United States) and developments supporting their rationales are discussed. In Canada, there is a significant movement away from conceptualizing competency based on time, and a move toward demonstration of specific competencies. The success of this movement may require complex (rather than reductionist) milestones that reflect students' progression through complexity and context and a method to narrate their journey. European countries (United Kingdom, France, and Germany) have stressed the complexity associated with time and milestones for medical students to truly achieve competence. To meet the changing demands of medicine, they view time as actually providing students with knowledge and exposure to achieve various milestones. In the United States, milestones are based on sampling throughout professional development to initiate lifelong learning. However, the use of milestones may not imply overall competence (reductionism). Milestones must be developed alongside outcomes-based curriculum with use of faculty and competency committees. The perspectives outlined in this paper underscore emerging challenges for implementing outcomes-based medical education and call for new conceptualizations of competence. [ABSTRACT FROM AUTHOR]

This study examined the relationship between performance on standardized measures of language proficiency and conversational measures of the same features used in academic discourse among 24 monolingual and 25 bilingual kindergarteners. Academic discourse performance was considered for both its linguistic and its genre features in two discourse forms: narrative and explanation. Bilinguals performed more poorly than monolinguals on standardized measures of language proficiency, yet they performed similarly to monolinguals in the discourse-based linguistic and genre features. Moreover, genre features were more strongly related to linguistic features assessed through discourse than to standardized tests of these same features. These findings indicate that standardized measures of language proficiency underrepresent the abilities of bilingual children and that children's second language proficiency may be more accurately reflected in conversation. [ABSTRACT FROM PUBLISHER]

The current study applied the person-fit statistic,lz, to data from a Canadian provincial achievement test to explore the usefulness of conducting person-fit analysis on large-scale assessments. Item parameter estimates were compared before and after the misfitting student responses, as identified bylz, were removed. The changes of item parameter estimates were found to be noticeable for some items. In addition, analyses were conducted to identify student and class factors associated with misfitting responses. Hierarchical linear modeling was used due to the hierarchical structure of the data. Although student-level and class-level variables were found as statistically significant predictors of the degree of person-fit as indicated by the values oflz, the percentage of variability accounted for by these variables was considerably small—only 3.65%. [ABSTRACT FROM AUTHOR]

Background: Past research suggests that the use of externally-applied scoring weights may not appreciably impact measurement qualities such as reliability or validity. Nonetheless, some credentialing boards and academic institutions apply differential scoring weights based on expert opinion about the relative importance of individual items or test components of Observed Structured Clinical Examinations (OSCEs). Aims: To investigate the impact of simplified scoring models that make little to no use of differential weighting on the reliability of scores and decisions on a high stakes OSCE required for medical licensure in Canada. Method: We applied four different weighting models of various complexities to data from three administrations of the OSCE. We compared score reliability, pass/fail rates, correlations between the scores and classification decision accuracy and consistency across the models and administrations. Results: Less complex weighting models yielded similar reliability and pass rates as the more complex weighting model. Minimal changes in candidates' pass/fail status were observed and there were strong and statistically significant correlations between the scores for all scoring models and administrations. Classification decision accuracy and consistency were very high and similar across the four scoring models. Conclusions: Adopting a simplified weighting scheme for this OSCE did not diminish its measurement qualities. Instead of developing complex weighting schemes, experts' time and effort could be better spent on other critical test development and assembly tasks with little to no compromise in the quality of scores and decisions on this high-stakes OSCE. [ABSTRACT FROM AUTHOR]

Copyright of Canadian Journal of Education / Revue Canadienne de l'Éducation is the property of Canadian Society for the Study of Education and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Context The shift from a time-based to a competency-based framework in medical education has created a need for frequent formative assessments. Many educational programmes use some form of written progress test to identify areas of strength and weakness and to promote continuous improvement in their learners. However, the role of performance-based assessments, such as objective structured clinical examinations ( OSCEs), in progress testing remains unclear. Objective The aims of this paper are to describe the use of an OSCE to assess learners at different stages of training, describe a structure for reporting scores, and provide evidence for the psychometric properties of different rating tools. Methods A 10-station OSCE was administered to internal medicine residents in postgraduate years ( PGYs) 1-4. Candidates were assessed using a checklist ( CL), a global rating scale ( GRS) and a training level rating scale ( TLRS). Reliability was calculated for each measure using Cronbach's alpha. Differences in performance by year of training were explored using analysis of variance ( anova). Correlations between scores obtained using the different rating instruments were calculated. Results Sixty-nine residents participated in the OSCE. Inter-station reliability was greater (0.88) using the TLRS compared with the CL (0.84) and GRS (0.79). Using all three rating instruments, scores varied significantly by year of training (p < 0.001). Scores from the different rating instruments were highly correlated: CL and GRS, r = 0.93; CL and TLRS, r = 0.90, and GRS and TLRS, r = 0.94 (p < 0.001). Candidates received feedback on their performance relative to examiner expectations for their PGY level. Conclusions Scores were found to have high reliability and demonstrated significant differences in performance by year of training. This provides evidence for the validity of using scores achieved on an OSCE as markers of progress in learners at different levels of training. Future studies will focus on assessing individual progress on the OSCE over time. [ABSTRACT FROM AUTHOR]