Search

Objective: To determine the differential effects of a paper-based versus a web-based portfolio in terms of portfolio quality, user-friendliness and student motivation., Methods: An experimental design was used to compare Year 1 medical students' reflective portfolios. The portfolios differed in presentation medium only (i.e. web-based versus paper-based). Content analysis, a student questionnaire and mentor interviews were used to evaluate portfolio quality, user-friendliness and student motivation. A total of 92 portfolios were scored independently by 2 raters using a portfolio quality-rating instrument., Results: Portfolio structure, quality of reflection and quality of evidence showed no significant effects of presentation medium. Multi-level analysis showed a significant effect for student motivation: web-based portfolios scored 0.39 more than paper-based portfolios (P < 0.05; effect size 0.76). The mentors reported no differences in portfolio quality, except that there were more visuals in web-based portfolios. Students spent significantly more time preparing the web-based than the paper-based portfolios (15.4 hours versus 12.2 hours; t = 2.1, P < 0.05; effect size 0.46). The 2 student groups did not differ significantly in terms of their satisfaction with the portfolio. The mentors perceived the web-based portfolios as more user-friendly., Conclusions: The web-based portfolios were found to enhance students' motivation, were more user-friendly for mentors, and delivered the same content quality compared with paper-based portfolios. This suggests that web-based presentation may promote acceptance of portfolios by students and teachers alike.

Context: The move towards competence-based medical education has created a need for instruments that support and assess competence development. Portfolios seem suitable but mixed reports of their success are emerging., Methods: To examine the effectiveness of portfolios, we searched PubMed and EMBASE using the keyword 'portfol*', PsychInfo and ERIC using the keywords 'portfol*' and 'medical education' and references of retrieved papers for empirical studies on portfolios in all phases of medical education. Thirty of 1939 retrieved papers met the inclusion criteria and were analysed. Data were collated against the research question, number of subjects, design, setting, findings and limitations, purpose and content, mentoring and assessment. We analysed impact using a modified version of Kirkpatrick's hierarchy., Results: Because differences across studies precluded statistical meta-analysis, the data were analysed by context, goals and procedure. Positive effects were strongest in undergraduate education. Important factors for success were: clearly communicated goals and procedures; integration with curriculum and assessment; flexible structure; support through mentoring, and measures to heighten feasibility and reduce required time. Moderately good inter-rater reliability was reported and global criteria and discussions among raters were beneficial. Formative and summative assessment could be combined. Without assessment, portfolios were vulnerable to competition from other summative assessment instruments., Conclusions: For portfolios to be effective in supporting and assessing competence development, robust integration into the curriculum and tutor support are essential. Further studies should focus on the effectiveness and user-friendliness of portfolios, the merits of holistic assessment procedures, and the competences of an effective portfolio mentor.

Background: Medical students are increasingly being taught in community‐based hospitals and clinics in addition to university‐based health systems, and there is sustained interest in the question of whether medical schools should pay healthcare organizations to allow students to precept to compensate for loss of productivity by preceptors. This study aimed to inform this subject by investigating the effect of medical students on preceptor productivity. Methods: The authors analysed retrospective data on preceptor productivity and time efficiency from 17 outpatient primary care physicians over 7 months and 38,205 patient visits. Then, the authors surveyed these same physicians to assess how they believe precepting medical students affected their productivity. Results: The retrospective data analysis found that physicians see slightly more patients per half‐day when accompanied by a student. In surveys, these physicians reported the same number of patients seen per half‐day with or without a medical student, but more time spent in clinic with a medical student. They also reported that allowing students to document in the medical record often helped reduce/offset the lengthened workday. Conclusions: Medical students either increase or do not change the number of patients seen per half‐day but can increase preceptor time spent in clinic per day. Generally, physicians accept this increased time for the sake of promoting education and the joy of teaching. These findings can inform medical schools and healthcare organizations when negotiating financial arrangements for preceptorships in community based primary care clinics. Specifically, these findings suggest that medical students precepting do not cost the hospital organization in terms of preceptor productivity. Hatfield et al. demonstrate that medical students do not reduce preceptor productivity when measured by number of patients seen per half‐day, raising questions about healthcare organization financial policy. [ABSTRACT FROM AUTHOR]

Context Portfolio-based learning is a popular educational tool usually examined by document review which is sometimes accompanied by an oral examination. This labour-intensive assessment method prohibits its use in the resource-constrained settings typical of developing countries. Objectives We aimed to determine the feasibility and internal consistency of a portfolio-based structured interview and its impact on student learning behaviour. Methods Year 4 medical students ( n = 181) recorded 25 patient encounters during a 14-week medical clerkship. Portfolios were examined in a 30-minute, single-examiner interview in which four randomly selected cases were discussed. Six standard questions were used to guide examiners in determining the ability of candidates to interpret and synthesise clinical data gathered during patient encounters. Examiners were trained to score responses using a global rating scale. Pearson’s correlation co-efficient, Cronbach’s α coefficient and the standard error of measurement (SEM) of the assessment tool were determined. The number of students completing more than the required number of portfolio entries was also recorded. Results The mean (± standard deviation [SD], 95% confidence interval [CI]) interview score was 67.5% (SD ± 10.5, 95% CI 66.0–69.1). The correlation coefficients for the interview compared with other component examinations of the assessment process were: multiple-choice question (MCQ) examination 0.42; clinical case-based examination 0.37; in-course global rating 0.08, and overall final score 0.54. Cronbach’s α coefficient was 0.88 and the SEM was 3.6. Of 181 students, 45.3% completed more than 25 portfolio entries. Conclusions Portfolio assessment using a 30-minute structured interview is a feasible, internally consistent assessment method that requires less examination time per candidate relative to methods described in published work and which may encourage desirable student learning behaviour. [ABSTRACT FROM AUTHOR]

Objective This study aimed to establish documentation standards for medical education activities, beyond educational research, for academic promotion consistent with principles of excellence and scholarship. Methods In 2006 a Consensus Conference on Educational Scholarship was convened by the Association of American Medical Colleges (AAMC) Group on Education Affairs (GEA) to outline a set of documentation standards for use by educators and academic promotion committees. Conference participants' work was informed by more than 15 years of literature on scholarship, educator portfolios and academic promotion standards. Results The 110 conference participants, including medical school deans, academic promotion committee members, department chairs, faculty and AAMC leaders, re-affirmed the 5 education activity categories (teaching, curriculum, advising and/or mentoring, education leadership and/or administration, and learner assessment), the contents of each category, and cross-category documentation standards. Educational excellence requires documentation of the quantity and quality of education activities. Documenting a scholarly approach requires demonstrating evidence of drawing from and building on the work of others, and documenting scholarship requires contributing work through public display, peer review and dissemination; both involve engagement with the community of educators. Implementation of these standards – quantity, quality and engagement with the education community – should occur in parallel with the development of an infrastructure to support educators, including sustained faculty development for educators, access to educational resources and journals, peer review mechanisms and consultation and support specific to each activity category. Conclusions Educators' contributions to their institutions must be visible to be valued. The establishment of documentation standards for education activities provides the foundation for academic recognition of educators. [ABSTRACT FROM AUTHOR]

Objectives: There are only a few descriptive reports on the implementation of distributed medical education (DME) and these provide accounts of successful implementation from the senior leadership perspective. In Saskatchewan, over a period of 4 years (2010–2014), four family medicine residency sites were established and two additional sites could not be developed. The aim of this study was to identify challenges, success factors and pitfalls in DME implementation based upon experiences of multiple stakeholders with both successful and unsuccessful outcomes. Methods: Data were obtained through document analysis (n = 64, spanning 2009–2016; perspectives of government, senior leadership, management and learners), focus groups of management and operations personnel (n = 10) and interviews of senior leaders (n = 4). Challenges and success factors were ascertained through categorisation. Iterative coding guided by three sensitising frameworks was used to determine themes in organisational dynamics. Results: Both challenges and success factors included contextual variables, governance, inter‐ and intra‐organisational relationships (most common success factor), resources (most common challenge), the learning environment and pedagogy. Management and operations were only a challenge. Organisational themes affecting the outcome and the pitfalls included the pace of development across multiple sites, collaborative governance, continuity in senior leadership, operations alignment and reconciliation of competing goals. Conclusions: Emerging opportunities for DME can be leveraged through collaborative governance, aligned operations and resolution of competing goals, even in constrained contexts, to translate political will into success; however, there are pitfalls that need to be avoided. Our findings based upon multi‐stakeholder perspectives add to the body of knowledge on deployment, carefully considering the conditions for success and associated pitfalls. The authors share challenges and pitfalls that need to be navigated to successfully implement Distributed Medical Education by drawing on the concept of organizational dynamics [ABSTRACT FROM AUTHOR]

Context Work-based assessments ( WBAs) represent an increasingly important means of reporting expert judgements of trainee competence in clinical practice. However, the quality of WBAs completed by clinical supervisors is of concern. The episodic and fragmented interaction that often occurs between supervisors and trainees has been proposed as a barrier to the completion of high-quality WBAs. Objectives The primary purpose of this study was to determine the effect of supervisor-trainee continuity on the quality of assessments documented on daily encounter cards ( DECs), a common form of WBA. The relationship between trainee performance and DEC quality was also examined. Methods Daily encounter cards representing three differing degrees of supervisor-trainee continuity (low, intermediate, high) were scored by two raters using the Completed Clinical Evaluation Report Rating ( CCERR), a previously published nine-item quantitative measure of DEC quality. An analysis of variance ( anova) was performed to compare mean CCERR scores among the three groups. Linear regression analysis was conducted to examine the relationship between resident performance and DEC quality. Results Differences in mean CCERR scores were observed between the three continuity groups (p = 0.02); however, the magnitude of the absolute differences was small (partial eta-squared = 0.03) and not educationally meaningful. Linear regression analysis demonstrated a significant inverse relationship between resident performance and CCERR score (p < 0.001, r2 = 0.18). This inverse relationship was observed in both groups representing on-service residents (p = 0.001, r2 = 0.25; p = 0.04, r2 = 0.19), but not in the Off-service group (p = 0.62, r2 = 0.05). Conclusions Supervisor-trainee continuity did not have an educationally meaningful influence on the quality of assessments documented on DECs. However, resident performance was found to affect assessor behaviours in the On-service group, whereas DEC quality remained poor regardless of performance in the Off-service group. The findings suggest that greater attention should be given to determining ways of improving the quality of assessments reported for off-service residents, as well as for those residents demonstrating appropriate clinical competence progression. [ABSTRACT FROM AUTHOR]

Context In teaching hospitals, junior trainees (first-year residents and third-year medical students) are responsible for patient follow-up and documentation under the supervision of senior team members. In order to support trainees in their role, supervisors need to understand how trainees approach these tasks and how they can be coached to develop best practices. Objectives The purpose of our study was to explore the range of practices used by junior trainees in clinical settings. Methods Constructivist grounded theory was used to guide the collection and analysis of data on follow-up and documentation during 34 observation periods with 17 junior trainees. Data sources included field notes, field interviews and de-identified copies of patient charts. We also held two focus groups with four attending physicians in each. Results We were able to describe three interrelated characteristics that influenced a trainee's approach to and ability to perform the tasks of patient follow-up and documentation: (i) diligence; (ii) relationship to the team (dependent, independent, collaborative), and (iii) level of performance ( Data Gatherer, Sensemaker, Manager). Diligence and relationship to the team appeared to influence the quality and focus of a trainee's approach at all levels of performance. Level of performance was felt, by focus group attending physicians, to reflect a developmental progression of knowledge and skills. Conclusions Our findings contribute to the existing literature in three ways. Firstly, they extend our understanding of how junior trainees approach the task of in-patient follow-up and clinical documentation and the value of those activities. Secondly, they provide new insights to support formative and summative assessment. Finally, they contribute to a growing body of literature exploring the factors that impact trainees' roles and interactions with the team. Future research should focus on validating our findings and exploring their utility in the development of novel assessment strategies. [ABSTRACT FROM AUTHOR]

Context Interest is growing in the use of qualitative data for assessment. Written comments on residents' in-training evaluation reports (ITERs) can be reliably rank-ordered by faculty attendings, who are adept at interpreting these narratives. However, if residents do not interpret assessment comments in the same way, a valuable educational opportunity may be lost. Objectives Our purpose was to explore residents' interpretations of written assessment comments using mixed methods. Methods Twelve internal medicine (IM) postgraduate year 2 (PGY2) residents were asked to rank-order a set of anonymised PGY1 residents ( n = 48) from a previous year in IM based solely on their ITER comments. Each PGY1 was ranked by four PGY2s; generalisability theory was used to assess inter-rater reliability. The PGY2s were then interviewed separately about their rank-ordering process, how they made sense of the comments and how they viewed ITERs in general. Interviews were analysed using constructivist grounded theory. Results Across four PGY2 residents, the G coefficient was 0.84; for a single resident it was 0.56. Resident rankings correlated extremely well with faculty member rankings ( r = 0.90). Residents were equally adept at reading between the lines to construct meaning from the comments and used language cues in ways similarly reported in faculty attendings. Participants discussed the difficulties of interpreting vague language and provided perspectives on why they thought it occurs (time, discomfort, memorability and the permanency of written records). They emphasised the importance of face-to-face discussions, the relative value of comments over scores, staff-dependent variability of assessment and the perceived purpose and value of ITERs. They saw particular value in opportunities to review an aggregated set of comments. Conclusions Residents understood the 'hidden code' in assessment language and their ability to rank-order residents based on comments matched that of faculty. Residents seemed to accept staff-dependent variability as a reality. These findings add to the growing evidence that supports the use of narrative comments and subjectivity in assessment. [ABSTRACT FROM AUTHOR]

Objectives As community settings are being used increasingly in undergraduate medical programmes, this study aimed to explore and compare the clinical experiences of students in hospital-based and community-based training programmes. It measured students' clinical participation and compared the perspectives of Year 3 medical students in three different models of clinical education: a tertiary hospital block programme; a community hybrid programme, and a rural longitudinal integrated clerkship ( LIC) programme. Methods The study used a mixed methodology approach to examine the clinical experiences of students through the analysis of logbooks and semi-structured student interviews. This involved the quantitative analysis of 88 logbook weeks, data from which were triangulated through the analysis of 101 individual interviews using grounded theory. Results A total of 35 students across the three different clinical training models participated in the study. The results demonstrate significant differences among the three models in students' clinical participation and suggest that community settings provide more opportunities to students for meaningful engagement in patient care activities. Conclusions Consistent wider and more direct access to patients for students, as found in the community-based model, provides a pathway for engaging students in the learning processes, and a step towards making them aware of their learning needs and knowledge. Interviews provide evidence that authentic clinical activities can be enhanced through structured systems of supervision and through the provision of authentic roles for students in clinical teams. [ABSTRACT FROM AUTHOR]

Objectives Medical teaching teams ( MTTs) must balance teaching and patient care in the face of three challenges: shifting team membership, varying levels of learners and patient complexity. To support care, MTTs rely on a combination of recurrent oral and written communication practices (genres), such as admission, progress and discharge notes. The purpose of this study was to explore how these genres influence the team's ability to collectively care for patients. Methods This was a multiple case study with data collected through observations and audio-recordings of 19 patient cases focusing on admission review discussions and chart documents throughout the hospitalisation. Participants included 14 medical students, 32 residents and 10 attending physicians rotating through one of three internal medicine MTTs. We used constant comparative analysis to identify recurrent patterns across the multiple cases, which were further elaborated in a return-of-findings focus group. Results The MTT genre system facilitated the care of patients through 'progressive collaborative refinement' ( PCR): MTTs use case and data reviews to collaboratively and progressively refine their understanding of the patient's problems and develop strategies for addressing them. Progressive collaborative refinement was apparent through modifications made in the documentation. Although modifications were a necessary component, they were not sufficient: some modifications were made without refinement. We characterised incidents of failed modification as 'fragmentation'. Three types were observed: conceptualisation, documentation and continuity of care providers. In most cases, all three were present and interacted to impede PCR. Conclusions Progressive collaborative refinement was used by MTTs to provide the optimal care to patients. Progressive collaborative refinement was impeded by a lack of continuity of care providers and gaps between communication genres that fragmented conceptualisation and documentation. Progressive collaborative refinement can be understood as both an overarching process and a shared but unstated ideal. Through defining and describing PCR, the present findings can be used to improve communication and teaching. [ABSTRACT FROM AUTHOR]

Medical Education 2012: 46: 1009-1016 Context Medical educators emphasise responses to medical misconduct, but little is known about medical misconduct and its implications for medical education. This article investigates the nature of medical malpractice in Hong Kong and offers guidance for the inclusion of a curriculum to prevent the occurrence of medical misconduct in medical education around the world. Methods A comprehensive review of judgements made by the Medical Council of Hong Kong during the period from July 2008 to December 2010 was conducted. Each of the 40 cases of inquiry related to medical misconduct were summarised and analysed according to 14 factors. Results Of the 40 cases, nearly half involved only one or two charges. The Council found the defendants guilty of professional misconduct on 148 of 169 charges, and ordered the following four types of penalty: removal order, suspension, warning letter, and reprimand. Cases are grouped into three categories involving: improper documentation, inappropriate management or prescription of drugs, and failure to interact appropriately with patients. The relevant ethical codes or legislation for each category are illustrated. Conclusions Various types of medical misconduct unquestionably caused suffering to the patients involved, their families and society. Hence, it is crucial for medical educators to teach students about the importance of medical ethics and the prevention of misconduct. [ABSTRACT FROM AUTHOR]

Context Although concern has been raised about the value of clinical evaluation reports for discriminating among trainees, there have been few efforts to formalise the dimensions and qualities that distinguish effective versus less useful styles of form completion. Methods Using brainstorming and a modified Delphi technique, a focus group determined the key features of high-quality completed evaluation reports. These features were used to create a rating scale to evaluate the quality of completed reports. The scale was pilot-tested locally; the results were psychometrically analysed and used to modify the scale. The scale was then tested on a national level. Psychometric analysis and final modification of the scale were completed. Results Sixteen features of high-quality reports were identified and used to develop a rating scale: the Completed Clinical Evaluation Report Rating (CCERR). The reliability of the scale after a national field test with 55 raters assessing 18 in-training evaluation reports (ITERs) was 0.82. Further revisions were made; the final version of the CCERR contains nine items rated on a 5-point scale. With this version, the mean ratings of three groups of ‘gold-standard’ ITERs (previously judged to be of high, average and poor quality) differed significantly ( P < 0.05). Discussion The CCERR is a validated scale that can be used to help train supervisors to complete and assess the quality of evaluation reports. [ABSTRACT FROM AUTHOR]

Context A learning portfolio was developed to support the development of trainee doctors piloting Foundation Programme prototypes across the Northern Deanery in 2004 and 2005. Trainee doctors and their educational supervisors were surveyed about their experiences of using the portfolio in the clinical workplace. Methods The evaluation consisted of semi-structured interviews with trainee doctors and supervisors, followed by postal questionnaire surveys in 2004 and 2005. Quantitative and qualitative data were triangulated to identify core findings. Results Questionnaires were returned from 182/305 (60%) trainee doctors and 104 out of 179 (58%) educational supervisors. The portfolio was felt to be a ‘good idea’ by 55% supervisors and 48% trainees. Trainees’ perceptions of the educational value of the portfolio remained consistently low over 2 surveys and they described a sense of ‘burden’, whereby they identified problems in workload and usability and in gaining feedback on performance. However, positive trainee attitudes towards the portfolio were significantly correlated with greater perceived educational benefits ( r = 0.855, P < 0.001). Discussion Learning portfolios are now an integral part of Foundation Programme training but this evaluation suggests that many trainee doctors and educational supervisors are yet to be convinced of their educational value. Gaining multi-source feedback, a substantial component of trainee doctors’ portfolios, impacts on the wider clinical team and presents a significant challenge to trainees. Educational supervisors continued to rely on feedback from clinical colleagues, rather than portfolio evidence, to monitor trainee doctors’ development. Such factors may serve to disengage trainees with the portfolio process by overshadowing any perceived educational gains. [ABSTRACT FROM AUTHOR]

Context The portfolio is becoming increasingly accepted as a valuable tool for learning and assessment. The validity of portfolio assessment, however, may suffer from bias due to irrelevant qualities, such as lay-out and writing style. We examined the possible effects of such qualities in a portfolio programme aimed at stimulating Year 1 medical students to reflect on their professional and personal development. In later curricular years, this portfolio is also used to judge clinical competence. Methods We developed an instrument, the Portfolio Analysis Scoring Inventory, to examine the impact of form and content aspects on portfolio assessment. The Inventory consists of 15 items derived from interviews with experienced mentors, the literature, and the criteria for reflective competence used in the regular portfolio assessment procedure. Forty portfolios, selected from 231 portfolios for which ratings from the regular assessment procedure were available, were rated by 2 researchers, independently, using the Inventory. Regression analysis was used to estimate the correlation between the ratings from the regular assessment and those resulting from the Inventory items. Results Inter-rater agreement ranged from 0.46 to 0.87. The strongest predictor of the variance in the regular ratings was ‘quality of reflection’ (R 0.80; R2 66%). No further items accounted for a significant proportion of variance. Irrelevant items, such as writing style and lay-out, had negligible effects. Conclusions The absence of an impact of irrelevant criteria appears to support the validity of the portfolio assessment procedure. Further studies should examine the portfolio's validity for the assessment of clinical competence. [ABSTRACT FROM AUTHOR]

Introduction We wished to determine whether assessors could make reliable and valid judgements about the quality of completed reflective personal development plans (PDPs) for the purpose of accrediting UK general practitioners (GPs) for a postgraduate education allowance using a marking matrix, and secondly, to plan a feasible model of PDP assessment in the context of forthcoming GP appraisal/revalidation that would overcome the main sources of error identified from this study. Methods Within generalisability theory, a variance components analysis on PDP scores estimated reliability and the effect on them of varying, for example, the number of assessors. We investigated the construct validity of the matrix through its internal consistency and detection of differences in the quality of PDPs. Results For a single PDP and one assessor, 37.6% of the variance in scores was due to true differences in the quality of the PDP. Between 5 and 7 PDP assessors are needed to achieve summative reliability of greater than 0.8. While increasing the number of judges is important, reliability could also be improved by addressing assessor subjectivity. Construct validity was demonstrated, as the matrix distinguished between good, satisfactory and poor PDPs, and it had good internal consistency. Conclusion PDP assessment has reasonable summative characteristics for the purpose of assessing GPs' reflective continuing professional development. If doctors could include their PDPs within their revalidation folders as evidence of their reflections on pursuing better clinical performance, we have described a reliable, valid and feasible method of external assessment. [ABSTRACT FROM AUTHOR]