Search

Purpose To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature. Method In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables. Results The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance. Conclusions Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

Objectives In line with a recent report entitled Effective Use of Educational Technology in Medical Education from the Association of American Medical Colleges Institute for Improving Medical Education (AAMC-IME), this study examined whether revising a medical lecture based on evidence-based principles of multimedia design would lead to improved long-term transfer and retention in Year 3 medical students. A previous study yielded positive effects on an immediate retention test, but did not investigate long-term effects. Methods In a pre-test/post-test control design, a cohort of 37 Year 3 medical students at a private, midwestern medical school received a bullet point-based PowerPoint™ lecture on shock developed by the instructor as part of their core curriculum (the traditional condition group). Another cohort of 43 similar medical students received a lecture covering identical content using slides redesigned according to Mayer’s evidence-based principles of multimedia design (the modified condition group). Results Findings showed that the modified condition group significantly outscored the traditional condition group on delayed tests of transfer given 1 week (d = 0.83) and 4 weeks (d = 1.17) after instruction, and on delayed tests of retention given 1 week (d = 0.83) and 4 weeks (d = 0.79) after instruction. The modified condition group also significantly outperformed the traditional condition group on immediate tests of retention (d = 1.49) and transfer (d = 0.76). Conclusions This study provides the first evidence that applying multimedia design principles to an actual medical lecture has significant effects on measures of learner understanding (i.e. long-term transfer and long-term retention). This work reinforces the need to apply the science of learning and instruction in medical education.

The growing appreciation of the need to adopt an evidence-based approach to teaching and assessment has led to a demand for faculty who are well versed in best practices in education. Surgeons with interest and expertise in instruction, curriculum development, educational research, and evaluation can have an important impact on the educational mission of a department of surgery. The increased fervor for accountability in education together with the challenges imposed by accreditation agencies and hospitals has made educational leadership responsibilities more time consuming and complex. In response to this, an increasing number of department chairs created Vice Chair for Education positions to support clerkship and program directors and ensure the department's education mission statement is fulfilled.

Medical Education 2011: 45: 818–826 Context The Association of American Medical Colleges’ Institute for Improving Medical Education’s report entitled ‘Effective Use of Educational Technology’ called on researchers to study the effectiveness of multimedia design principles. These principles were empirically shown to result in superior learning when used with college students in laboratory studies, but have not been studied with undergraduate medical students as participants. Methods A pre-test/post-test control group design was used, in which the traditional-learning group received a lecture on shock using traditionally designed slides and the modified-design group received the same lecture using slides modified in accord with Mayer’s principles of multimedia design. Participants included Year 3 medical students at a private, midwestern medical school progressing through their surgery clerkship during the academic year 2009–2010. The medical school divides students into four groups; each group attends the surgery clerkship during one of the four quarters of the academic year. Students in the second and third quarters served as the modified-design group (n = 91) and students in the fourth-quarter clerkship served as the traditional-design group (n = 39). Results Both student cohorts had similar levels of pre-lecture knowledge. Both groups showed significant improvements in retention (p

Background Composite score validity depends on its reliability, content, and comparison with other outcomes measures. This study examined all three aspects of a composite score used in a third-year surgery clerkship. Method Composite score reliability was calculated using stratified alpha; several other reasonable composite combinations were tested. Correlation coefficients between clerkship composite scores and grades were calculated with other achievement measures. Two consecutive years were studied (N = 162, N = 159). Results The clerkship composite score reliability was .76 (Year 1) and .81 (Year 2). Positive correlations (P ≤ .05) were found between composite scores and other clerkships' test scores and the United States Medical Licensing Examination Step 2 Clinical Knowledge examination. Positive correlations (P ≤ .05) were found between surgery clerkship grades and other clerkship grades. Conclusions This study documents the reliability of a composite score and provides evidence for its validity in a surgery clerkship.

This project sought to study the effectiveness of a curriculum to enhance the intraoperative clinical judgment and procedural skill of surgical residents.A multiinstitutional, prospective, randomized study was performed. A cognitive task analysis of laparoscopic cholecystectomy (LC) was conducted on which instructional activities and measurement instruments were designed. Residents were randomly assigned to a control or intervention group. Subjects took written pre- and posttests examining procedure-related judgment and knowledge. The intervention group participated in a three-session curriculum emphasizing LC critical decisions and error prevention. All subjects were evaluated performing the procedure on a cadaveric model. Scores from written and practical exams were compared using independent-sample and paired Student t tests.Written examination scores increased for both groups. The intervention group scored significantly higher (P.05) on the written posttest than the control group. There were no differences between groups on the practical examination. Reliability coefficients for the written examination ranged from .65 to .75. Reliability coefficients for the oral exam, technical skill, and error items on the porcine practical exam were .83, .90, and .53.The curriculum resulted in enhanced performance on a written exam designed to assess intraoperative judgment, but no differences in technical skills, showing important implications for future skills lab curriculum formats.

Background The purpose of the study is to describe the academic preparation, scope of duties, and scholarly activity of professional educators in surgery departments. Methods Educators with doctoral degrees employed as full-time faculty in surgery departments were surveyed to determine terms of employment, academic preparation, scope of duties, and job satisfaction. Results Twelve of 13 educators responded and participated in the study. Educators spent, on average, 22% of their time on research activities, 33% on administrative responsibilities, 13% on teaching, 13% counseling students and residents, and 7% writing grants. They spent approximately 34% of their time with surgical faculty, 19% with residents/fellows, and 14% with medical students. Educators' contributions to surgery departments included improvements in assessment and evaluation, educational conferences, recruitment, and research productivity. Conclusions Professional educators provide support needed to meet the growing demands and requirements of surgical education. Study findings may inform those interested in recruiting a professional educator to their faculty.

Background: Residency programs are required to teach and evaluate trainees in the area of professionalism and medical ethics. Prior to developing a curriculum in this area, residents and fellows were surveyed to assess learning needs. Description: A case-based survey was developed based on published curricula. Residents and fellows were asked to describe their comfort level in 11 clinical scenarios on a Likert-type scale ranging from 1 (not at all comfortable) to 10 (extremely comfortable). Evaluation: 151 surveys were returned for an overall response rate of 73%. Comfort levels ranged from a low of 3.1 to a high of 8.5 on the 10-point scale. Despite additional years of clinical training, fellows only reported an increased comfort level in 1 case. Conclusion: Learning needs exist in residents and fellows in the area of medical ethics. Use of a needs assessment was instrumental in planning and designing an ethics curriculum.

Students consistently identified inadequate feedback as a deficiency in our third-year clerkship.We asked students to solicit one faculty and one resident every 2 weeks for written feedback on a "feedback prescription pad." Each prescription requested four comments: two things the student did well and two things the student needs to improve. Students rated feedback using a five-point scale. A three-point categorization scheme was employed to assess the quality of feedback.Students' rating of feedback improved significantly compared with a previous time period (3.5 +/- 1.2 versus 2.6 +/- 1.2, P0.01). Interrater reliability of our categorization scheme was high (kappaor =0.75, P0.01) and demonstrated that only 10% of comments were specific enough to qualify as effective feedback.Feedback prescription pads were a simple method to facilitate feedback. Although students appreciated feedback, most feedback was inadequate. Faculty development programs to enhance student feedback should be a priority of clinical medical education.

This study was designed to evaluate the impact of changes made to our morbidity and mortality (MM) conference.A 23-item survey using corresponding Likert-type scales was created. Faculty and residents were asked to anonymously complete the surveys in June 1999. Based on this information, specific modifications were made to the conference. The same survey was administered to faculty and residents in the Fall of 2000. Analysis was performed using Student t tests.Postsurvey findings showed residents felt eight components improved significantly (P0.05). Faculty noted nonsignificant improvement in nine survey items and decline in nine items (five unchanged).Changes in content and structure made to enhance our MM conference's educational value resulted in significant improvements as perceived by the surgical residents. Interestingly, these changes had only minimal impact on faculty perceptions.

Purpose To assess use of the combined just-in-time teaching (JiTT) and peer instruction (PI) instructional strategy in a residency program's core curriculum. Method In 2010-2011, JiTT/PI was piloted in 31 core curriculum sessions taught by 22 faculty in the Northwestern University Feinberg School of Medicine's general surgery residency program. JiTT/PI required preliminary and categorical residents (n=31) to complete Web-based study questions before weekly specialty topic sessions. Responses were examined by faculty members "just in time" to tailor session content to residents' learning needs. In the sessions, residents answered multiple-choice questions (MCQs) using clickers and engaged in PI. Participants completed surveys assessing their perceptions of JiTT/PI. Videos were coded to assess resident engagement time in JiTT/PI sessions versus prior lecture-based sessions. Responses to topic session MCQs repeated in review sessions were evaluated to study retention. Results More than 70% of resident survey respondents indicated that JiTT/PI aided in the learning of key points. At least 90% of faculty survey respondents reported positive perceptions of aspects of the JiTT/PI strategy. Resident engagement time for JiTT/PI sessions was significantly greater than for prior lecture-based sessions (z=-2.4, P=.016). Significantly more review session MCQ responses were correct for residents who had attended corresponding JiTT/PI sessions than for residents who had not (chi-square=13.7; df=1; P Conclusions JiTT/PI increased learner participation, learner retention, and the amount of learner-centered time. JiTT/PI represents an effective approach for meaningful and active learning in core curriculum sessions.

This resource is a 1-day course for incoming surgery residents to help decrease their anxiety and prepare them to their roles as surgical interns. The course is comprised of five distinct sessions with clear goals and objectives. Each session utilizes appropriate interactive techniques including direct questions, open discussions, roleplay, and simulation with structured debriefings to achieve its goals. The five sessions feature the SBAR (situation, background, assessment, recommendation) technique, answering pages, hand offs, advanced trauma life support, and a scavenger hunt. Residents described the course as important and enjoyable, and stated that it achieved its goals. The overall course rating was a 4.8 out of 5. Residents indicated that the course improved their self-confidence and understanding of their roles and duties. More residents passed the knowledge test after attending this boot camp (24% passed pretest versus 68% passed posttest).

Although the practice of medicine is increasingly a multidisciplinary effort, clinical teaching of medical students is accomplished primarily within a departmental structure.The purpose of this study was to identify subject matter within the clinical curriculum that could serve as focus for multidisciplinary teaching.A questionnaire was sent to 13 clerkship directors (representing required clerkships) at a large Midwestern medical school in which they were asked to rate a list of 631 patient problems as critical (primary), important (secondary), or "nice to know," relevant to their respective clerkship objectives.All clerkship directors completed the questionnaire. There were 523 items that were considered primary, and over 90% of these items were listed as either primary or secondary in more than 1 clerkship. Twelve topics were considered primary or secondary by at least 5 clerkship directors. Four clerkship directors identified 43 patient problems, and 3 clerkship directors identified 92 topics as primary or secondary clerkship objectives.In this study, listing of patient problems across clerkships demonstrated significant overlap of the clinical curriculum, suggesting multiple opportunities for faculty collaboration in clinical education.

Background: Surgical education peer-reviewed publications have markedly increased over the last decade. The purpose of this study was to review the surgical education literature published over the last 10 years and address the following questions: What subjects in surgical education tend to be studied? What are the most to least commonly employed research designs and statistics? Has there been a change in how research data are collected? Where are these studies published? Methods: A literature search encompassing surgical education papers published between January 1988 and August 1998 was performed. Four investigators coded qualifying abstracts on journal type, subject of research, data collection methods, research design, and statistics. Each investigator was asked to code 10 articles at the start of the study to assess interrater reliability. Results: A total of 420 abstracts were evaluated. Interrater reliability yielded percent agreements ranging from 82% to 96%. Curriculum and teaching were the most frequent topics studied (40%), followed by assessment (23%) and program evaluation (18%). Most research designs used were descriptive (41%). Experimental design has progressively increased from 2% in 1988–89 to 16% in 1998. A total of 551 statistical methods were accounted for in the 420 abstracts. The most common statistical analyses used were descriptive statistics (32%). The predominant mode of data collection was through testing or direct observations (34%). Survey instruments followed closely as a popular data collection method at 27%.The majority of papers were published in peer-reviewed surgical journals (64%),followed by medical education journals (22%) and “other” journals (14%). Conclusions: An analysis of the surgical education literature demonstrates the growing emphasis on the use of educational research to explore relevant issues and problems. Descriptive research is most popular, with an increasing trend in experimental research. Publication of educational research in peer-reviewed surgical journals is becoming more popular. This study informs those interested in the surgical education research literature of current trends, and what they need to know for a more critical appraisal of this body of literature.

Background: The specific surgical knowledge and skills students learn during surgical rotations are reconsidered in light of recent changes in medical school curricula. The purpose of this study was to determine the priorities of a surgical curriculum based on input from three groups: surgical faculty (SF), primary care faculty (PCF), and community-based, practicing primary care physicians (PCP). Methods: A questionnaire was developed in which SF (n = 54), PCF (n = 85), and PCP (n = 876) were asked to rank the importance of 145 areas of knowledge and 48 areas of clinical skills on a 5-point Likert-type scale. Responses were rank ordered by the mean of importance ratings for each group. Differences among groups were evaluated using ANOVA. Results: Response rates were best for faculty (100% SF, 88% PCF, 61% PCP). All three groups considered general surgery related topics and general skills very important. Primary care physicians and PCF consistently ranked otolaryngology, ophthalmology, and orthopedic topics and skills higher than did SF. Surgery faculty ranked invasive surgical procedures higher than did PCP while PCP ranked orthopedic procedural skills more highly. Conclusions: There is significant overlap among physicians about what medical students should learn during a surgical rotation. Differences between groups centered on surgical subspecialty knowledge and clinical skills. These results provide a broad perspective about required subjects for a core surgical clerkship curriculum, which should include surgical subspecialty training.

There is a paucity of performance-based assessments that focus on intraoperative decision making. The purpose of this article is to review the performance outcomes and usefulness of two performance-based assessments that were developed using cognitive task analysis (CTA) frameworks.Assessment-A used CTA to create a "think aloud" oral examination that was administered while junior residents (PGY 1-2's, N = 69) performed a porcine-based laparoscopic cholecystectomy. Assessment-B used CTA to create a simulation-based, formative assessment of senior residents' (PGY 4-5's, N = 29) decision making during a laparoscopic ventral hernia repair. In addition to survey-based assessments of usefulness, a multiconstruct evaluation was performed using eight variables.When comparing performance outcomes, both approaches revealed major deficiencies in residents' intraoperative decision-making skills. Multiconstruct evaluation of the two CTA approaches revealed assessment method advantages for five of the eight evaluation areas: (1) Cognitive Complexity, (2) Content Quality, (3) Content Coverage, (4) Meaningfulness, and (5) Transfer and Generalizability.The two CTA performance assessments were useful in identifying significant training needs. While there are pros and cons to each approach, the results serve as a useful blueprint for program directors seeking to develop performance-based assessments for intraoperative decision making.

Background Exploring the trends in surgical education research offers insight into concerns, developments, and questions researchers are exploring that are relevant to teaching and learning in surgical specialties. Objective We conducted a review of the surgical education literature published between 2002 and 2012. The purpose was 2-fold: to provide an overview of the most frequently cited articles in the field of surgical education during the last decade and to describe the study designs and themes featured in these articles. Methods Articles were identified through Web of Science by using “surgical education” and “English language” as search terms. Using a feature in Web of Science, we tracked the number of citations of any publication. Of the 800 articles produced by the initial search, we initially selected 23 articles with 45 or more citations, and ultimately chose the 20 articles that were most frequently cited for our analysis. Results Analysis of the most frequently cited articles published in US journals between the years 2002–2012 identified 7 research themes and presented them in order of frequency with which they appear: use of simulation, issues in student/resident assessment, specialty choice, patient safety, team training, clinical competence assessment, and teaching the clinical sciences, with surgical simulation being the central theme. Researchers primarily used descriptive methods. Conclusions Popular themes in surgical education research illuminate the information needs of surgical educators as well as topics of high interest to the surgical community.

Medical school faculty members are charged with the critical responsibility of preparing the future physician and medical scientist workforce. Recent reports suggest that medical school curricula have not kept pace with societal needs and that medical schools are graduating students who lack the knowledge and skills needed to practice effectively in the 21st century. The majority of faculty members want to be effective teachers and graduate well-prepared medical students, but multiple and complex factors-curricular, cultural, environmental, and financial-impede their efforts. Curricular impediments to effective teaching include unclear definitions of and disagreement on learning needs, misunderstood or unstated goals and objectives, and curriculum sequencing challenges. Student and faculty attitudes, too few faculty development opportunities, and the lack of an award system for teaching all are major culture-based barriers. Environmental barriers, such as time limitations, the setting, and the physical space in which medical education takes place, and financial barriers, such as limited education budgets, also pose serious challenges to even the most committed teachers. This article delineates the barriers to effective teaching as noted in the literature and recommends action items, some of which are incremental whereas others represent major change. Physicians-in-training, medical faculty, and society are depending on medical education leaders to address these barriers to effect the changes needed to enhance teaching and learning.

PURPOSE To directly compare the generalizability of medical students' performance scores under systematically varied station times in two surgery end-of-clerkship performance-based examinations. METHOD The participants were 36 third-year students randomly assigned to the first two rotations of the core surgery clerkship during 1991-92 at Southern Illinois University School of Medicine. The students rotated through a 12-station examination that employed standardized patients (SPs). In the first rotation, the student took six five-minute stations and six ten-minute stations. In the second rotation, the time lengths were reversed for the same stations. The students' total scores were based on (1) subscores on checklists that were completed by the SPs and (2) subscores on the students' written responses to short questions about each station (these responses were provided at station couplets that were five minutes long, regardless of station length). Generalizability coefficients were computed from the pooled rotation results to provide reliabilities for scores from the two station lengths. RESULTS Generalizability decreased in the ten-minute stations, mostly attributable to less variability among students' performances. The checklist subscores accounted for most of this variability, while couplet subscores remained stable between station lengths. CONCLUSION The longer station length actually decreased the generalizability of the scores by decreasing the variability among students' performances; thus, allocating different times to stations can affect the score reliability, as well as impact on the overall testing time, of performance-based examinations.

Background: The Objective Structured Assessment of Technical Skill (OSATS) is a multistation performance-based examination that assesses the technical skills of surgery residents. This study explores the implementation issues involved in remote administration of the OSATS focusing on feasibility and the psychometric properties of the examination. Methods: An eight-station OSATS was administered to surgical residents in Los Angeles and Chicago. The University of Toronto and the local institutions shared responsibility for organization and administration of the examination. Results: There was good reliability for both the checklist (α = 0.68 for LA, 0.73 for Chicago) and global rating forms (α = 0.82 for both sites). Both iterations also showed evidence of construct validity, with a significant effect of training year for the checklist and global rating forms at both sites (analysis of variance: F = 8.66 to 19.93, P

Background In 2008, the Surgical Council on Resident Education selected 33 residency programs to pilot its General Surgery Resident Curriculum Website Portal. The portal aims to reduce program variability in curricula, align teaching and learning with essential content, and improve resident study and performance. Methods Two online surveys were sent to all program directors and their residents before releasing the portal. Data from 32 programs and 899 residents (84%) were analyzed to determine the extent to which preimplementation characteristics supported the portal's rationale and illuminated barriers to its use and impact on learning. Results The need for curriculum content and access to online texts varied markedly across programs. Residents had easy onsite access to the Internet and used it heavily for immediate purposes. Fewer residents used the Web for planned activities and proactive study. On average, residents reported studying an hour or less a day. Conclusions The portal appears to serve curricular resource needs and may better direct resident study. Programs are advised to consciously integrate the SCORE curriculum and portal into residency training and faculty development.

Background The purpose of this study was to compare the intraoperative learning needs and educational resource use of junior and senior residents. Our goal was to gain a better understanding of the progression of learning needs in surgical training. Methods Residents (n = 125) completed a previously validated, 27-item survey indicating the following: (1) the extent to which traditional learning resources are used when preparing for cases in the operating room, and (2) which intraoperative management topics in which they believed they were deficient despite preoperative preparation. Results On a scale of 1 to 5, with 5 indicating frequent use, postgraduate year (PGY)-5 residents (n = 39) indicated surgical atlases (4.15; SD, .90) and surgical texts (4.15; SD, .90) were their most frequently used resources when preparing for a case in the operating room. In contrast, PGY-1 residents (n = 32) indicated anatomy atlases (3.97; SD, .93) and advice from colleagues (3.64; SD, .90) were their most frequently used resources when preparing for a case in the operating room. Despite the differences in how the PGY-5 group and the PGY-1 group prepared for a case, of 12 intraoperative management topics both groups believed they were the least prepared for instrument use/selection and suture selection. Conclusions Today's residents represent a heterogeneous group of individuals with different learning needs based on level of experience, knowledge, and learning style. Our study highlights unexpected but critical learning needs for senior-level residents that can and should be readily addressed.

Background The Patient Assessment and Management Examination (PAME) is a standardized patient examination designed to assess management skills of senior residents. This study explored the relationship between faculty and resident self-evaluation by using PAME. Methods Nine postgraduate year (PGY) 4 and PGY5 residents were examined with a 5 case PAME. Faculty rated interactions between residents and standardized patients and residents rated themselves based on review of audio-video recordings of their interactions. We examined correlations between faculty and resident self-assessments. Results Faculty and resident ratings of physical examination skills was the only competency that correlated significantly. Correlations were not significant for the other 15 competencies (Pearson r , –.197 to .262). Correlation was no better when examined within each case. Conclusions Although PAME may be a useful tool, this study suggests that even senior residents do not assess their performance as clinicians similarly to faculty. Further research is needed to better understand the source of these disagreements.

MOST ACADEMIC MEDICAL CENTERS (AMCS) HAVE missions that aim for excellence in 3 areas: education, research, and patient care. For decades, “triple-threat” faculty members were able to substantively contribute to all 3. However, heightened competition for research grants, changes in the health care delivery system, more sophisticated instructional and evaluation methods, and increased requirements to account for time and productivity make it less feasible for individual faculty to significantly contribute to all 3. Instead, faculty members are likely to focus their efforts in 1 or 2 of these areas. Although exceptions exist, physician faculty members focus either on patient care and research or on patient care and education. The problem is that promotion and tenure systems work well for faculty members pursuing the former but not as well for those focusing on the latter. For instance, 1 report found that, compared with research faculty, the odds of holding a higher academic rank were 85% lower for academic clinicians and 69% lower for teacher-clinicians. In another report, faculty devoting more than 50% of their time to clinical care were more likely to be on a nontenure track and more likely to report slower career progress than those devoting less than 50% of their time to clinical care. In a third report, the time to promotion was significantly shorter for physician-scientists (with 80% designated research time) than for clinician-scholars (with 30% research time). If this issue is not resolved, AMCs run the risk that faculty charged with patient care and education will view their contributions as having limited value, which will decrease motivation and reduce incentives for future efforts. In this article, we briefly review the evolution and early promise of faculty promotion and tenure tracks. We explore ways clinician-educators are designated within academic promotion and tenure models, address the value of these faculty to the AMC system and the challenges they face, examine measurement and evaluation issues, and offer suggestions for change. History of Faculty Tracks

Purpose To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. Method Faculty and residents at two surgery programs participated in parallel before-and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. Results Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). Conclusions Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

PURPOSE To examine the issue of test security when the same stations on an objective structured clinical examination (OSCE) are repeated across clerkship rotations. Specifically, is there a significant difference in students' scores on stations repeated in three or four rotations within a single academic year? METHOD The sample consisted of 15 stations in the OSCE given at the end of the third-year surgery clerkship at the Southern Illinois University School of Medicine from 1989-90 through 1993-94. Each station was administered three or four times a year. One-way analyses of variance with contrast coding to test for linear trends were used. Results were considered significant at or below the .05 level. RESULTS Only three of the 15 stations showed significant linear trends. A two-part couplet orthopedic station showed a significant decreasing linear trend (p=.0001). Two stations showed significant increasing linear trends: a general surgery couplet station (p=.0004) and a plastic surgery station with an essay question (p=.0253). CONCLUSION There was no consistent evidence that students scored increasingly higher on OSCE stations repeated throughout the year. Thus, it would appear that a clerkship can repeat OSCE stations within an academic year without risk of a trend toward increasing scores.

BACKGROUND Pauses (wait time) after asking questions in pre-college classes result in improved discussion and answer accuracy. The authors hypothesized that this would extend to medical students. METHOD Third-year surgery clerks were randomized to three-second or six-second wait times after questions asked of them during a scripted lecture. Students were randomized within each session to answer 21 scripted questions. Students also completed a post-lecture written examination. RESULTS Correct responses ranged from 17% to 100% for oral and 22% to 100% for written questions. Answer accuracy could not be distinguished between three- and six-second wait times for oral or written questions. CONCLUSIONS The benefit of increasing wait times from three to six seconds appears not to extend to medical students. This may represent evolution of learning or different learning modes in medical students. Alternatively, maximum benefit may be achieved in medical students with shorter wait times.

We examined three research questions: How do residents' debts and savings compare to the general public? How do surgical residents' financial choices compare to other residents? How may institutions help residents' personal financial decisions?The Survey of Consumer Finances was modified and piloted tested to elicit financial information. The instrument was completed by 612 residents at 8 programs.Only 60% of residents budgeted expenses, and 25% and 10% maintained cash balances611 dollars and unpaid credit card balances10,000 dollars, respectively. Compared with controls, residents held greater median ratios of debt to household income (2.46 vs. 1.06, P0.0001), fewer assets to income (0.64 vs. 2.28, P0.0001), less net wealth to income -1.43 vs. 0.90, P0.0001), and lower retirement savings balance to household income (0.01 vs. 0.12, P0.0001). Surgery residents were the least financially conservative group. Mean annual resident contributions to retirement accounts were $1532 higher at institutions with versus without retirement plans (P0.01).Resident debts are higher and savings lower than the general public. This behavior is most common among surgery residents. Residents save more for retirement when they are eligible for tax-deferred retirement plans. Graduate medical programs should instruct residents on financial management.

Previous research has shown that the majority of medical schools do not formally teach medical writing, including writing of progress notes. The purpose of this research was to determine the impact consistently provided objective feedback has on enhancing the quality of student notes. A static group‐comparison design was used to determine the quality of progress notes written by students who received structured written feedback versus those given traditional feedback. Twenty patient names were selected from students’ logbooks from both groups, totaling 40 patient charts. Progress notes were blindly reviewed by 1 faculty member, using a checklist instrument, which was studied for evidence of reliability and validity. Results showed a statistically significant (p = .05) difference between the groups’ abilities to write assessment and plan portions of a progress note, but no differences were noted on student abilities to document subjective data. Findings suggest some students often have an inability to inte...

A comprehensive and up-to-date curriculum requires periodic formal review to ensure it continues to meet learners' needs.This study describes a model for evaluating a surgery clerkship curriculum designed to determine the appropriateness of its learning objectives to the general professional education of a physician.A survey was mailed to graduates who pursued generalist residencies. Respondents estimated the number of patients encountered annually with specified presenting complaints or disease entities and the percentage of time these were referred to surgeons. For 23 technical procedures, respondents estimated the frequency done annually and whether remaining proficient in the skill was considered important.The majority of graduates reported the need to remain proficient in 19 technical procedures. Numerous patient problems were identified as requiring careful instruction so that learners know when and when not to refer for surgical intervention.The clerkship was modified to include skills and topics not previously included or appropriately emphasized.

Background: This study was designed to examine the validity of a surgery clerkship’s performance evaluation system. The study also assessed faculty members’ confidence in how clerks are evaluated and promotion decisions made. Methods: Student files from five classes (n = 339) were analyzed. A 25-item published survey designed to study faculty perceptions of a student evaluation system was distributed. Chi-square tests of independence and descriptive statistics were used. Results: Faculty survey results showed faculty perceptions of strengths and weaknesses in the evaluation system. Significant relationships were found with prior performance indicators and clerkship performance yielding evidence of concurrent and predictive validity. Conclusions: Findings provide guidance for enhancing how clerks are evaluated as well as specific profiles of students who may need special attention or additional challenges during the surgery clerkship. This study provides a model for other clerkships to assess their student evaluations systems.

To develop, implement, and evaluate a course for improving the teaching skills of surgery residents.Responses from residents at four general surgery training programs to a needs assessment survey were used to develop a two-day course for improving teaching skills. Residents at two surgical training programs were randomly assigned to experimental and control groups, and experimental residents participated in and evaluated the newly devised course. Six to seven months later, experimental and control residents' teaching performances were evaluated using a five-station objective structured teaching evaluation (OSTE). Differences between the residents' performances were calculated using Mann-Whitney U, chi-square analysis, or Fisher's exact test.Participating residents rated the course highly. They considered the interactive nature of the course its greatest strength. As measured by the OSTE, the performances of the residents differed least significantly in the feedback station, where the residents in the experimental groups showed significant improvement on only one of seven items at one institution, and only one of nine items at the other. The greatest differences occurred in the microskills teaching station, where the residents at one institution performed significantly better than did their control counterparts on four of five items and in overall performance.This study demonstrates the value of a needs assessment in developing a course to improve residents' teaching skills. Such courses must provide active learning with opportunities for practicing skills and, following the course, ongoing feedback to maintain changes in teaching behaviors. The curriculum developed in this study has been put into a transportable form that includes an instructor's manual providing guidelines and suggestions for implementation.

Typically, the primary instructional method for ambulatory care education is direct interaction between a preceptor and a learner during a patient encounter. This paper describes instructional strategies teachers and learners can use in ambulatory care training that can occur before or after scheduled clinic hours, thus providing instruction without disrupting a preceptor's busy clinic. First, they describe how preceptors and clerkship or residency-program directors can orient learners prior to their arrival at assigned sites, so that learners are better prepared to assume their patient-care responsibilities. Then they discuss strategies for making use of various types of conferences and independent learning activities to enhance learners' clinical experiences. Conferences and independent study projects that occur before clinic hours can help learners bring a higher level of thinking and clinical sophistication to their role in the ambulatory care site; conferences and independent study activities that occur after clinic hours give learners an opportunity to reinforce and expand on what they have learned during clinic. In this way, learners' educational experiences are enhanced, the best use is made of preceptors' time and expertise, and clinic efficiency is not disrupted.

Medical education in the ambulatory care setting is characterized in part by the question of how to ensure educational effectiveness while simultaneously providing high-quality, cost-effective patient care. The constraints associated with managed care have only served to escalate the intensity of this dilemma. However, in spite of the difficulties faced by ambulatory care preceptors, there are educationally sound and time-efficient strategies clinical teachers may employ to improve ambulatory care education. Emphasizing the basic three-step process of planning, teaching, and reflection, the authors describe five such strategies: "wave" scheduling, orienting learners to patients, having learners do their case presentations in the examination room, employing the microskills of the "one-minute preceptor," and effectively reflecting on one's teaching in order to develop effective teaching scripts. Research in ambulatory care learning has indicated that learners must be given significant roles in patient care and that preceptors must observe trainees as they care for patients so that they can provide trainees with helpful feedback. Employing these strategies in the ambulatory care setting will help educators to accomplish these two objectives while minimizing disruption to cost-effective, high-quality clinical practice.

Background In August 1993 the American College of Surgeons sponsored a course entitled “Surgeons as Educators” (SAE) aimed at equipping academic surgeons with the knowledge and skills necessary to enhance surgical education administration, curriculum, teaching, and evaluation. Methods The instructional design model used to construct the course called for a formal needs assessment to determine the importance, current skill level, and priority of what needed to be learned to be an effective educator. The needs assessment was accomplished using a job analysis and questionnaire approach. The 68item questionnaire was mailed to 320 academic surgeons representing eight medical schools. Results A 62% response rate was achieved. Results indicated the education-related tasks or activities that faculty felt were important to their careers, as well as their perceived level of development in each area. Descriptive statistics were used to summarize the responses that were critical to the SAE faculty in helping prioritize, sequence, and time ration course content. Collective results became the foundation for developing the SAE curriculum by the course's five faculty members. Conclusions A well-done needs assessment does not necessarily guarantee course success; however, it is the first and critical step to planning an educationally sound faculty development course or program designed for adult learners.

Background Research has shown that medical students are generally accepted by patients, but specific details that elucidate patient perceptions of the advantages and disadvantages to student involvement have not been documented. This study was designed to determine variables that influence patient satisfaction with students. Patients and Methods Patients were interviewed by one of two faculty members using a questionnaire-style format that covered 12 variables regarding patient care. Patients were asked to rate the extent to which medical students helped or hindered their hospital stay, with regard to the 12 variables. Results Patients' attitudes were favorable regardless of the students' extent of clinical experience or clinical abilities or the patients' age or length of hospital stay. Patients reported that students spent time with them and answered their questions. Most patients stated that they would allow students to participate in their future hospital care. Conclusion Positive patient-student interactions can have important effects on patients' expectations and their acceptance of future encounters with students. This fact is becoming increasingly important due to the changes in health care and the decreasing incidence of inpatient surgical encounters.

The purpose of the study on which this article is based was to determine differences in interviewing skills between third-year medical students who were provided structured, evaluative feedback and those assessed via traditional feedback methods. Experimental group students received a written assessment for every orthopedic hospital patient worked up, while the control group received traditional assessment feedback. Comparison measures were made based on student performance on a simulated patient examination. Results indicated that students who were provided structured feedback scored significantly higher (p less than .05) than those assessed by traditional assessment means.

The purpose of the study reported in this article was to examine first-year medical students' attitudes toward their free time and how these attitudes relate to self-reported stress levels. Members of the entering class of 1982-83 at the Southern Illinois University School of Medicine were asked to complete surveys designed to measure perceived stress and leisure attitudes. The results indicated a significant negative correlation between the total leisure scale and the stress scale. This correlation supported the hypothesis that students who perceived that they had high stress levels scored low on the leisure attitude scale. These findings suggest a need for educating students about the benefits of well spent leisure time as a stress management mechanism that is important in coping with daily pressures and anxieties.

A fourth-year curriculum survey was forwarded to surgery resident program directors, clerkship directors, and residents. A 37 percent response rate was achieved with comparable representation of each respondent type from 130 institutions. The purpose of the survey was to determine how each of the 3 respondent types rated the importance of 42 subjects or clinical experiences to be taken in the fourth year of medical school for students pursuing surgical careers. The survey also included questions on such issues as externships, early match programs, and guidelines used at their institutions for their fourth year students. The results indicated a significantly high agreement among respondent types on what should comprise surgical students' fourth-year curriculum content. The majority feel the fourth year should be planned by students with an electives advisor to equally add breadth to their general professional education as well as provide prerequisite experiences in the career-chosen discipline.

In the research reported here, the authors compared (a) the abilities of third- and fourth-year students and physicians to solve problems requiring review of current literature with (b) the information-seeking procedures of the three groups in researching those problems. The subjects were given a patient care question to answer, and logs were maintained to document their information-seeking processes. The findings indicated no differences among the three groups in accurately solving patient care problems; a difference in information-seeking processes between third-year students and physicians; and no differences between third-year and fourth-year students or between physicians and fourth-year students in their approaches to seeking information. Mean scores indicated that all three groups need additional training on accessing the literature in making patient care decisions.