Search

PURPOSE To determine the extent to which systematic differences not related to group differences in clinical competence could be observed in the results of six administrations of the same multiple-station test of clinical skills. METHOD The same 16 stations were administered as part of the Ontario International Medical Graduates Program clinical skills screening examination on three different occasions, and on each occasion in two sessions. The interval between the first (fall 1986) and second (March 1990) administrations was four years, and that between the second and third (July 1990) administrations, four months. International medical graduates were the candidates in the first two administrations; the third administration was to a combined sample of fourth-year clinical clerks and interns. A different item functioning (DIF) approach with sessions within administration as the grouping variable was used to describe the extent of differences unrelated to clinical competence in the results of the different test administrations. For the generalizability and DIF calculations the designs were balanced by sampling down to 33 cases in each session. RESULTS Station DIF values varied considerably from station to station. DIF effects between sessions with administrations were less than those between administrations, and were less between the two administrations separated by four months than between the two administrations separated by four years. CONCLUSION Hypotheses concerning the relative magnitudes of station DIF and total-test aggregated values of DIF, as a function of the time intervals between test occasions, were substantiated, demonstrating that the greater the time interval between test administrations, the greater the magnitude of DIF.

Background The objective structured clinical examination (OSCE) has become an accepted technique for the evaluation of clinical competence in medicine. Although advances have been made in our knowledge of the psychometric aspects of the OSCE, extremely little has been written about feasibility and cost issues. Given the current economic imperative to control costs and the extremely scant literature on the costs of developing and administering an examination in medicine, the authors felt it timely and relevant to explore issues related to the cost of the OSCE. Method In 1991-92 and in 1992-93, costs and time requirements to implement and administer a structured oral (SO) examination and a six-station OSCE for a surgical clerkship at the University of Toronto Faculty of Medicine were gathered by review of invoices, interviews with those involved, and perusal of diaries kept by staff. Results To develop and administer the six-station OSCE, 327.5 hours of staff and faculty time were required for each rotation of surgical clerks (8.2 hours per student). The SO examination required 110 hours of staff and faculty time (2.75 hours per student). Direct expenses for the OSCE amounted to U.S. $6.90 per student per station, compared with no direct expense for the SO examination. Conclusion The OSCE was more time-consuming and more expensive in human and material costs than the SO examination. However, costs of the OSCE can be substantially reduced from approximately U.S. $35 to U.S. ! per student per station if test developers, standardized patients, support staff, and examiners can donate their time. The authors compare the costs and time requirements of their OSCE with those of other OSCEs reported in the literature, and they provide guidelines to assist educators in deciding whether the costs of an OSCE are justifiable in the educators' individual settings.

A major impediment to the use of the objective structured clinical examination (OSCE) is that it is a labor-intensive and costly form of assessment. The cost of an OSCE is highly dependent on the particular model used, the extent to which hidden costs are reported, and the purpose of the examination. The authors detail hypothetical costs of running a four-hour OSCE for 120 medical students at one medical school. Costs are reported for four phases of this process: development, production, administration, and post-examination reporting and analysis. Costs are reported at two ends of the spectrum: the high end, where it is assumed that little is paid for by the institution and that faculty receive honoraria for work put into the examination; and the low end, where it is assumed that the sponsoring institution defrays basic costs and that faculty do not receive honoraria for their participation. The total costs reported for a first-time examination were $104,400 and $59,460 (Canadian dollars) at the high and low ends, respectively. These translate to per-student costs of $870 and $496. The cost of running an OSCE is high. However, the OSCE is uniquely capable of assessing many fundamental clinical skills that are presently not being assessed in a rigorous way in most medical schools.

The performance of foreign medical graduates on multistation standardized patient-based tests and used to determine the validity and generalizability of global ratings of their clinical competence made by expert examiners. Data were derived from the entrance examinations of the 1989 and 1990 applicants to the Ontario Pre-Internship Program and the exit examination of 24 participants from the 1989 cohort. For each candidate, the examiners completed a detailed checklist and two five-point global ratings dealing with the candidate's approach to the patients' problem and attitude toward the patient. Generalizability coefficients for both ratings were satisfactory and stable across cohorts. Construct validity of the global ratings was demonstrated by comparing entry and exit ratings and by evidence of significant and positive correlations between the global ratings and total test scores. Tentative evidence of criterion validity of the global ratings was demonstrated. These findings suggest that global ratings by expert examiners can be used as an effective form of assessment in multistation standardized patient examinations.

In the fall of 1986, the Ministry of Health of the province of Ontario, Canada, implemented a special 36-week internship preparation program for graduates of foreign medical schools. The program accepts 24 candidates per year and is offered at the five Ontario medical schools. At the time of this study, two cohorts of participants had completed the program. As part of the evaluation process, each participant was administered a battery of the same Objective Structured Clinical Examination-type test stations, prior to and on completion of the pre-internship experience. Repeating the same examination permitted investigation of the predictive and construct validities of the clinical skills tests, as well as the stability of the validity and high reliability estimates that emerged from the testing of the first cohort. The results provide convincing evidence of the testing approach's predictive and construct validity and reliability.