Search

In 1998, the authors, acting on behalf of the National Board of Medical Examiners (NBME), undertook a review of the scoring policy for the United States Medical Licensing Examination (USMLE). The main goal was to determine the likely effect of changing from numeric score reporting to reporting pass-fail status. Several groups were surveyed across the nation to learn how they felt they would be affected by such a change, and why: all 54 medical boards; 1,600 randomly selected examinees (including 250 foreign medical graduates) who had recently taken either Step 1, Step 2, or Step 3 of the USMLE; 2,000 residency directors; the deans, education deans, and student affairs deans at all 125 U.S. medical schools accredited by the Liaison Committee on Medical Education; and all 17 members of the Council of Medical Specialty Societies. Responses from the different groups surveyed varied from 80% to a little less than half. The authors describe in detail the various views of the respondents and their reasons. Some members in each group favored each of the reporting formats, but the trend was to favor numeric score reporting. The majority of the responding examinees desired that their USMLE scores be sent to them in numeric form but sent to their schools and to residency directors in pass-fail form. Based on the responses and a thorough discussion of their implications, the Composite Committee (which determines USMLE score-reporting policy) decided that there is no basis at this time for changing the current policy, but that it would review the policy in the future when necessary.

The National Board of Medical Examiners (NBME) has reviewed its procedure for setting pass-fail standards in conjunction with the introduction of its comprehensive Part I and Part II examinations in 1991. This report gives background information on the procedures used for the past decade to set pass-fail standards for the Part I and Part II examinations, an overview of the NBME's research on standard setting, under way since 1987, and a statement of its plans for determining pass-fail standards for these examinations. In 1981 the NBME changed from the norm-referenced standard, used since the 1950s, to a criterion-group approach to setting pass-fail standards. Although the criterion-group system resulted in more stable standards, it still meant that the standard moved whenever the performance of the reference group changed. After conducting research, surveying constituencies, and examining alternatives, the NBME has adopted a new standard-setting plan that has the following components: a content-based standard-setting procedure; determination of standards by an appropriate group; use of a fixed standard; and periodic review of standards and standard-setting procedures. This new process will produce three types of improvements: it will incorporate deliberations informed by a wide range of information, including content review; annual review of examinees' performances and pass-fail results and triennial restudy of the process will add further quality control; and a fixed standard will mean that comparable performances will be required across administrations in order to pass.

The National Board of Medical Examiners (NBME) reviewed its procedures for setting pass-fail standards in conjunction with the introduction of its comprehensive Part I and Part II examinations in 1991. A component of the review was the 1990 survey of 12 NBME constituencies to assess opinions about pass-fail rates and standard setting. The survey responses, generally consistent across groups, revealed (1) support for a more even distribution of fail rates across the three examinations, (2) support for a fixed standard, with pass-fail rates that change depending on the performance of a cohort of examinees, and (3) a favorable reaction to setting performance standards based on a review of the examination content. These responses may serve as a resource and a frame of reference for determining the standards for the comprehensive Part I and Part II examinations.

The National Board of Medical Examiners has developed computer-based examination formats for use in evaluating physicians in training. This paper describes continued research on these formats including attitudes about computers and effects of factors not related to the trait being measured; differences between paper-administered and computer-administered multiple-choice questions; and the characteristics of simulation formats. The implications for computer-based testing and further research are discussed.

A computerized examination, including traditional multiple-choice questions and interactive, patient-physician simulations (CBX), was used to assess the clinical problem-solving skills of first-year residents. Results did not provide evidence of a relationship between residents' MCQ or CBX performance measures and levels of computer experience or computer anxiety.