Search

A study has been undertaken to determine whether it is possible for a set of standardized (simulated) patients to visit general practitioners, without being detected, in a health care system where doctors have fixed patient lists. Since sending standardized patients into doctors' offices is a new way to assess the performance of general practitioners; this paper describes in detail the methodology that has been used for visits. The paper looks first at the general preparation for visits and secondly at the specific preparation concerning the fine detail of the individual visit. The method was tested in 156 consultations with 39 general practitioners and in no cases were the standardized patients detected. None of the doctors visited felt offended and all were prepared to cooperate in future studies with standardized patients. It is concluded that the standardized patient method, following the step-by-step procedure described, is feasible in actual practice.

A study was undertaken whereby a set of standardized (simulated) patients visited general practitioners without being detected, in a health care system where doctors had fixed patient lists. Thirty nine general practitioners were each visited during normal surgery hours by four standardized patients who were designed to be indistinguishable from real patients. The objective of the study was to see whether the actual performance of general practitioners, as assessed by standardized patients, met predetermined consensus standards of care for actual practice. The patients presented standardized accounts of headache, diarrhoea, shoulder pain and diabetes. The mean group scores of the doctors on the predefined standards of care for the different complaints ranged from 33 to 68%. The results show that standardized patients may be the method of choice in the assessment of the quality of actual care of doctors. It is hypothesized that the substandard scores of the doctors do not reflect inadequate competence, but are a result of the difference between competence and performance.

Objectivity has been one of the hallmarks in the assessment of clinical competence in recent decades. A consistent shift can be noticed in which subjective measures are being replaced by objective measurement methods. In the transition from subjective to objective methods trade-offs are involved, both in the effort expended and in the range of behaviours assessed. The issue of the presumed superiority of objective measures is addressed in two successive papers. In this paper a distinction is made between objectivity as a goal of measurement, marked by freedom of subjective influences in general, and objectivity as a set of strategies designed to reduce measurement error. The latter has been termed objectification. The central claim of this paper is that these two approaches to assessment do not necessarily coincide. By reviewing a number of studies comparing subjective and objectified measurement methods, the claim of the supremacy of the latter with respect to reliability is discussed. The results of these studies indicate that objectified methods do not inherently provide more reliable scores. Objectified methods may even provide unwanted outcomes, such as negative effects on study behaviour and triviality of the content being measured. The latter issues, related to validity, efficiency and acceptability, are discussed in a second paper.

In a previous article the distinction is made between objectivity and objectification. Objectivity is considered a generic goal of measurement, marked by freedom of subjective influences in general, whereas the latter term is used to describe strategies to reduce measurement error. A survey of several studies indicated that objectified methods are not intrinsically more reliable than subjective measures. In this paper the consequences of objectification are analysed for issues related to validity, efficiency, transparency, and effect of these methods on students and teachers. Several studies comparing objectified and subjective methods are surveyed for this propose. The studies indicate that--as in the previous article on reliability--objectification and objectivity are not identical, and that there are many pitfalls in the objectification of measurement procedures. As a consequence, it is argued that objectified methods should not exclusively be chosen on the basis of their unconditional appeal to objectivity, but that the application of measurement methods should follow the specific purpose of the testing situation. In the context of the testing situation, arguments against and in favour of objectification should be weighted, and trade-offs are to be evaluated. The outcome of this evaluation may vary from situation to situation, and from institution to institution.

In the debate about 'what is a competent general practitioner?' little attention has been paid ot the actual practice situation of general practitioners. This paper, based on the 18 most important studies in the literature about medical competence, tries to re-initiate this debate by proposing a clear distinction between 'competence', (what a physician is capable of doing) and 'performance' (what a physician does in his day-to-day practice). With this distinction we looked at whether studies defined both competence and performance, how they dealt with these concepts, what measurement instruments were used and what the conclusions of the studies were. Although it is the common reasoning that competence is a good predictor of performance this concept could not be affirmed. This survey shows that the majority of studies use wrong concepts and come to invalid conclusions. With the empirical distinction between competence and performance however, this paper proposes new directions for the quality assessment of general practitioners.

Growth patterns of medical interviewing skills during a 6-year undergraduate curriculum are assessed by studying 563 medical students taken from five year-groups, interviewing simulated patients. In a cross-sectional, quasi-experimental design their skills are rated by means of the Maastricht History-taking and Advice Checklist (MAAS), an observation instrument which measures five categories of interviewing skills pertaining to initial medical consultations. The findings suggest that the skills for 'history-taking', 'presenting solutions' and 'structuring of the interview' are effectively learned. These learning effects result from a continuous small group teaching program with expert and peer review of videotaped encounters with simulated patients. The teaching effects of this program seem less for the skills pertinent to the phase of 'exploring the reasons for encounter' and to the 'basic interviewing skills', because the students' growing medical knowledge and the increasing ability to solve medical problems exert a counteracting influence on the acquisition of these easily deteriorating skills. The results might be helpful to curriculum planners in order to make their programs for medical interviewing skills more effective.

Variation in the accuracy of examiner judgements is a source of measurement error in performance-based tests. In previous studies using doctor subjects, examiner training yielded marginal or no improvement in the accuracy of examiner judgments. This study reports an experiment on accuracy of scoring in which provision of training and background of examiners are systematically varied. Experienced teaching staff, medical students and lay subjects were randomly assigned to either training or no-training groups. Using detailed behavioural check-lists, they subsequently scored videotaped performance on two clinical cases, and accuracy of their judgments was appraised. Results indicated that the need for and effectiveness of training varied across groups: it was least needed and least effective for the teaching staff group, more needed and effective for medical students, and most needed and effective for the lay group. The accuracy of the lay group after training approached the accuracy of untrained teaching staff. Trained medical students were as accurate as trained teaching staff. For teaching staff and medical students training also influenced the nature of errors made by reducing the number of errors of commission. It was concluded that training varies in effectiveness as a function of medical experience and that trained lay persons can be utilized as examiners in performance-based tests.

This study investigated the reproducibility of scores on the University of Limburg Skills Test, using data drawn from tests administered to medical students in all years of training during 1984-85, 1985-86, and 1986-87. Generalizability analyses indicated that, while inter-rater reliability is good, a minimum of three to four hours of testing time is required to obtain reproducible scores due to content-specificity of examinee performance.

Performance tests are logistically complex and time consuming. To reach adequate reliability long tests are imperative. Additionally, they are very difficult to adapt to the individual learning paths of students, which is necessary in problem-based learning. This study investigates a written alternative to performance-based tests. A Knowledge Test of Skills (KTS) was developed and administered to 380 subjects of various educational levels, including both first-year students and recently graduated doctors. By comparing KTS scores with scores on performance tests strong convergent validity was demonstrated. The KTS failed discriminant validity when compared with a general medical knowledge test. Also the identification of sub-tests discriminating between behavioural and cognitive aspects was not successful. This was due to the interdependence of the constructs measured. The KTS was able to demonstrate differences in ability level and showed subtle changes in response patterns over items, indicating construct validity. It was concluded that the KTS is a valid instrument for predicting performance scores and could very well be applied as supplementary information to performance testing. The relative ease of construction and efficiency makes the KTS a suitable substitute instrument for research purposes. The study also showed that in higher ability levels the concepts which were meant to be measured were highly related, giving evidence to the general factor theory of competence. However, it appeared that this general factor was originally non-existent in first-year students and that these competencies integrate as the educational process develops.