Search

• Because it corrects for chance agreement, kappa (K) is a useful statistic for calculating interrater concordance. However, K has been criticized because its computed value is a function not only of sensitivity and specificity, but also the prevalence, or base rate, of the illness of interest in the particular population under study. For example, it has been shown for a hypothetical case in which sensitivity and specificity remain constant at .95 each, that K falls from .81 to .14 when the prevalence drops from 50% to 1%. Thus, differing values of K may be entirely due to differences in prevalence. Calculation of agreement presents different problems depending on whether one is studying reliability or validity. We discuss quantification of agreement in the pure validity case, the pure reliability case, and those studies that fall somewhere between. As a way of minimizing the base rate problem, we propose a statistic for the quantification of agreement (the Y statistic), which can be related to K but which is completely independent of prevalence in the case of validity studies and relatively so in the case of reliability.

• We examined the level of agreement between diagnoses derived from data gathered by lay interviewers using the Diagnostic Interview Schedule (DIS) in a general population survey (the Epidemiologic Catchment Area project) and both DIS and clinical diagnoses made by psychiatrists. Overall percent agreement between the lay DIS and the psychiatrists clinical impression ranged from 79% to 96%. The chancecorrected concordance was .60 or better for eight of the 11 diagnoses. Specificities were all 90% or better. Sensitivities were lower, but lay results showed a bias for only two diagnoses: major depression was significantly underdiagnosed and obsessive illness was overdiagnosed. We compared the present results with those of previous studies from clinical settings. We explored possible reasons for disagreement and discussed the implications of the findings for psychiatric epidemiologic research.

• Several methods are used to minimize and measure error in the NIMH Epidemiologic Catchment Area program. Sampling methods involve the inclusion of group quarters such as prisons, nursing homes, and mental hospitals in the sample frame and the use of probability sampling throughout. Interviewing, methods include use of identical diagnostic interview protocols, centralized training of interview supervisors, standard instructions to interviewers, and reinterview of a subsample by clinicians. In the area of completion, the methods include a 75% to 80% respondent completion rate, a 95% or greater completion rate for individual questions, use of informant interviews where necessary, and statistical adjustments to correct for low completion rates in some subgroups. Analytic methods include use of a computerized diagnostic algorithm, common estimation formulas on identically formatted data files, and estimation of exact variances that take account of the multistage sample design.

• Lifetime rates are presented for 15 DSM-III psychiatric diagnoses evaluated in three large household samples on the basis of lay interviewers' use of the Diagnostic Interview Schedule. The most common diagnoses were alcohol abuse and dependence, phobia, major depressive episode, and drug abuse and dependence. Disorders that most clearly predominated in men were antisocial personality and alcohol abuse and dependence. Disorders that most clearly predominated in women were depressive episodes and phobias. The age group with highest rates for most disorders was found to be young adults (aged 25 to 44 years). Correlates with race, education, and urbanization are presented.

• The new DSM-III and the Feighner criteria give definitions of schizophrenia that are restrictive in that they require six months of illness before the diagnosis can be made. The definitions are compared with definitions of schizophrenia in the Research Diagnostic Criteria and CATEGO in terms of predictive validity using a cohort of 125 psychotic patients from the United States-United Kindgom Diagnostic Project. The results showed that while the DSM-III and Feighner definitions were more restrictive, they also had a higher degree of predictive specificity than the other two definitions. No patient diagnosed by DSM-III or Feighner criteria showed a change in diagnosis over time, and these patients were more likely to have presistent schizophrenic symptoms and to have social handicap. The appropriateness of the requirement of six months of illness and the practicality of using outcome as a validity measure are discussed.

• A psychiatric diagnostic interview that can be reliably and validly administered by nonpsychiatric physicians and lay interviewers has both research and clinical applications. We examined the interrater reliability and procedural validity of the Renard Diagnostic Interview (RDI), an instrument designed for these purposes. Randomly selected psychiatric inpatients were interviewed once by a psychiatrist using our standard departmental research interview and were then given RDIs by two psychiatrists, two lay interviewers, or one of each. The reliability of the RDI is estimated by examining diagnostic concordance for the two RDI interviews. Procedural validity is estimated by examining diagnostic concordance between the RDI and the traditional departmental interview. Both reliability and procedural validity were found to be high, and the study demonstrates that lay interviewers using the RDI after a brief period of training can obtain accurate diagnostic information.

• A new interview schedule allows lay interviewers or clinicians to make psychiatric diagnoses according to DSM-III criteria, Feighner criteria, and Research Diagnostic Criteria. It is being used in a set of epidemiological studies sponsored by the National Institute of Mental Health Center for Epidemiological Studies. Its accuracy has been evaluated in a test-retest design comparing independent administrations by psychiatrists and lay interviewers to 216 subjects (inpatients, outpatients, ex-patients, and nonpatients).

• Eleven black and 19 white men with conditions diagnosed as manic-depressive disease, manic type were given a systematic psychiatric interview. In addition, as many of their first-degree relatives as could be contacted were also interviewed. Demographic, clinical, and family history variables were compared for the two races.With the exception of a greater preponderance of alcoholism in the paternal relatives of the black men, few differences were found between the two groups in terms of the variables studied. It was concluded that the clinical and familial expression of bipolar affective disorder is similar in the two races.

From all US Army enlistees leaving Vietnam in September 1971, a random sample of 943 men was selected. Of these, 470 represented a "general" sample of all enlistees returning at that time, and 495 represented a "drug-positive" sample whose urine samples had been positive for opiates at the time of departure. We attempted to locate and personally interview all of the men in the samples.Results indicate that before arrival, hard drug use was largely casual, and less than 1% had ever been addicted to narcotics. In Vietnam, almost half the general sample tried narcotics and 20% reported opiate addiction. After return, usage and addiction essentially decreased to pre-Vietnam levels. We discuss the use of nonnarcotic drugs, predictors and correlates of drug use in the samples, and the relationship of drugs to post-Vietnam social adjustment.

Thirty male manic probands and their first-degree relatives were investigated by means of a systematic psychiatric interview in order to determine the frequency of familial illness and possible mode of transmission of bipolar manic-depressive disorder. The infrequency of male-to-male transmission of affective disorder and the relative preponderance of the illness on the maternal side of the family both support similar previous findings that are suggestive of sex linkage in this illness. The incidence of alcoholism was found to be high and more prevalent on the paternal side of the families.

• The Composite International Diagnostic Interview (CIDI), written at the request of the World Health Organization/US Alcohol, Drug Abuse, and Mental Health Administration Task Force on Psychiatric Assessment Instruments, combines questions from the Diagnostic Interview Schedule with questions designed to elicit Present State Examination items. It is fully structured to allow administration by lay interviewers and scoring of diagnoses by computer. A special Substance Abuse Module covers tobacco, alcohol, and other drug abuse in considerable detail, allowing the assessment of the quality and severity of dependence and its course. This article describes the design and development of the CIDI and the current field testing of a slightly reduced "core" version. The field test is being conducted in 19 centers around the world to assess the interviews' reliability and its acceptability to clinicians and the general populace in different cultures and to provide data on which to base revisions that may be found necessary. In addition, questions to assess International Classification of Diseases, ninth revision, and the revised DSM-III diagnoses are being written. If all goes well, the CIDI will allow investigators reliably to assess mental disorders according to the most widely accepted nomenclatures in many different populations and cultures.

• A clinical reexamination by psychiatrists is a useful comparison for exploring lay Diagnostic Interview Schedule—derived psychiatric diagnoses in general population surveys, but as a validity standard psychiatric reexamination is less than ideal. There are many potential sources of disagreement that have nothing to do with the validity of either the lay or the psychiatrist examination. Another approach to comparing lay and psychiatrist diagnoses is to examine their relative predictive power. We describe such a comparison using outcome variables derived from the one-year follow-up examination of Epidemiologic Catchment Area respondents done at the St Louis site. We examine several outcome variables across nine diagnostic categories. Within the limits of the available data, lay and psychiatrist diagnoses appear to be essentially equal in terms of the number of outcomes better predicted, and few of the differences in predictive power approach statistical significance. We discuss the implications of these findings for the analysis of lay interviewer Diagnostic Interview Schedule—derived diagnoses.

• To test the contribution of the criterion of six months of illness to the predictive usefulness of the DSM-III diagnosis of schizophrenia, we compared, from a sample of 121 patients with functional psychosis, those who had been ill for six months or more at index but did not meet the nontemporal DSM-III criteria and those who met the full criteria. We found that while the six-month criterion is an important component, it is not the sole reason for the predictive power of these criteria. Many who met only this criterion had relatively good outcomes, and, as a group, their outcomes were only slightly worse than for the entire series of psychotic patients. Those who met the full criteria had a significantly poorer outcome in terms of both continued psychopathology and social functioning.

• In order to estimate the concurrent validity of a structured psychiatric interview, we compared interview diagnoses obtained for 101 psychiatric inpatients to those recorded in the same patients' hospital charts. For most diagnoses considered, concordance was found to be high. For those in which concordance was low, we examined the reasons for the diagnostic discrepancy. Diagnostic errors that were judged to have occurred on the basis of the structured interview often seemed to have resulted from a lack of longitudinal clinical observation. However, more errors were judged to have occurred in the hospital charts, apparently because of physician oversight. We conclude that the concurrent validity of this structured interview is high and that such examinations might be useful not only for research but also for the routine initial evaluation of psychiatric patients.

• In a study of interrater diagnostic reliability, 101 psychiatric inpatients were independently interviewed by physicians using a structured interview. Newly admitted patients were randomly selected and examined by one of three psychiatrists. A second psychiatrist reexamined the same patient about 24 hours later. Interviews from the two examinations were evaluated independently, and diagnoses were made on the basis of objective criteria. The degree of diagnostic agreement for the two examinations was calculated using the kappa statistic. Agreement was found to be high as compared to other studies in the psychiatric literature, despite the fact that in most previous investigations diagnoses were not made independently. The results were also compared to studies of reliability of medical judgments. Possible reasons for the high interrater reliability are discussed and include the use of a structured interview and objective diagnostic criteria.

• This article reviews some methodological aspects of studies of diagnostic reliability in psychiatry. We define and discuss the concept of interrater reliability and review some of the ways in which the design of the reliability study can influence the results. Three basic methodological issues are raised, including: the importance of structured interviews and objective diagnostic criteria, the importance of a test/retest vs an interviewer/ observer design, and the calculation of reliability in a way that takes chance agreement into account.

IN REPLY. — It was good to read that Drs Ganguli and Saul have been using the DIS for new purposes—to study the recognition and management of depressive disorders in the medical care sector. Of course, we are also pleased that they consider it the best completely structured interview available.They cited the previous letter from Dr Endicott that questioned whether the tests we conducted pertained to validity as well as reliability. We believe we answered that in our response to Dr Endicott's letter, and they did not attempt to rebut our response. In brief, the clinical interview recommended by Dr Endicott as a validation tool would have to use the same diagnostic criteria (eg, DSM-III) as the DIS to serve as a reasonable yardstick by which to evaluate the DIS. At that time (and indeed still), there was no such standardized

TO THE EDITOR.— We would like to carry Dr Kraemer's "Peanuts" analogy one step further. Suppose the team statistician uses a method of rating players that is dependent on the number of times at bat. For players who have been on the team all year, the rating system will be a practical means of comparison, but it will strike out when applied to the brilliant player newly arrived from the farm team in midseason or batters who missed part of the season due to injury. Such has been the story with k. It has been used extensively as a measure in clinical studies, and as long as prevalences are consistently high, it is a consistent measure of the instrument reliability. But when it is extended to population studies, or used in clinical studies in which prevalences of specific diagnoses vary greatly, interpretation of k values must be made cautiously. Actually, Dr Kraemer claims

TO THE EDITOR.— In response (Archives 1982;39:1443-1445) to a letter by Ganguli and Saul (Archives 1982;39:1442-1443), Robins and Helzer noted that the Mini-Mental State Examination,1 incorporated into the Diagnostic Interview Schedule (DIS) to assess cognitive impairment, had been reported by Folstein et al to differentiate between pseudodementia and true organic brain syndromes. We would like to make that statement more specific.The Mini-Mental State Examination was designed as a clinical method for grading cognitive impairment. It produces a score that can be used to follow the course of patients or as a case detection technique after cutoff scores are established.In a clinical psychiatric setting, a low Mini-Mental score can be associated with many disorders including mental retardation, delirium, manicdepressive disorder, and schizophrenia, as illustrated by Folstein et al in their pre-DSM-III article in 1975.1,2 However, on a medical unit like the one from which Ganguli and

IN REPLY. — We certainly agree with Dr Fenton that the criterion of six months of illness adds considerably to the predictive power of the DSM-III definition, and presumably the Feighner definition, of schizophrenia. As we stated in the article that Dr Fenton's letter addresses: "It is clear that [the requirement of six months of illness] is the main predictor of the measures we have included under general outcome...." However, we hesitate to jump to the conclusion that "... for both definitions predictive validity derives only from the six-month duration requirement." The key may well be a combination of the psychotic phenomena specified by these two definitions and some duration of illness, which, as we suggested, may be less than six months. Fortunately, there is no need to jump to conclusions at all since this is an empirical question and a study currently under way by us may help to provide the answer.