Search

Earlier work showed how to perform fixed-effects meta-analysis of studies or trials when each provides results on more than one outcome per patient and these multiple outcomes are correlated. That fixed-effects generalized-least-squares approach analyzes the multiple outcomes jointly within a single model, and it can include covariates, such as duration of therapy or quality of trial, that may explain observed heterogeneity of results among the trials. Sometimes the covariates explain all the heterogeneity, and the fixed-effects regression model is appropriate. However, unexplained heterogeneity may often remain, even after taking into account known or suspected covariates. Because fixed-effects models do not make allowance for this remaining unexplained heterogeneity, the potential exists for bias in estimated coefficients, standard errors and p-values. We propose two random-effects approaches for the regression meta-analysis of multiple correlated outcomes. We compare their use with fixed-effects models and with separate-outcomes models in a meta-analysis of periodontal clinical trials. A simulation study shows the advantages of the random-effects approach. These methods also facilitate meta-analysis of trials that compare more than two treatments.

Until recently clinical decision-making relied on intuition, unsystematic experiences and pathophysiological rationale. Developments in the understanding of the nature of clinical trials, systematic reviews and techniques to evaluate diagnostic technology over the past 30 years made the evidence-based clinical practice possible. During this period of evolution it became apparent that not all evidence is equal. Studies that use weak designs tend to estimate a greater treatment effect, whereas strong designs yield a more conservative treatment effect. A good number of orthodontic problems lend themselves to investigations of controlled clinical trials. Reports of early treatment effectiveness of malocclusions contained in this issue demonstrate the point.

In recent years, methods have been developed to evaluate effectiveness, cost-effectiveness, and quality of oral health services and delivery mechanisms. These evaluation techniques are important to the application of oral health services, because most oral health services have not been adequately evaluated. For many services, evaluation may not be necessary. However, there is wide variation in clinical practices and accumulating evidence that many health practices, although based on reasonable pathophysiologic grounds, are not resulting in their expected health benefits. This has led to the suggestion that all clinical precepts may be questioned. A number of measures are used by researchers to evaluate the quality and effectiveness of oral health services. These include: Effectiveness and appropriateness: Methods available to assess health practices range from clinical observation to strictly controlled randomized clinical trials. Cost-effectiveness: Benefit-cost and cost effectiveness analyses. Quality: No tool exists that will single-handedly measure quality. Current methodologies should be based on the following three-dimensional conceptual framework: Structure (evaluation of facilities, equipment, personnel, and organization to deliver care), process (observance of the patient-provider interaction), and outcome (measures of health status and patient outcomes). Little to no information on the quality and effectiveness of oral health delivery systems is available in the United States; the most common system in place is solo, private, fee-for-service practice. Specific questions, conditions, or practices that need evaluation include: Alternative frequencies of oral prophylaxis; Use of fluoride and other remineralization techniques for early decay; Alternative methods to replace missing teeth, including the appropriateness of dental implants; Methods to prevent and treat symptoms of dry mouth; Treatment of the medically compromised patient; Prevention of oral complications of cancer treatment; Appropriateness of new and existing diagnostic technologies; Diagnosis and treatment of oral cancer; and Alternative oral health delivery systems, financing, and expanded responsibilities for allied health professionals.

The results of periodontal therapy vary by disease severity, outcome measure, and method of data analysis. Several clinical trials and a subsequent meta-analysis have demonstrated that, for teeth with severe disease, surgery decreases probing depth (PD) and increases attachment level (AL) more than non-surgical treatment. For other disease levels, the choice of therapy depends on the outcome measure. When clinical trials use two or more outcome measures (such as PD and AL), investigators ordinarily analyze each outcome separately. When the correlations are incorporated among the outcomes, a meta-analysis can use generalized-least-squares (GLS) regression to analyze multiple outcomes jointly. We applied the GLS multiple-outcomes model in a meta-analysis of 5 trials comparing surgical and non-surgical periodontal treatments, each assessing the outcomes PD and AL one year after treatment. The clinical conclusions are similar to those reported earlier, but our estimates of the relative benefits of surgical and non-surgical treatment should be more accurate, because the GLS method takes into account correlation between AL and PD. When correlations between the two outcomes rise, as they do with increasing severity of disease, the GLS estimates depart from those derived from separate analyses of PD and AL.

The development of recommendations for what is acceptable clinical practice for patients with adult periodontitis is challenging. First, how much and what type of evidence is needed? This depends very much on the clinical problem, the nature of the evidence that is available, and the potential ill effects that may ensue if the effects of the treatment that is recommended are not what they are hoped to be. All treatments are applied with the implicit assumption that they do more good than harm. Treatments that are expensive, invasive, irreversible, or potentially risky, require more evidence than treatments that are not invasive, have reversible effects, and are without risk. Recall the hierarchy of types of evidence used in the AHCPR pain guidelines ranging from a meta-analysis of RCTs to uncontrolled observations in patients or expert opinion. A randomized controlled trial, or a meta-analysis of RCTs carries more weight than a much larger series of cases because of the great potential for bias when observations are made under uncontrolled conditions. One hundred poorly controlled studies do not provide evidence for optimal clinical decisions. Second, who bears the burden of proof? Certainly, any newly proposed therapy must be proven safe and effective by its proponents. But, what about treatments that have been the standard of practice, can they be grandfathered in, or must evidence be gathered for them? Given changes in understanding of mechanisms of disease pathogenesis, classification, rates of activity, and measurement of outcomes, it may be time to call all clinical dogma regarding treatment of adult periodontitis into question. Reliable evidence is not available for many of the treatments for adult periodontitis. Most of the evidence that is available was collected using methods and an underlying conceptual scheme regarding diagnosis and progression of disease that is very different from current beliefs. One could argue that even for the treatments that have been evaluated in RCTs, disease activity rates were so low by current assessment, that there was no disease to prevent or treat.

This decision-analysis model assesses ambulatory cardiac monitoring (ACM), ACM followed by exercise testing, and electrophysiologic studies (EPS) in the evaluation and selection of antiarrhythmic agents in postinfarct patients with malignant arrhythmias. With existing data, we find no consistent advantage for one method of drug testing over another, although ACM appears to require fewer resources than does EPS. More patients qualify for EPS, but this fact does not increase the proportion of patients for whom a drug can be identified. These methods may test different aspects of arrhythmia activity and drug response, and sequential use may provide additional benefits. Such benefits must be determined empirically.

This introduction and the three essays that follow it examine ambulatory cardiac monitoring for specific clinical indications. They also examine the ways in which evidence from the literature may be synthesized through the framework of decision analysis to guide its appropriate use and identify areas in which more research is needed. The essays discuss ambulatory cardiac monitoring for evaluation of syncope in the elderly; detection of silent ischemia after a myocardial infarction; and selection of antiarrhythmic drugs for malignant ventricular arrhythmias.

The rising costs of health care and interest in the evaluation of health services and systems have sparked an increased need for technology assessment. A variety of available methods of assessment are described in the Institute of Medicine's book Assessing Medical Technologies (1). Although such methods as the randomized controlled trial (RCT) are widely accepted and used, obtaining information by such methods often takes a considerable amount of time, expense, and sophistication in study design. These costs suggest that a broader range of methods for collecting information about health care technologies should be considered. A vast resource of data collected on patients, ranging from a provider's practice records to national data sets, might be useful for technology assessment if it could be properly appraised.

Decision analysis is used to compare the cost and disability of alternative treatment strategies for asymptomatic mandibular third molars. The analysis shows that extracting only those third molars that remain impacted and become pathologically involved is always associated with less expected cost and disability than prophylactic removal of asymptomatic wisdom teeth.

Earlier work showed how to perform fixed-effects meta-analysis of studies or trials when each provides results on more than one outcome per patient and these multiple outcomes are correlated. That fixed-effects generalized-least-squares approach analyzes the multiple outcomes jointly within a single model, and it can include covariates, such as duration of therapy or quality of trial, that may explain observed heterogeneity of results among the trials. Sometimes the covariates explain all the heterogeneity, and the fixed-effects regression model is appropriate. However, unexplained heterogeneity may often remain, even after taking into account known or suspected covariates. Because fixed-effects models do not make allowance for this remaining unexplained heterogeneity, the potential exists for bias in estimated coefficients, standard errors and p-values. We propose two random-effects approaches for the regression meta-analysis of multiple correlated outcomes. We compare their use with fixed-effects models and with separate-outcomes models in a meta-analysis of periodontal clinical trials. A simulation study shows the advantages of the random-effects approach. These methods also facilitate meta-analysis of trials that compare more than two treatments.

A systematic review was performed in response to a request by the National Institute of Dental Research to evaluate in broad terms the strength of evidence regarding therapy for temporomandibular disorders (TMD). This report describes the epidemiology of research for TMD in broad terms indicating the total number of citations, the proportion related to therapy, and the distribution according to study design and language or country of origin. Medline and hand searching of article bibliographies and of selected journals produced the set of citations evaluated. From 1980 to 1992, there were more than 4,000 references to TMD, of which about 1,200 regarded therapy. Forty-one percent of the 1,200 references were classified as reviews and only 15% were clinical studies. Less than 5% (n = 51) were randomized controlled trials. This review identified a vast amount of literature on TMD with articles published in several different languages, indicating a worldwide interest in this problem. Because assimilation of this literature cannot be expected of the average practitioner treating patients who have TMD, or of most researchers in this area, it is likely not being used to its maximum potential. The literature on therapy for TMD consists primarily of uncontrolled observations of patients such as uncontrolled clinical trials, case series, case reports, and simple descriptions of techniques. It is generally agreed that such uncontrolled observations, while contributing to knowledge about therapy of TMD, are subject to considerable bias and thus difficult to interpret. If treatment of TMD is going to follow the trend in medicine to base patient-care decisions on evidence rather than expert opinion or pathophysiologic rationales, then more rigorously controlled clinical trials of most therapies will be necessary.

Twenty years ago, in Effectiveness and Efficiency, Archie Cochrane emphasized the importance of randomized controlled trials (RCTs) in guiding decisions about health care (Cochrane, 1972). Randomized trials are not always required to assess the effects of health care (the good and bad effects of some forms of health care are obvious), and sometimes trials are not feasible. But, for many forms of care, trials involving sufficient numbers of participants are essential to distinguish reliably between the effects of care and the effects of biases or chance. Just as important as conducting the trials, though, is disseminating the results through systematic reviews of the findings. Such reviews depend on the difficult task of identifying all relevant trials, and several ef f orts are going on internationally to coordinate this work. If people are to benefit from the results of trials, all the steps between research and practice must be accomplished effectively. Trials must be properly designed, conducted, analyzed, and reported. Their results must be assembled in systematic, up-to-date, and accessible reviews. The results of these reviews must be taken into account by decisionmakers, and finally, based on these decisions, there must be effective systems to audit how well local or national guidelines for health care are followed. Currently, weaknesses exist at all these steps. Cochrane drew attention to a particular weakness, however, when he criticized the medical profession for not having organized a system for producing up-to-date reviews of the results of RCTs. Experience gained over the past decade provides a useful basis for developing such a system (Chalmers, 1991). In particular, it has become clear that the same scientific principles that are applied to the design and conduct of primary research must also be applied to the process of reviewing that research (Mulrow, 1987; Haynes, 1991). Impressive examples now exist of the power of systematic reviews to provide reliable answers to important questions-for example, the effects of treatment on early breast cancer (Early Breast Cancer Trialists' Collaborative Group, 1992). Recent studies have shown that if systematic reviews, updated periodically, had been started at the beginning of a series of related trials, reliable recommendations for treatment would have been made earlier (Lau et al., 1992). Unsystematically conducted reviews in journals and textbooks have sometimes taken more than a decade to recommend treatments that a systematic review of trials would have shown to prevent premature death; in addition, other treatments have been endorsed long after evidence from trials had suggested that they were useless or actually harmful (Antman et al., 1992). The usual, unsystematic approach to reviewing the effects of care also increases the probability that resources will be wasted. For example, a systematic review of RCTs a decade ago would have shown that a short course of corticosteroids given to mothers expected to give birth prematurely substantially reduces the risk of neonatal morbidity and death (Crowley et al., 1990). Repeated failure to conduct, and apply the results of, systematic reviews of these trials has not only resulted in the unnecessary suffering of tens of thousands of babies but has also meant that neonatal care has been more expensive than it need have been (Mugford et al., 1991). Similarly, research funding bodies and ethics committees should be concerned about the extent to which resources are wasted on unnecessary research-for example, in repeated demonstrations of the protective effects of prophylactic antibiotics for some forms of surgery (Baum et al., 1981).

A meta-analysis was performed on 5 randomized controlled trials comparing surgical with non-surgical treatment for periodontal disease. The specific procedures considered were the modified Widman flap compared with scaling and root planning or curettage with anesthesia. We chose the most consistently reported outcomes, pocket depth and attachment level, for analysis. At 1 year of follow-up, surgical treatment reduced pocket depth more than non-surgical for all initial levels of disease, but by 5 years, only the deepest initial pockets (> 7 mm) showed significant improvement over non-surgically treated teeth (0.51 mm reduction, p < 0.01). Attachment level showed significantly better early results for non-surgical treatment for less diseased teeth, but by 5 years, all significant differences had disappeared. We computed quality scores following a method described by Chalmers. The mean quality score for study data analysis and presentation was 0.37 +/- 0.009 and for the study protocol, the mean quality score was 0.19 +/- 0.002. We find that this meta-analysis supports findings relating response to therapy with initial level of disease severity. We also find that the choice of outcome measure influences the choice of therapy, with surgical therapy providing greater benefit for probing depth and non-surgical therapy providing greater benefit for attachment level. These results must be viewed, however, in light of the low quality scores of the evaluated studies and the potential for bias due to lack of binding, the small mean treatment differences, and the observer measurement variability.

This review focuses on evaluations of particular clinical problems or technologies in dentistry that have used the recently developed technology assessment techniques of decision analysis, meta-analysis, and cost-effectiveness analysis. It also discusses general methodologic and implementation issues in the assessment of health-care technologies.

The use of systemic tetracycline in the treatment of periodontal disease has been controversial. To investigate this controversy, we performed a quality assessment and attempted to perform a meta-analysis of 13 published studies. We evaluated the quality of the study protocol and data analysis and presentation for each study. We were unable to combine data from the majority of studies due to heterogeneity of the outcomes evaluated and limitations in data reported in the individual studies. Therefore, only 2 studies were included in the quantitative meta-analysis. On a scale of 0-1, the mean score for this group of studies was 0.27 (+/- 0.19) for study protocol and 0.31 (+/- 0.11) for data analysis and presentation. Mean reduction in probing depth for the group treated with tetracycline plus scaling was 2.45 mm; for the group which received only scaling, 2.02 mm; for the group that received only tetracycline, 1.98 mm; and for the control group, 0.65 mm. We conclude that analysis of data from the published literature does not demonstrate that the use of systemic tetracycline is more beneficial than conventional treatment in the management of adult periodontal disease. More information is needed in order to perform an extensive meta-analysis of this subject.

This review of publications based on data from the Veterans Affairs Normative Aging Study identified four articles that were considered assessments of health care technologies. Three evaluated methods for research on body composition, the fourth studied ventricular size in patients with presenile dementia. Several additional articles are discussed that provided data on baseline values or risk factors for disease that might be useful in generating hypotheses to be tested in later assessments of technologies.

Of the 52 studies emanating from the Veterans Affairs Dental Longitudinal Study, 6 were identified as technology assessments. Three of these studies evaluated dental radiographs used for the diagnosis of oral lesions. Two studies compared alternative technologies for the replacement of missing teeth, and one reported the effects of NSAIDs on alveolar bone loss. Four additional articles are discussed that provide data that will be useful in future assessments of dental technologies.

In this concluding article, we summarize the major ways that seven registries and data sets have been used for technology assessment in the published literature. The registries fell into three groups according to the original purpose of the registry. Within each group, we note the types of contributions that researchers using the registries made to technology assessment.

The presence of paired or multiple organs (arches, quadrants, teeth) and the chronic nature of many dental diseases suggest the use of split-mouth (trials in which each subject receives greater than or equal to 2 treatments, each to a separate section of the mouth) and cross-over research designs (trials in which each subject receives greater than or equal to 2 treatments in sequence). While these designs offer potential savings in resources, their usefulness can be negated if several strict scientific and statistical assumptions are not met. The primary prerequisites for the use of split-mouth and cross-over designs are that: (1) the disease to be investigated is relatively stable and uniformly distributed; (2) the effects of the treatments to be evaluated are short-lived or reversible for cross-over studies, or are localized for split-mouth designs. Other important factors that influence the appropriate use of these designs include: the method of treatment sequencing and assignment, and the cross-over rules used; blinding of patient assignment, patients and observers; assessment of order effects including period, carry-over or spill-over effects; the choice of statistical analysis, the sample size utilized, and the special importance of patients lost to study or of faulty data points. The objective of this study was to review 3 journals for studies using split-mouth or cross-over designs to determine how the assumptions underlying these research designs are considered and applied in dental research. The majority of studies used adequate methods for treatment allocation and sequencing; however, many studies failed to take advantage of the research designs in the statistical analysis of data. In addition, very few studies considered the possibility of order effects or reduced bias through blinding procedures.

Diagnostic or screening tests are used to help determine whether or not a patient has a certain condition or disease. The ability of a diagnostic test to correctly classify subjects is expressed by the four test characteristics-sensitivity, specificity, predictive value positive, and predictive value negative. This paper describes these characteristics and discusses methods for choosing optimal tests or cutoff points to maximize expected value considering the consequences of incorrect diagnoses. Data drawn from ongoing studies of facial pain are used to illustrate some of these concepts.

The management of impacted third molars has recieved considerable attention in recent years, because removal is both common, costly, and frequently associated with morbidity. While there is little research evidence to support or refute the prophylactic extraction of asymptomatic teeth, two decision analyses have both suggested that this practice does not maximize the benefit for patientst-", Decisions analysis, although both prescriptive and designed to maximize benefits, does not always reflect the way clinicians act. The differences between clini­ cians' subjective assessments about what is best to do and the recommendations of quantitative analyses need to be reconciled. A first step in this reconciliation is to determine what motivates clinicians' decisions. A recent paper in this journal reported one such approach". The investigators used the techniques of judgement analysis to identify factors influencing clinicians' decisions to recommend extraction of asymptomatic impacted mandibular third molars. In this study, the subjective opinions of general dental practitioners (GDPs) and oral surgeons were elicited to determine both the strengh of their beliefs about whether or not to extract, and the weight the risk of developing pathology played in their decisions. It is interesting to note the differences that were found between GDPs and oral surgeons in this study, to see how their estimates varied from data available in the research literature, and consider how this information might be used to change behaviour to improve clinical decision making. On average, GDPs and oral surgeons had only slighty different beliefs concerning the need for extrac­ tion, but dramatic differences in the relative import­ ance they placed on the different pathological sequelae motivating their decisions. General practitioners focused on the unlikely event of cyst development, while oral surgeons were most motivated by concern about the development of pericoronitis. The discussion in this paper includes some useful speculation about why these differences arise, including the frequency with which conditions are observed by different practi­ tioners, the importance they attach to the various outcomes, and the emphasis placed on these outcomes during professional education. Comparison of the importance these clinicians placed on the potential pathological sequelae with estimates of their occurrence from the published literature reveal large discrepancies. In no case could the weights assigned to the risk of developing the various patholo­ gies be explained by currently available prevalence data. The authors suggest that clinicians may be more influenced by the potential consequences of the various pathologies than by the likelihood of their occurrence. The demonstration of wide variations in clinical practice and the accumulating evidence that many health care practices (although based on reasonable pathophysiological grounds,) are not achieving their expected health benefits, has led to the suggestion that all clinical dogma may be called into question'':", Two trends in medicine are important to consider here.