PEDro's bias: summary quality scores should not be used in meta-analysis

Systematic reviews of randomized controlled trials and statistical combination of results from different trials in meta-analysis have risen to such prominence that few question their usefulness in evidence-based clinical practice. Although systematic reviews are indeed generally useful, the results of meta-analyses must be cautiously interpreted [1]. If the estimates of the trials included in the meta-analysis deviate from the truth in a systematic fashion, the summary estimate from the meta-analysis will likely be biased as well. There is strong empirical evidence showing that the meta-analyses of clinical trials in which the allocation of patients to treatment groups was not concealed, or in which the assessment of outcomes was not blinded, overestimate the treatment effects [2e5]. It is now widely accepted that the quality of a trial should be assessed before including it in a meta-analysis. Likewise, it is a good practice to ascertain if the results differ between trials at greater or lesser risk of bias. As yet, however, there is no consensus as to how this assessment should be done. In this commentary, we discuss the scoring of trials using quality scales, using the example of a scale widely used in trials of physiotherapy. We contrast the use of summary scores with an alternative approach, which is based on an assessment of individual components such as concealment of allocation and blinding. Quality scales assess several criteria related to the design, conduct, and analysis of trials, and each earns points that are aggregated into an overall score. The score determines the classification of the study as one of the higher or lower methodological quality, with the implication that bias has been prevented to a greater or lesser degree. Chalmers et al. [6] in 1981 were among the first to develop such a scale, with possible summary scores ranging from 0 to 44. Since then, many other scales were developed