Is quality control of Cochrane reviews in controversial areas sufficient?

Objectives: Some conclusions from the Cochrane Database of Systematic Reviews (CDSR) seem volatile in areas of controversy and have recently changed conclusions. With this perspective in mind we decided to test the validity and sensitivity of negative conclusions in a systematic review of low-level laser therapy (LLLT) for osteoarthritis (OA) from the Cochrane Library was valid and robust.
Settings/location: None as this was a systematic review of literature.
Design: Validity was tested against a 9-item checklist for systematic reviews. Review selections were analyzed for possible discrepancies between trial and review reports and omissions of relevant trials and data. Data from discrepancies and omissions were then entered into sensitivity and subgroup analyses.
Outcome Measures: Continuous and dichotomous data for pain
Subjects: Patients with osteoarthritis.
Interventions: LLLT.
Results: Only clinicians who had performed LLLT trials, and had negative results were invited into the review group. The review was oblivious to findings published after 1993 about physiologic mechanisms and dose response patterns for LLLT. We found 18 questionable selections that favored a negative review conclusion in 17 of 18 cases. These were largely omissions of relevant positive data and selective inclusion of negative data from trials with small, ineffective doses. When existing and omitted data from relevant trials were combined, results changed from negative to significantly positive for continuous and categorical data. Subgrouping trials by location and recommended doses taken from current guidelines, revealed a highly significant effect of LLLT for treating knee OA.
Conclusions: The results of this CDSR was not robust and seems to be colored by questionable selections or omissions. For alternative pain therapies, it should be considered if lack of expertise on therapy mechanisms, or conflicts of interests with competing pain drug manufacturers, may bias conclusions. Review groups should recruit a balanced mix of current views and expertise and expand the use of sensitivity analyses to improve quality of CDSRs in areas of controversy.