Search

Background: Transparency and reproducibility are expected to be normative practices in clinical trials used for decision-making on marketing authorisations for new medicines. This registered report introduces a cross-sectional study aiming to assess inferential reproducibility for main trials assessed by the European Medicines Agency., Methods: Two researchers independently identified all studies on new medicines, biosimilars and orphan medicines given approval by the European Commission between January 2017 and December 2019, categorised as 'main studies' in the European Public Assessment Reports (EPARs). Sixty-two of these studies were randomly sampled. One researcher retrieved the individual patient data (IPD) for these studies and prepared a dossier for each study, containing the IPD, the protocol and information on the conduct of the study. A second researcher who had no access to study reports used the dossier to run an independent re-analysis of each trial. All results of these re-analyses were reported in terms of each study's conclusions, p-values, effect sizes and changes from the initial protocol. A team of two researchers not involved in the re-analysis compared results of the re-analyses with published results of the trial., Results: Two hundred ninety-two main studies in 173 EPARs were identified. Among the 62 studies randomly sampled, we received IPD for 10 trials. The median number of days between data request and data receipt was 253 [interquartile range 182-469]. For these ten trials, we identified 23 distinct primary outcomes for which the conclusions were reproduced in all re-analyses. Therefore, 10/62 trials (16% [95% confidence interval 8% to 28%]) were reproduced, as the 52 studies without available data were considered non-reproducible. There was no change from the original study protocol regarding the primary outcome in any of these ten studies. Spin was observed in the report of one study., Conclusions: Despite their results supporting decisions that affect millions of people's health across the European Union, most main studies used in EPARs lack transparency and their results are not reproducible for external researchers. Re-analyses of the few trials with available data showed very good inferential reproducibility., Trial Registration: https://osf.io/mcw3t/., (© 2022. The Author(s).)

Background: Different methodological choices such as inclusion/exclusion criteria and analytical models can yield different results and inferences when meta-analyses are performed. We explored the range of such differences, using several methodological choices for indirect comparison meta-analyses to compare nalmefene and naltrexone in the reduction of alcohol consumption as a case study., Methods: All double-blind randomized controlled trials (RCTs) comparing nalmefene to naltrexone or one of these compounds to a placebo in the treatment of alcohol dependence or alcohol use disorders were considered. Two reviewers searched for published and unpublished studies in MEDLINE (August 2017), the Cochrane Library, Embase, and ClinicalTrials.gov and contacted pharmaceutical companies, the European Medicines Agency, and the Food and Drug Administration. The indirect comparison meta-analyses were performed according to different inclusion/exclusion criteria (based on medical condition, abstinence of patients before inclusion, gender, somatic and psychiatric comorbidity, psychological support, treatment administered and dose, treatment duration, outcome reported, publication status, and risk of bias) and different analytical models (fixed and random effects). The primary outcome was the vibration of effects (VoE), i.e. the range of different results of the indirect comparison between nalmefene and naltrexone. The presence of a "Janus effect" was investigated, i.e. whether the 1st and 99th percentiles in the distribution of effect sizes were in opposite directions., Results: Nine nalmefene and 51 naltrexone RCTs were included. No study provided a direct comparison between the drugs. We performed 9216 meta-analyses for the indirect comparison with a median of 16 RCTs (interquartile range = 12-21) included in each meta-analysis. The standardized effect size was negative at the 1st percentile (- 0.29, favouring nalmefene) and positive at the 99th percentile (0.29, favouring naltrexone). A total of 7.1% (425/5961) of the meta-analyses with a negative effect size and 18.9% (616/3255) of those with a positive effect size were statistically significant (p < 0.05)., Conclusions: The choice of inclusion/exclusion criteria and analytical models for meta-analysis can result in entirely opposite results. VoE evaluations could be performed when overlapping meta-analyses on the same topic yield contradictory result., Trial Registration: This study was registered on October 19, 2016, in the Open Science Framework (OSF, protocol available at https://osf.io/7bq4y/ ).

Nalmefene was the first treatment approved by the European Medicines Agency for reducing alcohol consumption in adult patients with alcohol dependence. It is often presented as a paradigm shift in therapeutics, but major issues limit the interpretation of the evidence supporting its use. The randomised trials submitted provided no evidence of harm reduction, the differences on consumption outcomes were of questionable clinical relevance, the target population was defined a posteriori and the drug was compared to a placebo although naltrexone was already used off-label. No post-approval randomised study is currently designed to clearly address these issues. In addition, nalmefene trials have been uncritically cited, even in guidelines. This experience reveals weaknesses in drug evaluations in alcohol dependence, which call for changes. We propose to dispense with alcohol consumption as a surrogate outcome, to consider comparative effectiveness issues, and to recommend randomised post-approval studies in case of controversial approval.

In a meta-analysis published in BMC Medicine, we explored whether evidence-based medicine can actually be sure that 'sucrose = sucrose' in the treatment of depression. This paper, based upon a reductio ad absurdum, addressed an epistemological question using a 'scientific' approach, and could be disconcerting as suggested by Cipriani and Geddes' commentary. However, most papers are based upon a mixture of observations and discussions about sense and meaning. Ultimately, there is nothing more than a story, told with words or numbers. Randomised controlled trials provide information about average patients that do not exist. These results ignores an entire segment of therapeutics that plays a crucial role, namely care. This information is usually set out using a 'grammar' that is ambiguous, since statistical tests of hypothesis have raised epistemological questions that are not as yet solved. Moreover, many of these stories remain untold, and unpublished. For these reasons evidence-based medicine is a vehicle for many paradoxes and controversies. Reductio ad absurdum can be useful in precisely this case, to underline how and why the medical literature can sometimes give an impression of absurdity of this sort. Even if the data analysis in our paper was rather rhetorical, we agree that it should comply with the classic standards of reporting and we provide the important extra data that Cipriani and Geddes have requested.

Background: Antidepressants are often considered to be mere placebos despite the fact that meta-analyses are able to rank them. It follows that it should also be possible to rank different placebos, which are all made of sucrose. To explore this issue, which is rather more epistemological than clinical, we designed an unusual meta-analysis to investigate whether the effects of placebo in one situation are different from the effects of placebo in another situation., Methods: Published and unpublished studies were searched for by three reviewers on Medline, the Cochrane Library, Embase, clinicaltrials.gov, Current Controlled Trial, in bibliographies, and by mailing key organizations. The following studies in first-line treatment for major depressive disorder were considered to construct an "evidence network": 1) randomized controlled trials (RCTs) versus placebo on fluoxetine, venlafaxine and 2) fluoxetine versus venlafaxine head-to-head RCTs.Two network meta-analyses were run to indirectly compare response and remission rates among three different placebos: 1) fluoxetine placebo, 2) venlafaxine placebo, and 3) venlafaxine/fluoxetine placebo (that is, placebo compared to both venlafaxine and fluoxetine). Publication biases were assessed using funnel plots and statistically tested., Results: The three placebos were not significantly different in terms of response or remission. The antidepressant agents were significantly more efficacious than the placebos, and venlafaxine was more efficacious than fluoxetine. The funnel plots, however, showed a major publication bias., Conclusion: The presence of significant levels of publication bias indicates that we cannot even be certain of the conclusion that sucrose equals sucrose in trials of major depressive disorder. This result should remind clinicians to step back to take a more objective view when interpreting a scientific result. It is of crucial importance for their practice, far more so than ranking antidepressant efficacy.