Assessing the robustness of negative vascular surgery randomized controlled trials using their reverse fragility index

The reverse fragility index (RFI) describes the number of event conversions needed to convert a statistically non-significant dichotomous outcome to significant. The objective of this study was to assess the RFI of vascular surgery randomized controlled trials (RCTs) comparing endovascular vs. open surgery in the treatment of abdominal aortic aneurysms (AAA), carotid artery stenosis (CAS), and peripheral artery disease (PAD).MEDLINE and Embase were searched for RCTs investigating AAA, CAS, or PAD with statistically non-significant binary primary outcomes. The primary outcome of this study was the median RFI. Calculation of the RFI was done by creating two-by-two contingency tables and subtracting events from the group with fewer events while adding non-events to the same group until a two-tailed Fisher exact test produced a statistically significant result (p≤0.05).Of 4187 articles, 49 studies reporting 103 different primary endpoints were included. The overall median RFI was 7 (IQR 5, 13). The specific RFIs for AAA, CAS, and PAD were 10 (6-15.5), 6 (5-9.5), and 7 (5.5 -10) respectively. 42 (47%) endpoints had a loss to follow-up greater than RFI, of which 10 (24%) were AAA trials, 23 (55%) CAS trials, and 9 (21%) PAD trials. Pearson correlation demonstrated a significant positive relationship between a study's RFI to the impact factor of its publishing journal (r=0.38 [95% CI 0.20 to 0.54], p0.01), length of follow-up (r=0.43 [95% CI: 0.26 to 0.58], p0.01), and sample size (r = 0.28, 95% CI: 0.09 to 0.45, p0.01).A small number of events (median 7) were required to change the outcome of negative RCTs from statistically non-significant to significant with 47% of studies missing data that could have reversed the finding of its primary outcome. Reporting of the RFI relative to the loss-to-follow-up may be of benefit in future trials and provide confidence towards the robustness of P-value.