The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

BACKGROUND

Evidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.

PURPOSE

To calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).

STUDY DESIGN

Meta-analysis; Level of evidence, 2.

METHODS

A search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.

RESULTS

There were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial's sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).

CONCLUSION

More than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

Introducing the fragility index-A case study using the Term Breech Trial

The fragility index (FI) is a sensitivity analysis of the statistically significant result of a clinical study. It is the number of hypothetical changes in the primary event of one of the two cohorts in a 1-to-1 comparative trial to render the statistically significant result non-significant (ie, to alter the P-value from ≤0.05 to >0.05). The FI can be compared with the patient drop-out rates and protocol violations, which, if much higher than the FI, may arguably suggest less robustness/stability of the trial's results. To illustrate the concept, we have chosen the Term Breech Trial (TBT) as a case study. The TBT results favor planned cesarean birth, as opposed to planned vaginal delivery, in the term singleton fetus with breech presentation. Our analysis shows that the FI of the TBT is 21, which is small in comparison to the number (hundreds) of protocol violations present. Some experts have suggested the inclusion of the FI in data analysis and subsequent discussion of clinical trial data. Routine use of such a metric may be valuable in encouraging readers to maintain a healthy degree of skepticism, especially when interpreting trial results which may directly influence clinical practice.

Empirical assessment of fragility index based on a large database of clinical studies in the Cochrane Library

RATIONALE AIMS AND OBJECTIVES

The fragility index (FI) and fragility quotient (FQ) are increasingly used measures for assessing the robustness of clinical studies with binary outcomes in terms of statistical significance. The FI is the minimum number of event status modifications that can alter a study result's statistical significance (or nonsignificance), and the FQ is calculated as the FI divided by the study's total sample size. The literature has no widely recognized criteria for interpreting the fragility measures' magnitudes. This article aims to provide an empirical assessment for the FI and FQ based on a large database of clinical studies in the Cochrane Library.

METHODS

We explored the overall empirical distributions of the FI and FQ based on five common methods (Fisher's exact test, χ2 test, risk difference, odds ratio, and relative risk) for determining statistical significance of binary outcomes in clinical research. We also considered three different scenarios for the FI calculation and evaluated the relationship between p values and FIs or FQs using Spearman'sρ $\rho $ . Finally, we summarized empirical thresholds based on the overall distributions of the FI and FQ to facilitate their interpretations in future research.

RESULTS

For about 20% of studies with significant results, the statistical significance was changed after modifying the event status of only one participant. Studies with significant results were considered slightly fragile if the significance hinged on the statuses of about five events. Studies were extremely fragile if FI≤ $\le $ 1 or FQ≤ $\le $ 0.01. The FIs were strongly correlated with p values for significant studies, while Spearman'sρ $\rho $ varied according to the total sample sizes of studies.

CONCLUSIONS

The statistical significance of clinical studies could be changed after modifying a few events' statuses. Many studies' findings are fairly fragile. The distributions of the FI and FQ provide insights for appraising the robustness of evidence in clinical decision-making.

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.

The Statistical Fragility of Platelet-Rich Plasma as Treatment for Plantar Fasciitis: A Systematic Review and Simulated Fragility Analysis

Background

Plantar fasciitis (PF) is the most common cause of heel pain and can be a source of extensive physical disability and financial burden. Platelet-rich plasma (PRP) offers a potentially definitive, regenerative treatment modality that, if effective, could change the current paradigm of PF care. However, randomized controlled trials (RCTs) on the clinical benefits of PRP for refractory PF offer inconsistent conclusions, potentially because of the broader limitations of using P value thresholds to declare statistical and clinical significance. In this study, we use the Continuous Fragility Index (CFI) and Quotient (CFQ) to appraise the statistical robustness of data from RCTs evaluating PRP for treatment of PF.

Methods

RCTs comparing outcomes after PRP injection vs alternative treatment in patients with chronic PF were evaluated. Representative simulated data sets were generated for each reported outcome event using summary statistics. The CFI was determined by manipulating each data set until reversal of significance (α=0.05) was achieved. The corresponding CFQ was calculated by dividing the CFI by the sample size.

Results

Of 259 studies screened, 20 studies (59 outcome events) were included in this analysis. From these simulations, the median CFI for all events was 9, suggesting that varying the treatment of 9 patients would be required to reverse trial significance. The corresponding CFQ was 0.177. Studies with reported P value <.05 were more statistically fragile (CFI=10, CFQ=0.122) than studies with reported P value >.05 (CFI=5, CFQ=0.179). Of 36 outcome events reporting lost to follow-up data, 10 events (27.8%) lost ≥9 patients.

Conclusion

Our findings suggest that, on average, the statistical fragility of RCTs evaluating PRP for nonoperative PF therapy is at least comparable to that of the sports medicine literature. However, several included studies had concerningly low simulated fragility scores. Orthopaedic surgeons may benefit from preferentially relying on studies with higher CFI and CFQ values when evaluating the utility of PRP for chronic PF in their own clinical practice. Given the importance of RCT data in clinical decision making, fragility indices could help give context to the stability of statistical findings.

Level of Evidence

Level I, systematic review.

The Statistical Fragility of Platelet-Rich Plasma as Treatment for Chronic Noninsertional Achilles Tendinopathy: A Systematic Review and Meta-analysis

Background:

Randomized controlled trial (RCT) outcomes reaching statistical significance, frequently determined by P <.05, are often used to guide decision making. Noted lack of reproducibility of some RCTs has brought special attention to the limitations of this approach. In this meta-analysis, we assessed the robustness of RCTs evaluating platelet-rich plasma (PRP) for the treatment of chronic noninsertional Achilles tendinopathy (AT) by using fragility indices.

Methods:

The present study was a systematic review and meta-analysis of RCTs comparing outcomes after PRP injection vs alternative treatment in patients with AT. Representative data sets were generated for each reported continuous outcome event using summary statistics. Fragility indices refer to the minimal number of patients whose status would have to change from a nonevent to an event to turn a statistically significant result into a nonsignificant result, or vice versa. The fragility index (FI) and continuous FI (CFI) were determined for dichotomous and continuous outcomes, respectively, by manipulating each data set until reversal of significance (a=0.05) was achieved. The corresponding fragility quotient (FQ) and continuous FQ (CFQ) were calculated by dividing FI/CFI by sample size.

Results:

Of 432 studies screened, 8 studies (52 outcome events) were included in this analysis. The 12 dichotomous outcomes had a median FI of 4.5 (FQ: 0.111), and the 40 continuous outcomes had a median CFI of 5 (CFQ: 0.154). All 52 outcome events included lost-to-follow-up data, and 12 (23.1%) indicated a greater number of patients lost to follow-up than the FI or CFI.

Conclusion:

Our findings suggest that RCTs evaluating PRP for AT therapy lack statistical robustness, because changing only a small number of events may alter outcome significance.

Level of Evidence:

Level II, therapeutic study.