The Reverse Fragility Index: Interpreting the Current Literature on Long-Term Survivorship of Computer-Navigated Versus Conventional TKA: A Systematic Review and Cross-Sectional Study of Randomized Controlled Trials

The fragility index and reverse fragility index of FDA investigational device exemption trials in spinal fusion surgery: a systematic review

PURPOSE

FDA investigational device exemption (IDE) studies are considered a gold standard of assessing safety and efficacy of novel devices through RCTs. The fragility index (FI) has emerged as a means to assess robustness of statistically significant study results and inversely, the reverse fragility index (RFI) for non-significant differences. Previous authors have defined results as fragile if loss to follow up is greater than the FI or RFI. The aim of this study was to assess the FI, RFI, and robustness of data supplied by IDE studies in spinal surgery.

METHODS

This was a systematic review of the literature. Inclusion criteria included randomized controlled trials with dichotomous outcome measures conducted under IDE guidelines between 2000 and 2023. FI and RFI were calculated through successively changing events to non-events until the outcome changed to non-significance or significance, respectively. The fragility quotient (FQ) and reverse fragility quotient (RFQ) were calculated by dividing the FI and RFI, respectively, by the sample size.

RESULTS

Thirty-two studies met inclusion criteria with a total of 40 unique outcome measures; 240 outcomes were analyzed. Twenty-six studies reported 96 statistically significant results. The median FI was 6 (IQR: 3-9.25), and patients lost to follow up was greater than the FI in 99.0% (95/96) of results. The average FQ was 0.027. Thirty studies reported 144 statistically insignificant results and a median RFI of 6 (IQR: 4-8). The average RFQ extrapolated was 0.021, and loss to follow up was greater than the RFI in 98.6% (142/144) of results.

CONCLUSIONS

IDE studies in spine surgery are surprisingly fragile given their reputations, large sample sizes, and intent to establish safety in investigational devices. This study found a median FI and RFI of 6. The number of patients lost to follow-up was greater than FIand RFI in 98.8% (237/240) of reported outcomes. FQ and RFQ tell us that changes of two to three patients per hundred can flip the significance of reported outcomes. This is an important reminder of the limitations of RCTs. Analysis of fragility in future studies may help clarify the strength of the relationship between reported data and their conclusions.

Interpreting the Current Literature on Outcomes of Robotic-Assisted Versus Conventional Total Knee Arthroplasty Using Fragility Analysis: A Systematic Review and Cross-Sectional Study of Randomized Controlled Trials

BACKGROUND

Fragility analysis is a method of further characterizing outcomes in terms of the stability of statistical findings. This study assesses the statistical fragility of recent randomized controlled trials (RCTs) evaluating robotic-assisted versus conventional total knee arthroplasty (RA-TKA versus C-TKA).

METHODS

We queried PubMed for RCTs comparing alignment, function, and outcomes between RA-TKA and C-TKA. Fragility index (FI) and reverse fragility index (RFI) (collectively, "FI") were calculated for dichotomous outcomes as the number of outcome reversals needed to change statistical significance. Fragility quotient (FQ) was calculated by dividing the FI by the sample size for that outcome event. Median FI and FQ were calculated for all outcomes collectively as well as for each individual outcome. Subanalyses were performed to assess FI and FQ based on outcome event type and statistical significance, as well as study loss to follow-up and year of publication.

RESULTS

The overall median FI was 3.0 (interquartile range, [IQR] 1.0 to 6.3) and the median reverse fragility index was 3.0 (IQR 2.0 to 4.0). The overall median FQ was 0.027 (IQR 0.012 to 0.050). Loss to follow-up was greater than FI for 23 of the 38 outcomes assessed.

CONCLUSIONS

A small number of alternative outcomes is often enough to reverse the statistical significance of findings in RCTs evaluating dichotomous outcomes in RA-TKA versus C-TKA. We recommend reporting FI and FQ alongside P values to improve the interpretability of RCT results.

Reverse Fragility Index: Comparing Revision Rates Between Direct Anterior and Other Approaches in Total Hip Arthroplasty. A Systematic Review of Randomized Controlled Trials

BACKGROUND

Despite increasing adoption of the direct anterior (DA) approach in total hip arthroplasty (THA), uncertainty persists regarding its outcomes beyond the 1-year mark in comparison to other approaches. We used the reverse fragility index (RFI) to evaluate the robustness of reported findings in the literature.

METHODS

We conducted a systematic review of randomized controlled trials (RCTs) comparing implant revision rates between DA and other approaches in THA, defined as all those different from DA. Our primary outcome was the RFI, which gauges the number of events needed for a nonsignificant result to become significant, in the revision rate between DA and other approaches. We also calculated the reverse fragility quotient by dividing the RFI by each study's sample size. Median values and interquartile ranges (IQRs) were displayed.

RESULTS

A total of 10 RCTs with a total of 971 patients were included. The median RFI was 5 (IQR, 4 to 5), indicating the study's results would be statistically significant if the outcomes of 5 patients in 1 treatment arm were reversed. The median reverse fragility quotient was 0.049 (IQR, 0.04 to 0.057), indicating that a change of outcome in 4.9% of patients would render the revision rate significant. The median number of patients lost to follow-up was 4 (IQR, 0 to 7). Of the 10 RCTs, 6 had more patients lost to follow-up than their respective RFI values.

CONCLUSIONS

Notable fragility was evidenced in most studies comparing DA to other approaches for THA. Surgeons should not solely rely on the P value to determine clinical significance and instead use multiple metrics.

LEVEL OF EVIDENCE

II.

The Fragility of Statistical Findings in the Femoral Neck Fracture Literature: A Systematic Review of Randomized Controlled Trials

OBJECTIVES

Randomized controlled trials (RCTs) in the femoral neck fracture literature frequently report P -values for outcomes, which have substantial implications in guiding surgical management. This study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to assess the statistical stability of outcomes reported in RCTs evaluating the management and treatment of femoral neck fractures.

METHODS

DATA SOURCES

DESIGN

PubMed, Embase, and MEDLINE were queried for RCTs (January 1, 2010 to February 28, 2023).

SETTING

RCTs that evaluated surgical management or treatment of femoral neck fractures were included.

STUDY SELECTION CRITERIA

RCTs with 2 treatment arms reporting categorical dichotomous outcomes were included. Non-RCT studies, RCTs with greater than 2 treatment arms, and RCTs without a femoral neck fracture cohort were excluded.

DATA EXTRACTION AND SYNTHESIS

OUTCOME MEASURES AND COMPARISONS

The FI and rFI were calculated as the number of outcome event reversals required to alter statistical significance for significant ( P < 0.05) and nonsignificant ( P ≥ 0.05) outcomes, respectively. The FQ was calculated by dividing the FI by the sample size for the study.

RESULTS

Nine hundred eighty-five articles were screened, with 71 studies included for analysis. The median FI across a total of 197 outcomes was 4 [interquartile range (IQR) 2-5] with an associated FQ of 0.033 (IQR 0.017-0.060). Forty-seven outcomes were statistically significant with a median FI of 2 (IQR 1-4) and associated FQ of 0.02 (IQR 0.014-0.043). One hundred fifty outcomes were statistically nonsignificant with a median rFI of 4 (IQR 3-5) and associated FQ of 0.037 (IQR 0.019-0.065).

CONCLUSIONS

Statistical findings in femoral neck fracture RCTs are fragile, with reversal of a median 4 outcomes altering significance of study findings. The authors thus recommend standardized reporting of P -values with FI and FQ metrics to aid in interpreting the robustness of outcomes in femoral neck fracture RCTs.

LEVEL OF EVIDENCE

Therapeutic Level III. See Instructions for Authors for a complete description of levels of evidence.

The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

PURPOSE

To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature.

METHODS

All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF.

RESULTS

Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal.

CONCLUSIONS

The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT.

CLINICAL RELEVANCE

Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.

Statistical fragility of outcomes in acellular dermal matrix literature: A systematic review of randomized controlled trials

BACKGROUND

Acellular dermal matrix (ADM) is commonly used in plastic and reconstructive surgery. With the abundance of randomized controlled trials (RCTs) reporting P-values for ADM outcomes, this study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to evaluate the statistical stability of the outcomes in ADM RCTs.

METHODS

PubMed, Embase, SCOPUS, Medline, and Cochrane databases were reviewed for ADM RCTs (2003-present) reporting a dichotomous, categorical outcome. FI and rFI (event reversals influencing outcome significance) and FQ (standardized fragility) were calculated and reported as median. Subgroup analysis was performed based on intervention types.

RESULTS

Among the 127 studies screened, 56 RCTs with 579 outcomes were included. The median FI stood at 4 (3-5) and FQ was 0.04 (0.03-0.07). Only 101 outcomes were statistically significant with a median FI of 3 (1-6) and FQ of 0.04 (0.02-0.08). The nonsignificant outcomes had a median FI of 4 (3-5) and FQ of 0.04 (0.03-0.07). Notably, 26% of the outcomes had several patients lost to follow up equal to or surpassing the FI. Based on the intervention type, the median FIs showed minor fluctuations but remained low.

CONCLUSIONS

Outcomes from ADM-related RCTs were statistically fragile. Slight outcome reversals or maintenance of patient follow-up can alter the significance of results. Therefore, future researchers are recommended to jointly report FI, FQ, and P-values to offer a comprehensive view of the robustness in ADM literature.

Reverse Fragility Index Comparing Rates of Rerupture After Open Achilles Tendon Repair Versus Early Functional Rehabilitation: A Systematic Review of Randomized Controlled Trials

BACKGROUND

Despite similar published rates of rerupture among patients treated with early functional rehabilitation and open repair for acute Achilles tendon rupture, uncertainty still exists regarding the optimal treatment modality. The reverse fragility index (RFI) is a statistical tool that provides an objective measure of the study's neutrality by determining the number of events that need to change for a nonsignificant result to be significant.

PURPOSE

The purpose was to utilize the RFI to appraise the strength of neutrality of randomized controlled trials (RCTs) comparing the rerupture rates of acute Achilles tendon ruptures treated with open repair versus early functional rehabilitation.

STUDY DESIGN

Systematic review; Level of evidence, 1.

METHODS

A systematic review was performed including all RCTs comparing the rerupture rates after operative repair and early functional rehabilitation for acute Achilles tendon ruptures. Studies were included that explicitly used early functional rehabilitation, defined as weightbearing and exercise-based interventions initiated within 2 weeks, as compared with open repair and reported a nonsignificant difference in rerupture rates. The RFI, with rerupture as the primary outcome, was calculated for each study (significance threshold, P < .05). The RFI quantifies a study's strength of neutrality and is defined as the minimum number of event reversals necessary to change a nonsignificant result to statistically significant.

RESULTS

Nine RCTs were included, with 713 patients and 46 reruptures. The median (interquartile range) rerupture rate was 7.69% (6.38%-9.64%) overall, 4.00% (2.33%-7.14%) in the operative group, and 10.00% (5.26%-12.20%) in the nonoperative group. The median RFI was 3, indicating that an outcome reversal of 3 patients was necessary to change the results from nonsignificant to statistically significant. The median number of patients lost to follow-up was 6 (3-7). Of 9 studies, 7 (77.8%) had a loss to follow-up greater than or equal to its RFI.

CONCLUSION

The statistical nonsignificance of studies reporting equivalent rerupture rates in the management of acute Achilles tendon ruptures with open repair versus nonoperative management with early functional rehabilitation can be reversed by changing the outcome status of only a few patients.

The Reverse Fragility Index for Mortality End Points in Randomized Trials Comparing Uncemented and Cemented Hemiarthroplasty for Intracapsular Hip Fractures

BACKGROUND

Interpreting clinical relevance of randomized clinical trials (RCTs) is challenging when P-values are marginally above or below the P = .05 threshold. This study examined the robustness of statistically insignificant mortality events from RCTs comparing hemiarthroplasty femoral fixation for displaced intracapsular hip fractures through the reverse fragility index (RFI).

METHODS

RCTs were identified using Pubmed, OVID/Medline, and Cochrane databases. Mortality endpoints were stratified into 3 categories: (1) within 30-days, (2) within 90-days, and (3) at latest follow-up. The RFI was derived by manipulating reported mortality events utilizing a contingency table while maintaining a constant number of participants. The reverse fragility quotient (RFQ) was quantified by dividing the RFI by the study sample.

RESULTS

Eight RCTs (2,494 participants) were included. The median RFI and RFQ within 30-days was 3.0 (interquartile range [IQR]: 3.0 to 6.0) and 0.016 (IQR: 0.015 to 0.021), suggesting nonsignificant findings were contingent on 1.6 mortality events/100 participants. The median RFI and RFQ within 90-days was 6.0 (IQR: 4.0 to 7.0) and 0.028 (IQR: 0.024 to 0.038), suggesting nonsignificant findings were contingent on 2.8 mortality events/100 participants. At latest follow-up, the median RFI and RFQ was 7.0 (IQR: 6.0 to 12.0) and 0.038 (IQR: 0.029 to 0.054), suggesting nonsignificant findings were contingent on only 3.8 mortality events/100 participants. Median loss to follow-up was 16.0 (IQR: 11.0 to 58.0; 228% greater than RFI), and exceeded the RFI in 6/7(85.7%) studies.

CONCLUSIONS

A small number of events (median of 7) was required to convert a statistically nonsignificant finding to one that is significant for the endpoint of mortality. The median loss to follow-up exceeded the median RFI by greater than 200%, suggesting methodological limitations such as patient allocation could alter conclusions.

Fragility of randomised controlled trials for systemic lupus erythematosus and lupus nephritis therapies

OBJECTIVE

We aimed to evaluate the robustness of phase III randomised controlled trials (RCTs) for SLE and lupus nephritis (LN) using the fragility index (FI), the reverse FI (RFI) and the fragility quotient (FQ).

METHODS

We searched for phase III RCTs that included patients with active SLE or LN. Data on primary endpoints, total participants and the number of events for each arm were obtained. We calculated the FI score for RCTs with statistically significant results (number of patients required to change from event to non-event to make the study lose statistical significance), the RFI for RCTs without statistically significant results (number of patients required to change from non-event to event to make study gain statistical significance) and the FQ score for both (FI or RFI score divided by the sample size).

RESULTS

We evaluated 20 RCTs (16 SLE, four LN). The mean FI/RFI score was 13.6 (SD 6.6). There were nine RCTs with statistically significant results (seven SLE, two LN), and the mean FI score was 10.2 (SD 6.2). The lowest FI was for the ILLUMINATE-2 trial (FI=2), and the highest FI was for the BLISS-52 trial (FI=17).Twelve studies had non-statistically significant results (10 SLE, two LN) with a mean RFI score of 15.6 (SD 6.1). The lowest RFI was for the ILLUMINATE-1 trial (RFI=4), and the highest RFI was for the TULIP-1 trial (RFI=27). The lowest FQ scores were found in the ILLUMINATE trials and the highest in the Rituximab trials (EXPLORER and LUNAR), meaning that the last ones were the most robust results after accounting for sample size.

CONCLUSIONS

The evidence of therapies for patients with SLE and LN is derived mostly from fragile RCTs. Clinicians and trialists must be aware of the fragility of these RCTs for clinical decision-making and designing trials for novel therapeutics.