Strength in numbers? The fragility index of studies from the Scandinavian knee ligament registries

The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

PURPOSE

To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature.

METHODS

All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF.

RESULTS

Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal.

CONCLUSIONS

The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT.

CLINICAL RELEVANCE

Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.

The reverse fragility index: RCTs reporting non-significant differences in failure rates between hamstring and bone-patellar tendon-bone autografts have fragile results

PURPOSE

The reverse fragility index (RFI) is a novel metric to appraise the results of studies reporting statistically non-significant results. The purpose of this study was to determine the statistical robustness of randomized controlled trials (RCTs) reporting non-significant differences in anterior cruciate ligament reconstruction (ACLR) graft failure rates, defined as re-rupture/revision ACLR rate, between hamstring tendon (HT) and bone-patellar tendon-bone (BTB) autografts by calculating RFIs.

METHODS

A systematic review was performed to identify RCTs that compared HT to BTB grafts for ACLR through January 2022. Studies reporting non-significant differences in graft re-rupture and revision ACLR rate (n.s.) were included. The RFI, defined as the fewest number of event reversals needed to change the non-significant graft re-rupture/revision outcome to statistically significant (P < 0.05), was recorded for each study. In addition, the number of studies in which the loss to follow-up exceeded the RFI was recorded.

RESULTS

Among the 16 included RCTs, the median (interquartile range [IQR]) sample size was 71 (64-114), and the median (IQR) total number of graft re-rupture/revision ACLR events was 4 (4-6). The median (IQR) graft re-rupture/revision ACLR rate was 4.3% (3.0-6.4) overall, 4.1% (2.6-6.7) in the BTB group, and 5.4% (3.0-6.3) in the HT group. The median (IQR) RFI was 3 (3-4), signifying that a reversal of the outcome in 3 patients in one arm was needed to flip the studies' result from non-significant to statistically significant (P < 0.05). The median (IQR) number of participants lost to follow-up was 11 (3-13), and 13 (81.3%) of the included studies had a loss to follow-up greater than the studies' RFI.

CONCLUSION

The results of RCTs reporting statistically non-significant re-rupture/revision ACLR rates between HT and BTB autografts would become significant if the outcome were reversed in a small number of patients-a number that was less than the loss to follow-up in the majority of studies. Thus, the neutrality of these studies is fragile, and a true statistically significant difference in re-rupture/revision rates may have been undetected.

LEVEL OF EVIDENCE

Level I.

The Statistical Fragility of Foot and Ankle Surgery Randomized Controlled Trials

Fragility index (FI) is a metric used to interpret the results of randomized controlled trials (RCTs), and describes the number of subjects that would need to be switched from event to non-event for a result to no longer be significant. Studies that analyze FI of RCTs in various orthopedic subspecialties have shown the RCTs to be largely underpowered and highly fragile. However, FI has not been assessed in foot and ankle RCTs. The MEDLINE and Embase online databases were searched from 1/1/2011 through 11/19/2021 for RCTs involving foot and ankle conditions. FI, fragility quotient (FQ), and difference between the FI and number of subjects lost to follow-up was calculated. Spearman correlation was performed to determine the relationship between sample size and FI. Overall, 1262 studies were identified of which 18 were included in the final analysis. The median sample size was 65 (interquartile range [IQR] 57-95.5), the median FI was 2 (IQR 1-2.5), and the median FQ was 0.026 (IQR 0.012-0.033). Ten of 15 (67%) studies with non-zero FI values had FI values less than the number of subjects lost to follow-up. There was linear association between FI and sample size (R² = 0.495, p-value: .031). This study demonstrates that RCTs in the field of foot and ankle surgery are highly fragile, similar to other orthopedic subspecialties.

Risk Factors for Anterior Cruciate Ligament Graft Failure in Professional Athletes: An Analysis of 342 Patients With a Mean Follow-up of 100 Months From the SANTI Study Group

BACKGROUND

Anterior cruciate ligament (ACL) injuries are among the most common knee injuries sustained in elite sport, and athletes generally undergo ACL reconstruction (ACLR) to facilitate their return to sport. ACL graft rupture is a career-threatening event for elite athletes.

PURPOSE/HYPOTHESIS

The purpose of this study was to determine the risk factors for graft failure in professional athletes undergoing ACLR. It was hypothesized that athletes who underwent combined ACLR with a lateral extra-articular procedure (LEAP) would experience significantly lower rates of graft rupture in comparison with those who underwent isolated ACLR.

STUDY DESIGN

Cohort study; Level of evidence, 3.

METHODS

Professional athletes who underwent primary ACLR with a minimum follow-up of 2 years were identified from the Santy database. Patients were excluded if they underwent major concomitant procedures, including multiligament reconstruction surgery or osteotomy. Further ipsilateral knee injury, contralateral knee injury, and any other reoperations or complications after the index procedure were identified by interrogation of the database and review of the medical notes.

RESULTS

A total of 342 athletes with a mean follow-up of 100.2 ± 51.9 months (range, 24-215 months) were analyzed. Graft failures totaling 31 (9.1%) were reported, requiring revision surgery because of symptomatic instability. The rate of graft failure was significantly higher when ACLR was not combined with a LEAP (15.5% vs 6.0%; P = .0105) and in athletes aged 21 years or younger (13.8% vs 6.6%; P = .0290). A multivariate analysis was performed using the Cox model and demonstrated that athletes undergoing an isolated ACLR were at >2-fold risk of ACL graft rupture (hazard ratio [HR], 2.678 [1.173; 4.837], P = .0164) when compared with those undergoing a combined ACLR with a LEAP. Additionally, athletes aged ≤21 years were also at >2-fold risk of graft failure (HR, 2.381 [1.313; 5.463]; P = .0068) than those aged >21 years. Sex, sport, and graft type were not found to be significant risk factors for graft failure.

CONCLUSION

Professional athletes undergoing isolated ACLR and aged ≤21 years are at >2-fold greater risk of graft failure. Orthopaedic surgeons treating elite athletes should combine an ACLR with a LEAP to improve ACL graft survivorship.

Scoping Review on ACL Surgery and Registry Data

Purpose of Review

To present an overview of registry-based anterior cruciate ligament (ACL) research, as well as provide insight into the future of ACL registries.

Recent Findings

During the past decades, the ACL registries have had an important role in increasing our understanding of patients with ACL injuries and their treatment. The registry data has deepened our understanding of factors that have been associated with an increased risk of sustaining an ACL injury and for evaluation of treatment factors and their impact on patient-related outcomes. Recently, registry-based ACL research using artificial intelligence (AI) and machine learning (ML) has shown potential to create clinical decision-making tools and analyzing outcomes. Thus, standardization of collected data between the registries is needed to facilitate the further collaboration between registries and to facilitate the interpretation of results and subsequently improve the possibilities for implementation of AI and ML in the registry-based research.

Summary

Several studies have been based on the current ACL registries providing an insight into the epidemiology of ACL injuries as well as outcomes following ACL reconstruction. However, the current ACL registries are facing future challenges, and thus, new methods and techniques are needed to ensure further good quality and clinical applicability of study findings based on ACL registry data.

Statistical Fragility of Single-Row Versus Double-Row Anchoring for Rotator Cuff Repair: A Systematic Review of Comparative Studies

Background:

Comparative studies and randomized controlled trials (RCTs) often use the P (probability) value to convey the statistical significance of their findings. P values are an imperfect measure, however, and are vulnerable to a small number of outcome reversals to alter statistical significance. The inclusion of a fragility index (FI) and fragility quotient (FQ) may aid in the interpretation of a study’s statistical strength.

Purpose/Hypothesis:

The purpose of this study was to examine the statistical stability of studies comparing single-row to double-row rotator cuff repair. It was hypothesized that the findings of these studies would be vulnerable to a small number of outcome event reversals, often fewer than the number of patients lost to follow-up.

Study Design:

Systematic review; Level of evidence, 3.

Methods:

We analyzed comparative studies and RCTs on primary single-row versus double-row rotator cuff repair that were published between 2000 and 2021 in 10 leading orthopaedic journals. Statistical significance was defined as a P < .05. The FI for each outcome was determined by the number of event reversals necessary to alter significance. The FQ was calculated by dividing the FI by the respective sample size.

Results:

Of 4896 studies screened, 22 comparative studies, 10 of which were RCTs, were ultimately included for analysis. A total of 74 outcomes were examined. Overall, the median FI was 2 (interquartile range [IQR], 1-3), and the median FQ was 0.035 (IQR, 0.020-0.057). The mean FI was 2.55 ± 1.29, and the mean FQ was 0.043 ± 0.027. In 64% of outcomes, the FI was less than the number of patients lost to follow-up.) Additionally, 81% of significant outcomes needed just a single outcome reversal to lose their significance.

Conclusion:

Over half of the studies currently used to guide clinical practice have a number of patients lost to follow-up greater than their FI. The results of these studies should be interpreted within the context of these limitations. Future analyses may benefit from the inclusion of the FI and the FQ in their statistical analyses.

The Statistical Fragility of Single-Bundle vs Double-Bundle Autografts for ACL Reconstruction: A Systematic Review of Comparative Studies

Background:

The statistical significance of a given study outcome can be liable to small changes in findings. P values are common, but imperfect statistical methods to convey significance, and inclusion of the fragility index (FI) and fragility quotient (FQ) may provide a clearer perception of statistical strength.

Purpose/Hypothesis:

The purpose was to examine the statistical stability of studies comparing primary single-bundle to double-bundle anterior cruciate ligament reconstruction (ACLR) utilizing autograft and independent tunnel drilling. It was hypothesized that the study findings would be vulnerable to a small number of outcome event reversals, often less than the number of patients lost to follow-up.

Study Design:

Systematic review; Level of evidence, 2.

Methods:

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, the authors searched PubMed for comparative studies and randomized controlled trials (RCTs) published in select journals, based on impact factor, between 2005 and 2020. Risk-of-bias assessment and methodology scoring were conducted for the included studies. A total of 48 dichotomous outcome measures were examined for possible event reversals. The FI for each outcome was determined by the number of event reversals necessary to alter significance. The FQ was calculated by dividing the FI by the respective sample size.

Results:

Of the 1794 studies screened, 15 comparative studies were included for analysis; 13 studies were RCTs. Overall, the mean FI and FQ were 3.14 (IQR, 2-4) and 0.050 (IQR, 0.032-0.062), respectively. For 72.9% of outcomes, the FI was less than the number of patients lost to follow-up.

Conclusion:

Studies comparing single-bundle versus double-bundle ACLR may not be as statistically stable as previously thought. Comparative studies and RCTs are at substantial risk for statistical fragility, with few event reversals required to alter significance. The reversal of fewer than 4 outcome events in a treatment group can alter the statistical significance of a given result; this is commonly less than the number of patients lost to follow-up. Future comparative study analyses might consider including FI and FQ with P values in their statistical analysis.

Fragility Index and Fragility Quotient in Randomized Controlled Trials on Corticosteroids in ARDS Due to COVID-19 and Non-COVID-19 Etiology

Background: The effectiveness of corticosteroids in acute respiratory distress syndrome (ARDS) and COVID-19 still remains uncertain. Since ARDS is due to a hyperinflammatory response to a direct injury, we decided to perform a meta-analysis and an evaluation of robustness of randomised clinical trials (RCTs) investigating the impact of corticosteroids on mortality in ARDS in both COVID-19 and non-COVID-19 patients. We conducted a systematic search of the literature from inception up to 30 October 2020, using the MEDLINE database and the PubMed interface. We evaluated the fragility index (FI) of the included RCTs using a two-by-two contingency table and the p-value produced by the Fisher exact test; the fragility quotient (FQ) was calculated by dividing the FI score by the total sample size of the trial. Results: Thirteen RCTs were included in the analysis; five of them were conducted in COVID-19 ARDS, including 7692 patients, while 8 RCTS were performed in non-COVID ARDS with 1091 patients evaluated. Three out of eight RCTs in ARDS had a FI > 0 while 2 RCTs out of five in COVID-19 had FI > 0. The median of FI for ARDS was 0.625 (0.47) while the median of FQ was 0.03 (0.014). The median of FI for COVID-19 was 6 (2) while the median of FQ was 0.059 (0.055). In this systematic review, we found that FI and FQ of RCTs evaluating the use of corticosteroids in ARDS and COVID-19 were low.

The Fragility of Significance in the Hip Arthroscopy Literature: A Systematic Review

The purpose of the present study was to perform the first examination of the utility of p values and the degree of statistical fragility in the hip arthroscopy literature by applying both the Fragility Index (FI) and the Fragility Quotient (FQ) to dichotomous comparative trials. We hypothesized that dichotomous comparative trials evaluating categorical outcomes in the hip arthroscopy literature are statistically fragile.

METHODS

The PubMed and MEDLINE databases were queried from 2008-2018 for comparative studies evaluating dichotomous data in the hip arthroscopy literature. The present analysis included both randomized controlled trials (RCTs) and non-RCTs in which dichotomous data and associated p values were reported. Fragility analysis was performed with use of the Fisher exact test until an alteration of significance was determined.

RESULTS

Of the 5,836 studies screened, 4,156 met the search criteria, with 52 comparative studies included for analysis. One hundred and fifty total outcome events with 33 significant (p < 0.05) outcomes and 117 nonsignificant (p ≥ 0.05) outcomes were identified. The final FI incorporating all 150 outcome events from 52 comparative studies was only 3.5 (interquartile range, 2 to 6), with an associated FQ of 0.032 (interquartile range, 0.017 to 0.063). Twenty-two studies (42.3%) either failed to report loss to follow-up (LTF) data or reported LTF greater than the overall FI of 3.5.

CONCLUSIONS

The peer-reviewed hip arthroscopy literature may not be as stable as previously thought, as the sole reliance on a threshold p value has proven misleading. We therefore recommend reporting of the FI and FQ, in conjunction with p values, to aid in the evaluation and interpretation of statistical robustness and quantitative significance in future comparative hip arthroscopy studies.

The fragility of randomized placebo-controlled trials for irritable bowel syndrome

BACKGROUND

The fragility index (FI) represents the number of participants whose status in a trial would have to change from a non-event (not experiencing the primary endpoint) to an event (experiencing the primary endpoint) in order to turn a statistically significant result into a non-significant result. We sought to evaluate the fragility indices of irritable bowel syndrome [IBS-mixed (IBS-M), IBS-constipation (IBS-C), & IBS-diarrhea (IBS-D)] trials.

METHODS

Irritable bowel syndrome trials published in high-impact journals were identified from Medline. Trials had to be in adults, randomized, parallel-armed, with at least one statistically significant binary outcome, and an achieved primary endpoint of therapeutic efficacy. FI and correlation coefficients were calculated, and regression modeling used to identify predictors of a high FI.

KEY RESULTS

Twelve trials were analyzed with a median FI of 6 (range: 0-38). Median sample size in all trials was 366 (range: 44-856). Trial publication year (p = 0.71), journal impact factor (p = 0.52), duration of study (p = 0.12), and number need to treat [NNT] (p = 0.29) were not predictive of a high FI. While a lower p-value correlated with a higher FI (p = 0.039), no correlation was noted between FI and impact factor (R = -0.20, p = 0.52), trial publication year (R = 0.12, p = 0.71), duration of trial (R = -0.46, p = 0.13), NNT (R = -0.34, p = 0.29), and sample size (R = 0.23, p = 0.5). The highest FI was in a Ramosetron trial (FI = 30) for IBS-D.

CONCLUSION & INFERENCES

A median of six participants is needed to nullify results in the included IBS trials suggesting how easily statistical significance based on a threshold p-value may be overturned.

The Statistical Fragility of Hamstring Versus Patellar Tendon Autografts for Anterior Cruciate Ligament Reconstruction: A Systematic Review of Comparative Studies

BACKGROUND

Evidence-based medicine utilizes data to inform clinical decision making, despite the ability of a small number of outcome reversals to change statistical significance. P values are common measurements of statistical significance that possess inherent flaws. The inclusion of the fragility index (FI) and fragility quotient (FQ) may provide a clearer conveyance of statistical strength.

PURPOSE/HYPOTHESIS

The purpose was to examine the statistical stability of studies comparing hamstring tendon and bone-patellar tendon-bone autografts in primary single-bundle anterior cruciate ligament reconstruction with independent tunnel drilling. We hypothesized that the findings of these studies are vulnerable to a small number of outcome event reversals, often fewer than the number of patients lost to follow-up.

STUDY DESIGN

Systematic review.

METHODS

Comparative studies and randomized controlled trials (RCTs) published in 10 leading orthopaedic journals between 2000 and 2020 were analyzed. Statistical significance was defined as a P value ≤.05. FI for each outcome was determined by the number of event reversals necessary to alter significance. FQ was calculated by dividing the FI by the respective sample size.

RESULTS

Of the 1803 studies screened, 643 met initial search criteria, with 18 comparative studies ultimately included for analysis, 8 of which were RCTs. A total of 114 outcomes were examined. Overall, the mean (interquartile range) FI and FQ were 3.77 (2-4) and 0.040 (0.016-0.055), respectively. The FI was less than the number of patients lost to follow-up for 76.3% of outcomes.

CONCLUSION

Studies examining graft choice for anterior cruciate ligament reconstruction may not be as statistically stable as previously thought. Comparative studies and RCTs are at substantial risk for statistical fragility, with few event reversals required to alter significance. The reversal of <4 outcome events in a treatment group can alter the statistical significance of a given result; this is commonly fewer than the number of patients lost to follow-up. Future comparative study analyses might consider including FI and FQ with P values in their statistical analysis.

Urologic oncology randomized controlled trials are frequently fragile - A review of the urology literature

BACKGROUND

Randomized controlled trials (RCT) in urologic oncology are the basis of patient management. Considerable debate exists on the limitation of statistical reporting of randomized controlled trials. Fragility index (FI) is a measure of the number of events upon which the trials statistical results depend on. The FI is defined as the minimum number of patients whose status would have to change from a 'non-event' to 'event', in order to turn a statistically significant result to a non-significant result and vice versa. Our aim is to examine the FI of RCT's in urologic oncology published in the urology literature.

MATERIAL AND METHODS

We exhaustively searched MEDLINE and EMBASE from January 1, 2016 to December 31 2019 for RCT's in urology journals. Only studies reporting dichotomous outcomes were included and FI was calculated for each outcome. The distributions of FI across different journals and types of outcome (primary/ secondary, significant/ non-significant) were assessed. We examined the correlation of FI with sample size and P-value.

RESULTS

We identified 216 RCT's, 79 were eligible for analysis. Median FI was 3.0 (2.5, 6.0). One hundred and forty-six (89.6%) outcomes from 72 (94.7%) RCT's had a FI lower than 10 and 6 (3.7%) outcomes from 6 (7.9%) RCT's had a FI = 0. There was no statistically significant difference in FI between different types of outcomes. There was a modest correlation between the FI and the sample size (r_s 0.50, P < 0.01) and a weak correlation between the FI the P value (r_s 0.15 (P < 0.04)).

CONCLUSIONS

Randomized controlled trials in urology journals that study dichotomous outcomes often report fragile results that should be regarded with caution. Reporting the FI alongside P values may enhance the interpretation and implementation of urologic oncology RCT's.

How fragile are Mediterranean diet interventions? A research-on-research study of randomised controlled trials

INTRODUCTION

The Mediterranean diet (MD) is a traditional regional dietary pattern and a healthy diet recommended for the primary and secondary prevention of various diseases and health conditions. Results from the higher level of primary evidence, namely randomised controlled trials (RCTs), are often used to produce dietary recommendations; however, the robustness of RCTs with MD interventions is unknown.

METHODS

A systematic search was conducted and all MD RCTs with dichotomous primary outcomes were extracted from PubMed. The fragility (FI) and the reverse fragility index (RFI) were calculated for the trials with significant and non-significant comparisons, respectively.

RESULTS

Out of 27 RCTs of parallel design, the majority failed to present a significant primary outcome, exhibiting an FI equal to 0. The median FI of the significant comparisons was 5, ranging between 1 and 39. More than half of the comparisons had an FI <5, indicating that the addition of 1-4 events to the treatment arm eliminated the statistical significance. For the comparisons with an FI=0, the RFI ranged between 1 and 29 (Median RFI: 7). When the included RCTs were stratified according to masking, the use of a composite primary endpoint, sample size, outcome category, or dietary adherence assessment method, no differences were exhibited in the FI and RFI between groups, except for the RFI among different compliance assessment methods.

CONCLUSIONS

In essence, the present study shows that even in the top tiers of evidence hierarchy, research on the MD may lack robustness, setting concerns for the formulation of nutrition recommendations.

Degree of Anterolateral Ligament Injury Impacts Outcomes After Double-Bundle Anterior Cruciate Ligament Reconstruction

PURPOSE

To evaluate the effect of anterolateral ligament (ALL) injury identified on preoperative magnetic resonance imaging (MRI) on postoperative outcomes after double-bundle (DB) anterior cruciate ligament reconstruction (ACLR).

METHODS

For this retrospective study, the inclusion criteria were patients who were at least 3 years out of DB ACLR. Exclusion criteria included a delay in MRI over 4 weeks, delay in surgery over 6 months, single-bundle ACLR, and revision surgery. Enrolled patients were divided into 2 groups according to the ALL injury grade in preoperative MRI by a musculoskeletal radiologist who was blinded to the perioperative findings (the high-grade group with complete or nearly complete tear: n = 53 and the low-grade group with intact ALL or partial tear: n = 33). Knee laxity, clinical outcomes using the International Knee Documentation Committee (IKDC) examination form, and revision rates were compared at the last follow-up (8.1 ± 2.2 years). An independent t test was applied to compare continuous variables, and χ² or Fisher exact test was used to compare the nominal variables.

RESULTS

The anterior translation was 3.2 ± 1.9 mm in the high-grade group and 1.6 ± 1.0 mm in the low-grade group (P < .001). The high-grade group showed 18 cases with a pivot-shift grade of 2 or 3 (40.0%); however, the low-grade group showed only 1 case with a pivot-shift grade 2 or 3 (3.0%) (P = .002). The high-grade group also showed inferior outcomes in the IKDC objective grade (grade A: 49.0%; grade B: 17.0%; grade C: 30.2%; grade D: 3.8% vs grade A: 90.9%; grade B: 6.1%; grade C: 3.0%; grade D: 0%, P = .001) and IKDC subjective score (87.5 ± 9.9 vs 93.9 ± 5.3, P < .001). In addition, the high-grade group showed a greater revision rate (11.3% vs 0%, P = .045).

CONCLUSIONS

DB ACLR for patients with high-grade ALL injury resulted in increased knee laxity, worse clinical outcomes, and higher revision rate compared to patients with low-grade ALL injury.

LEVEL OF EVIDENCE

Level III, retrospective comparative study.

How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines

Objectives

“Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016” provides guidelines in regard to prompt management and resuscitation of sepsis or septic shock. The study is aimed to assess the robustness of randomized controlled trials (RCTs) that formulate these guidelines in terms of fragility index and reverse fragility index.

Method

RCTs that contributed to these guidelines having parallel two-group design, 1:1 allocation ratio, and at least one dichotomous outcome were included in the study. The median fragility index was calculated for RCTs with significant statistical outcomes, whereas the median reverse fragility index was calculated for RCTs with nonsignificant statistical results.

Results

Hundred RCTs that met the inclusion criteria were analyzed. The median fragility index was 5.5 [95% confidence interval (CI) 1–30] and median reverse fragility index was 13 (95% CI 12.07–16.8) at a p value of 0.05. The median reverse fragility index was 16 (95% CI 10–26) at a p value of 0.01. Most of the RCTs included in this analysis were of good quality, having a median Jadad score of 6.

Conclusion

This analysis found that the surviving sepsis guidelines were based on highly robust RCTs with statistically insignificant results and on some moderately robust RCTs with statistically significant results. RCTs with statistically insignificant results were more robust than RCTs with statistically significant results in regard to these guidelines.

Highlights

The study assessed the robustness of randomized controlled trials (RCTs) that were used to formulate surviving sepsis guidelines. Most RCTs showed statistically nonsignificant results. RCTs with statistically significant results were moderately fragile whereas RCTs with nonsignificant results were more robust.

How to cite this article

Choupoo NS, Das SK, Saikia P, Dey S, Ray S. How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines. Indian J Crit Care Med 2021;25(7):773–779.

Dismantling the Fragility Index: A demonstration of statistical reasoning

The Fragility Index has been introduced as a complement to the P-value to summarize the statistical strength of evidence for a trial's result. The Fragility Index (FI) is defined in trials with two equal treatment group sizes, with a dichotomous or time-to-event outcome, and is calculated as the minimum number of conversions from nonevent to event in the treatment group needed to shift the P-value from Fisher's exact test over the .05 threshold. As the index lacks a well-defined probability motivation, its interpretation is challenging for consumers. We clarify what the FI may be capturing by separately considering two scenarios: (a) what the FI is capturing mathematically when the probability model is correct and (b) how well the FI captures violations of probability model assumptions. By calculating the posterior probability of a treatment effect, we show that when the probability model is correct, the FI inappropriately penalizes small trials for using fewer events than larger trials to achieve the same significance level. The analysis shows that for experiments conducted without bias, the FI promotes an incorrect intuition of probability, which has not been noted elsewhere and must be dispelled. We illustrate shortcomings of the FI's ability to quantify departures from model assumptions and contextualize the FI concept within current debate around the null hypothesis significance testing paradigm. Altogether, the FI creates more confusion than it resolves and does not promote statistical thinking. We recommend against its use. Instead, sensitivity analyses are recommended to quantify and communicate robustness of trial results.

Fragility Index in Cardiovascular Randomized Controlled Trials

BACKGROUND

Efficacy of an intervention is commonly evaluated using P values, in addition to effect size measures such as absolute risk reduction, relative risk reduction, and numbers needed to treat. However, these measures are not always intuitive to clinicians. The fragility index (FI) is a more intuitive number that can facilitate interpretation but can only be used with binary outcomes. FI is the minimum number of patients who must be moved from the nonevent group to the event group to turn a significant result nonsignificant. In this retrospective analysis, we assessed the robustness of cardiovascular randomized controlled trials (RCTs), which report a positive (statistically significant) primary outcome by using the FI.

METHODS AND RESULTS

We searched Medline from 2007 to 2017 to identify cardiovascular RCTs published in 6 high impact journals (The Lancet, New England Journal of Medicine, Journal of the American Medical Association, Circulation, Journal of the American College of Cardiology and European Heart Journal). Only RCTs with sample sizes >500 and a 2-by-2 factorial design or dichotomous primary outcomes were selected. FI was calculated using a defined approach. Among the cohort of 123 RCTs that met inclusion criteria, median FI was 13 (interquartile range, 5-26). In 28 trials (22.8%), FI ranged between 1 and 4. In 37 trials (30.1%), number of patients lost to follow-up was higher than the FI. Pharmaceutical interventions had higher FI compared with other interventions, FI=19 (7-52; P=0.002). Median FI varied according to subspecialty (electrophysiology=2; heart failure=11; interventional cardiology=8; P=0.020) and multiregional RCTs had higher FI=22 (12-53.25; P=0.023). FI did not differ based on risk of bias indicators, funding, or publication year.

CONCLUSIONS

Considerable variations in FI were observed among cardiovascular trials, suggesting the need for careful interpretation of results, particularly when number of patients lost to follow-up exceeds FI.