The fragility of findings of randomized controlled trials in shoulder and elbow surgery

Challenging the Concept of Statistical Fragility: Is There Any Value Added?

ABSTRACT

Today, well-designed randomized clinical trials (RCTs) are considered the pinnacle of clinical research, and they inform many practices in orthopaedics. When designing these studies, researchers conduct a power analysis, which allows researchers to strike a balance between (1) enrolling enough patients to detect a clinically important treatment effect (i.e., researchers can be confident that the effect is unlikely due to chance) and (2) cost, time, and risk to patients, which come with enrolling an excessive number of patients. Because researchers will have a desire to conduct resource-efficient RCTs and protect patients from harm, many studies report a p value that is close to the threshold for significance. The concept of the fragility index (FI) was introduced as a simple way to interpret RCT findings, but it does not account for RCT design. The adoption of the FI conflicts with researchers' goals of designing efficient RCTs that conserve resources and limit ineffective or harmful treatments to patients. The use of the FI may reflect many clinicians' lack of familiarity with interpreting p values beyond "significant" or "nonsignificant." Instead of inventing new metrics to convey the same information provided by the p value, greater emphasis should be placed on educating clinicians on how to interpret p values and, more broadly, statistics, when reading scientific studies.

The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

PURPOSE

To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature.

METHODS

All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF.

RESULTS

Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal.

CONCLUSIONS

The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT.

CLINICAL RELEVANCE

Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.

The fragility index: how robust are the outcomes of head and neck cancer randomised, controlled trials?

BACKGROUND

The fragility index represents the minimum number of patients required to convert an outcome from statistically significant to insignificant. This report assesses the fragility index of head and neck cancer randomised, controlled trials.

METHODS

Studies were extracted from PubMed/Medline, Scopus, Embase and Cochrane databases.

RESULTS

Overall, 123 randomised, controlled trials were included. The sample size and fragility index medians (interquartile ranges) were 103 (56-213) and 2 (0-5), respectively. The fragility index exceeded the number of patients lost to follow up in 42.3 per cent (n = 52) of studies. A higher fragility index correlated with higher sample size (r = 0.514, p < 0.001), number of events (r = 0.449, p < 0.001) and statistical significance via p-value (r = -0.367, p < 0.001).

CONCLUSION

Head and neck cancer randomised, controlled trials demonstrated low fragility index values, in which statistically significant results could be nullified by altering the outcomes of just two patients, on average. Future head and neck oncology randomised, controlled trials should report the fragility index in order to provide insight into statistical robustness.

The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

BACKGROUND

Evidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.

PURPOSE

To calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).

STUDY DESIGN

Meta-analysis; Level of evidence, 2.

METHODS

A search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.

RESULTS

There were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial's sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).

CONCLUSION

More than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

The statistical fragility of the management options for reverse shoulder arthroplasty: a systematic review of randomized control trial with fragility analysis

Reverse shoulder arthroplasty (RSA) is used in the treatment of traumatic and arthritic pathologies, with expanding clinical indications and as a result there has been an increase in clinical research on the topic. The purpose of this study was to examine the statistical fragility of randomized control trials (RCTs) reporting outcomes from RSA. A systematic search was undertaken to find RCTs investigating RSA. The Fragility Index (FI) was calculated using Fisher's exact test, by sequentially altering the number of events until there was a reversal of significance. The Fragility Quotient (FQ) was calculated by dividing the FI by the trial population. Each trial was assigned an overall FI and FQ calculated as the median result of its reported findings. Overall, 19 RCTs warranted inclusion in the review, representing 1146 patients, of which 41.2% were male, with a mean age of 74.2 ± 4.3 years and mean follow-up of 22.1 ± 9.9 months. The median RCT population was 59, with a median of 9 patients lost to follow-up. The median FI was 4.5, and median FQ was 0.083, indicating more patients did not complete the trial than the number of outcomes which would have to change to reverse the finding of significance. This review found that the RCT evidence for RSA management may be vulnerable to statistical fragility, with a handful of events required to reverse a finding of significance.

The statistical fragility of studies on rotator cuff repair with graft augmentation

BACKGROUND

Clinical decision-making often relies on evidence-based medicine. Our purpose was to determine the fragility index (FI) and fragility quotient (FQ) for studies evaluating rotator cuff repair (RCR) with graft augmentation. A lost to follow-up (LTF) value greater than the FI indicates statistical instability for the reported outcomes and conclusions.

METHODS

We performed a systematic review using Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines by searching PubMed, the Cochrane library, and Embase in June 2022 to identify studies of RCR with graft augmentation. Comparative studies with at least 1 statistically analyzed dichotomous outcome were included. Seventeen studies published in seven peer-reviewed journals from 2003 to 2019 were subsequently evaluated. The FI was determined by changing each reported outcome event within 2 × 2 contingency tables until statistical significance or nonsignificance was reversed. The associated FQ was determined by dividing the FI by the sample size. LTF values were also extracted from each included study.

RESULTS

The included studies had a total of 1098 patients with 36 dichotomous outcomes. The associated median FI was 4 (interquartile range 2-5), indicating that the reversal of 4 patients' outcomes would have reversed the finding of significant difference. The median FQ was 0.08 (interquartile range 0.04-0.15), indicating that in a sample of 100 patients, reversal of 8 patients' outcomes would reverse statistical significance. The median number of patients LTF was 3 (range 0-25), with 56% of reported outcomes having LTF greater than their respective FI.

CONCLUSION

Studies of RCR with graft augmentation lack statistical stability, with few altered outcome events required to reverse statistical significance. Larger comparative studies with better follow-up will strengthen the statistical stability of the evidence for RCR with graft augmentation. For future investigations and reports, we recommend including FI and FQ along with traditional statistical significance analyses to provide better context on the strength of conclusions.

Evidence-Based Practice Should Supersede Evidence-Based Medicine Through Consideration of Clinical Experience and Patient Characteristics in Addition to the Published Literature

On the surface, the benefits of evidence-based medicine (EBM) seem self-evident. However, reliance on the scientific literature alone has limitations. Studies may be biased, statistically fragile, and/or not reproducible. Reliance solely on EBM may ignore physician clinical experience and individual patient characteristics and input. Reliance solely on EBM may overvalue quantitative, statistical significance, resulting in a false sense of certainty. Reliance solely on EBM may fail to consider lack of generalizability of published studies to individually unique patients. The concept of evidence-based practice goes beyond EBM and incorporates (1) EBM, (2) clinical expertise, and (3) individual patient characteristics, values, and preferences. Even if branded as evidence-based, a suggested treatment may not be the best treatment. Evidence-based practice must be considered before determining what is best for our patients.

The Statistical Fragility of Foot and Ankle Surgery Randomized Controlled Trials

Fragility index (FI) is a metric used to interpret the results of randomized controlled trials (RCTs), and describes the number of subjects that would need to be switched from event to non-event for a result to no longer be significant. Studies that analyze FI of RCTs in various orthopedic subspecialties have shown the RCTs to be largely underpowered and highly fragile. However, FI has not been assessed in foot and ankle RCTs. The MEDLINE and Embase online databases were searched from 1/1/2011 through 11/19/2021 for RCTs involving foot and ankle conditions. FI, fragility quotient (FQ), and difference between the FI and number of subjects lost to follow-up was calculated. Spearman correlation was performed to determine the relationship between sample size and FI. Overall, 1262 studies were identified of which 18 were included in the final analysis. The median sample size was 65 (interquartile range [IQR] 57-95.5), the median FI was 2 (IQR 1-2.5), and the median FQ was 0.026 (IQR 0.012-0.033). Ten of 15 (67%) studies with non-zero FI values had FI values less than the number of subjects lost to follow-up. There was linear association between FI and sample size (R² = 0.495, p-value: .031). This study demonstrates that RCTs in the field of foot and ankle surgery are highly fragile, similar to other orthopedic subspecialties.

The Statistical Fragility of Operative vs Nonoperative Management for Achilles Tendon Rupture: A Systematic Review of Comparative Studies

BACKGROUND

The statistical significance of randomized controlled trials (RCTs) and comparative studies is often conveyed utilizing the P value. However, P values are an imperfect measure and may be vulnerable to a small number of outcome reversals to alter statistical significance. The interpretation of the statistical strength of these studies may be aided by the inclusion of a Fragility Index (FI) and Fragility Quotient (FQ). This study examines the statistical stability of studies comparing operative vs nonoperative management for Achilles tendon rupture.

METHODS

A systematic search was performed of 10 orthopaedic journals between 2000 and 2021 for comparative studies focusing on management of Achilles tendon rupture reporting dichotomous outcome measures. FI for each outcome was determined by the number of event reversals necessary to alter significance (P < .05). FQ was calculated by dividing the FI by the respective sample size. Additional subgroup analyses were performed.

RESULTS

Of 8020 studies screened, 1062 met initial search criteria with 17 comparative studies ultimately included for analysis, 10 of which were RCTs. A total of 40 outcomes were examined. Overall, the median FI was 2.5 (interquartile range [IQR] 2-4), the mean FI was 2.90 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). The FI was less than the number of patients lost to follow-up for 78% of outcomes.

CONCLUSION

Studies examining the efficacy of operative vs nonoperative management of Achilles tendon rupture may not be as statistically stable as previously thought. The average number of outcome reversals needed to alter the significance of a given study was 2.90. Future analyses may benefit from the inclusion of a fragility index and a fragility quotient in their statistical analyses.

The Fragility Index of Statistically Significant Findings From Randomized Controlled Trials Comparing the Management Strategies of Anterior Shoulder Instability

BACKGROUND

Debate centering on the management of anterior shoulder instability (ASI) in recent years has led to many randomized controlled trials (RCTs) being published on the topic. The fragility index (FI) has subsequently emerged as a novel method of assessing significant findings reported in RCTs, particularly those with small sample sizes.

PURPOSE

To evaluate the FI of statistically significant findings in RCTs that reported the outcomes of management strategies of patients with ASI.

STUDY DESIGN

Systematic review; Level of evidence, 1.

METHODS

Using PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, 2 independent reviewers performed a systematic review of RCTs focusing on the outcomes of management strategies of patients with ASI. There were 3 main categories of RCTs included: (1) nonoperative management in internal rotation (IR) versus external rotation (ER), (2) nonoperative management versus a surgical intervention, and (3) surgical management with arthroscopic Bankart repair versus open Bankart repair. The Fisher exact test was utilized to calculate the FI for the reversal of statistical significance in all statistically significant dichotomous outcomes.

RESULTS

A total of 21 RCTs were included, including 1589 shoulders (mean age, 29.4 years) with a mean follow-up of 26.8 months. There were 10 RCTs (831 shoulders) that reported outcomes after the nonoperative management of ASI in IR versus ER, with a mean FI of 6.8. There were 5 RCTs (324 shoulders) that reported outcomes comparing the nonoperative and operative management of ASI, with a mean FI of 3.5. There were 6 RCTs (434 shoulders) that reported outcomes after the operative management of ASI with either arthroscopic Bankart repair or open Bankart repair, with a mean FI of 9.6.

CONCLUSION

The overall FI of RCTs reporting the outcomes of management strategies for patients with ASI was high, suggesting a moderate fragility of statistically significant outcomes including recurrence, revision stabilization, and return to play.

Editorial Commentary: A Consensus of Experts Complements the Clinical Evidence on Diagnosis and Treatment of Anterior Glenohumeral Instability

The current diagnostic and treatment strategies for anterior glenohumeral instability have been refined by high-quality clinical and basic science studies, but many controversies remain. These include the bone loss threshold for arthroscopic Bankart repair and the influence of other clinical factors on this decision, the optimal bracing position following anterior glenohumeral dislocation, and the optimal coracoid graft orientation during the Latarjet procedure. Randomized clinical trials often present conflicting results, and many of these are small-sample and fragile studies. Obtaining an expert consensus on the topic, by means of the Delphi method, is an attractive alternative to such clinical trials. Several studies employing variations of the Delphi method have addressed the diagnosis and treatment of anterior glenohumeral instability. These have stressed the importance of a meticulous technique during arthroscopic Bankart repair and of recognition of glenoid and humeral bone loss and treating this appropriately. These studies have also helped identify areas where consensus is modest or lacking to motivate additional clinical research study.

Comparative Studies in the Shoulder Literature Lack Statistical Robustness: A Fragility Analysis

Purpose

Evidenced-based decision-making is rooted in comparative clinical studies; however, a small number of outcome event reversals have the potential to change study significance. The purpose of this study was to determine the utility of applying fragility analysis to comparative studies in the published orthopaedic shoulder literature.

Methods

Comparative clinical shoulder research studies reporting 1:1 dichotomous categorical data were analyzed in 6 leading orthopaedic journals between 2006 and 2016. Statistical significance was defined as a P value of less than .05. The fragility index (FI) for each study outcome was determined by the number of event reversals required to change the P value to either greater or less than 0.05, thus changing the study conclusions. The associated fragility quotient (FQ) was determined by dividing the FI by the total population comprising a particular outcome.

Results

Of the 23,897 studies screened, 3,591 met search criteria, with 198 comparative studies ultimately included for analysis, 67 of which were randomized controlled trials. There were 357 total outcome events with 74 reported as significant and 283 as not significant. The FI was 4 (IQR 2-6) with an associated FQ of 0.066 (interquartile range [IQR] 0.038-0.102). There was no difference in statistical fragility between randomized and nonrandomized trials with both revealing a FI of 4 and FQ of 0.068 (IQR 0.044-0.107) and 0.065 (IQR 0.031-0.101), respectively.

Conclusions

This current analysis reveals that comparative shoulder studies published in six leading orthopaedic journals are at risk of statistical fragility. As such, contemporary clinical shoulder literature may not be as robust as traditionally perceived with the reversal of only a few outcome events required to change study significance. Therefore, we advocate the reporting of both FI and FQ in addition to the P value as statistical complements to all comparative investigations to provide a more comprehensive understanding of trial stability and significance in the published shoulder literature.

Clinical Relevance

Comparative study designs are commonly employed in shoulder research. Several studies in both the general medical and orthopaedic literature have identified a lack of statistical robustness through comprehensive fragility analysis. Our findings demonstrate the P value may be an inadequate independent statistical metric requiring the complement of a FI and FQ to aid in the interpretation and understanding of study significance for clinical decision-making.

Editorial Commentary: High Level of Evidence Randomized Controlled Trials May Report Statistically Fragile Conclusions

Randomized controlled trials (RCTs) are considered the gold standard for evidence-based medicine, and although a well-designed and executed RCT can be extremely powerful, many RCTs have significant flaws that may significantly impact the strength of their conclusions. The fragility index is a metric that objectifies the strength of results from RCTs and is one such metric that should be considered when critically evaluating individual studies. Although the RCTs pertaining to hip arthroscopy have a median value of 4, which is slightly greater than other orthopaedic subspecialties investigated thus far, this is far from a robust result, which mostly highlights the difficulty in performing an RCT in the surgical realm. Some topics of investigation may be better suited for well performed case-control, matched cohort, and meta-analysis studies (among other investigation types), which may provide better and more robust conclusions than a poorly performed RCT. In essence, don't be enamored by level of evidence alone.

The Fragility of Statistically Significant Findings From Randomized Controlled Trials in Hip and Knee Arthroplasty

BACKGROUND

The Fragility Index (FI) is a method for evaluating the robustness of statistically significant findings from randomized controlled trials (RCTs) beyond the P value in trials with dichotomous outcomes. The FI is defined as the number of patients in one arm of a trial that would have to have a different outcome to change the results of the trial from statistically significant to nonsignificant. This review assessed the FI in arthroplasty RCTs.

METHODS

A systematic search was conducted in MEDLINE, Embase, and Web of Science for RCTs related to primary total joint arthroplasty (TJA) from 2010 to 2020. Trials with a statistically significant dichotomous primary outcome were included. The FI was calculated using Fisher's exact test to determine how many events would need to be reversed to change a study from statistically significant to nonsignificant.

RESULTS

A total of 34 RCTs were included. The median sample size was 103 patients (range 24-791). The median FI was 1 (range 0-45), meaning that reversing the outcome of just one patient in either treatment group of each trial would change it from a significant to a nonsignificant result.

CONCLUSION

Hip and knee arthroplasty RCTs with statistically significant dichotomous outcomes in TJA are fragile. The median FI in TJA is lower than the FI in any of the other previously reported orthopedic subspecialties. Fragility is another reason to be cautious when conducting or interpreting small trials, and to continue to strive toward large trials to answer important questions in TJA.

LEVEL OF EVIDENCE

Level I.

The Fragility Index of Hip Arthroscopy Randomized Controlled Trials: A Systematic Survey

PURPOSE

To characterize the fragility index (FI) of statistically significant results from randomized controlled trials (RCTs) in hip arthroscopy.

METHODS

The PubMed-MEDLINE, Embase, and Cochrane databases were queried for hip arthroscopy RCTs published between January 2010 and July 2020. RCTs were included if they contained only 2 treatment arms, randomized patients to a 1:1 allocation to each arm, and reported at least 1 statistically significant dichotomous outcome. The fragility quotient was calculated for each RCT by dividing the FI by the sample size. Smaller FIs indicated more fragile results. Risk of bias was assessed using the Cochrane risk-of-bias tool for randomized trials.

RESULTS

We identified 8 hip arthroscopy RCTs that met all inclusion and exclusion criteria. Most of the studies were assessed to have an overall low risk of bias. In the 2 studies with a moderate risk of bias and 1 study with a high risk of bias, concerns were raised about high rates of crossover and loss to follow-up. The median FI was 4, with FIs ranging from 0 to 14, but half of the studies had an FI of 2 or less. In 4 of the 8 studies, the number of patients lost to follow-up was greater than the FI.

CONCLUSIONS

A systematic survey of hip arthroscopy RCTs resulted in a low FI, indicating that the findings tended to be fragile. A low FI was consistent with findings reported in other orthopaedic and medical literature. Given these results, there is a possibility for findings to be altered by factors such as loss to follow-up, measurement subjectivity, crossover, and biased study design. Results on the fragility of hip arthroscopy RCTs were similar to those reported in general or orthopaedic-specific literature.

LEVEL OF EVIDENCE

Level II, systematic review of Level I and II studies.

The fragility and reverse fragility indices of proximal humerus fracture randomized controlled trials: a systematic review

BACKGROUND

The quality of evidence of the orthopedic literature has been often called into question. The fragility index (FI) has emerged as a means to evaluate the robustness of a significant result. Similarly, reverse fragility index (RFI) can be used for nonsignificant results to evaluate whether one can confidently conclude that there is no difference between groups. The analysis of FI and RFI in proximal humerus fracture (PHF) management is of particular interest, given ongoing controversy regarding optimal management and patient selection. The aim of this study was to report the FI, RFI and quality of the evidence in the proximal humerus fracture literature.

METHODS

A systematic review was conducted based on the PRISMA (preferred reporting items for systematic reviews and meta-analyses) guidelines, which utilized EMBASE, MEDLINE and Cochrane Library databases. Inclusion criteria included randomized controlled clinical trials related to the management of proximal humerus fractures, published from 2000 to 2020 with dichotomous outcome measures and 1:1 allocation. The FI and RFI were calculated by successively changing one nonevent to an event for each outcome measure until the result was made nonsignificant or significant, respectively. The fragility quotient, (FQ), calculated by dividing the FI by the total sample size, was calculated as well.

RESULTS

There were 25 studies that met our criteria with 48 outcome measures recorded. A total of 21 studies had at least one fragile result, with ten studies including a fragile result in the conclusion of the abstract. A total of 31 outcome measures had nonsignificant results and the median RFI was found to be 4, with 71% greater than number of patients lost to follow up. Seventeen outcomes had significant results, with a median FI of 1, with 65% greater than or equal to the number patients lost to follow up. A total of 18 of 25 studies (72%) included a power analysis. In particular, ten studies reported a statistical analysis of complication rates, 90% of which were fragile. The median FQ was found to be 0.037.

CONCLUSIONS

The literature on PHF management is frequently fragile. Outcome measures are often fragile, particularly with regards to comparing complication rates and reoperation rates in treatment arms. Comparing to the studies in other subspecialties PHF RCTs are relatively more fragile and underpowered. Standardized reporting of FI, FQ and RFI can help the reader to reliably draw conclusions based on the fragility of outcome measures.

How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines

Objectives

“Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016” provides guidelines in regard to prompt management and resuscitation of sepsis or septic shock. The study is aimed to assess the robustness of randomized controlled trials (RCTs) that formulate these guidelines in terms of fragility index and reverse fragility index.

Method

RCTs that contributed to these guidelines having parallel two-group design, 1:1 allocation ratio, and at least one dichotomous outcome were included in the study. The median fragility index was calculated for RCTs with significant statistical outcomes, whereas the median reverse fragility index was calculated for RCTs with nonsignificant statistical results.

Results

Hundred RCTs that met the inclusion criteria were analyzed. The median fragility index was 5.5 [95% confidence interval (CI) 1–30] and median reverse fragility index was 13 (95% CI 12.07–16.8) at a p value of 0.05. The median reverse fragility index was 16 (95% CI 10–26) at a p value of 0.01. Most of the RCTs included in this analysis were of good quality, having a median Jadad score of 6.

Conclusion

This analysis found that the surviving sepsis guidelines were based on highly robust RCTs with statistically insignificant results and on some moderately robust RCTs with statistically significant results. RCTs with statistically insignificant results were more robust than RCTs with statistically significant results in regard to these guidelines.

Highlights

The study assessed the robustness of randomized controlled trials (RCTs) that were used to formulate surviving sepsis guidelines. Most RCTs showed statistically nonsignificant results. RCTs with statistically significant results were moderately fragile whereas RCTs with nonsignificant results were more robust.

How to cite this article

Choupoo NS, Das SK, Saikia P, Dey S, Ray S. How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines. Indian J Crit Care Med 2021;25(7):773–779.

Dismantling the Fragility Index: A demonstration of statistical reasoning

The Fragility Index has been introduced as a complement to the P-value to summarize the statistical strength of evidence for a trial's result. The Fragility Index (FI) is defined in trials with two equal treatment group sizes, with a dichotomous or time-to-event outcome, and is calculated as the minimum number of conversions from nonevent to event in the treatment group needed to shift the P-value from Fisher's exact test over the .05 threshold. As the index lacks a well-defined probability motivation, its interpretation is challenging for consumers. We clarify what the FI may be capturing by separately considering two scenarios: (a) what the FI is capturing mathematically when the probability model is correct and (b) how well the FI captures violations of probability model assumptions. By calculating the posterior probability of a treatment effect, we show that when the probability model is correct, the FI inappropriately penalizes small trials for using fewer events than larger trials to achieve the same significance level. The analysis shows that for experiments conducted without bias, the FI promotes an incorrect intuition of probability, which has not been noted elsewhere and must be dispelled. We illustrate shortcomings of the FI's ability to quantify departures from model assumptions and contextualize the FI concept within current debate around the null hypothesis significance testing paradigm. Altogether, the FI creates more confusion than it resolves and does not promote statistical thinking. We recommend against its use. Instead, sensitivity analyses are recommended to quantify and communicate robustness of trial results.