Fragility Index in Cardiovascular Randomized Controlled Trials

A Randomized Trial of Drug Route in Out-of-Hospital Cardiac Arrest

BACKGROUND

In patients with out-of-hospital cardiac arrest, the effectiveness of drugs such as epinephrine is highly time-dependent. An intraosseous route of drug administration may enable more rapid drug administration than an intravenous route; however, its effect on clinical outcomes is uncertain.

METHODS

We conducted a multicenter, open-label, randomized trial across 11 emergency medical systems in the United Kingdom that involved adults in cardiac arrest for whom vascular access for drug administration was needed. Patients were randomly assigned to receive treatment from paramedics by means of an intraosseous-first or intravenous-first vascular access strategy. The primary outcome was survival at 30 days. Key secondary outcomes included any return of spontaneous circulation and favorable neurologic function at hospital discharge (defined by a score of 3 or less on the modified Rankin scale, on which scores range from 0 to 6, with higher scores indicating greater disability). No adjustment for multiplicity was made.

RESULTS

A total of 6082 patients were assigned to a trial group: 3040 to the intraosseous group and 3042 to the intravenous group. At 30 days, 137 of 3030 patients (4.5%) in the intraosseous group and 155 of 3034 (5.1%) in the intravenous group were alive (adjusted odds ratio, 0.94; 95% confidence interval [CI], 0.68 to 1.32; P = 0.74). At the time of hospital discharge, a favorable neurologic outcome was observed in 80 of 2994 patients (2.7%) in the intraosseous group and in 85 of 2986 (2.8%) in the intravenous group (adjusted odds ratio, 0.91; 95% CI, 0.57 to 1.47); a return of spontaneous circulation at any time occurred in 1092 of 3031 patients (36.0%) and in 1186 of 3035 patients (39.1%), respectively (adjusted odds ratio, 0.86; 95% CI, 0.76 to 0.97). During the trial, one adverse event, which occurred in the intraosseous group, was reported.

CONCLUSIONS

Among adults with out-of-hospital cardiac arrest requiring drug therapy, the use of an intraosseous-first vascular access strategy did not result in higher 30-day survival than an intravenous-first strategy. (Funded by the National Institute for Health and Care Research; PARAMEDIC-3 ISRCTN Registry number, ISRCTN14223494.).

Evaluation of the robustness of randomized controlled trials for the treatment modalities of esophageal cancer using the fragility index - a systematic review

BACKGROUND

Esophageal cancer remains a significant global health challenge. Several treatment modalities were explored in randomized controlled trials (RCTs) in recent decades. This study evaluates the robustness of RCTs focusing on esophageal cancer treatment using the fragility index (FI) and reverse fragility index (RFI).

METHODS

A systematic review of RCTs studying different treatment modalities for esophageal cancer from 2000 to 2023 was conducted. The FI and RFI were utilized to gauge the robustness of statistically significant and non-significant outcomes, respectively. The FI represents the minimal number of patient outcomes that would need to alter to overturn a trial's statistical significance, while RFI indicates the minimal changes required to achieve significance in non-significant results.

RESULTS

Out of 4028 studies retrieved, 21 RCTs were included for final analysis. The studies spanned 2001 to 2023 with a mean followup of 66 months (range, 29-108 months) and median number of patients of 194 (range, 45-802). The most common treatment modalities examined in these studies were neoadjuvant chemoradiotherapy (n = 7, 33.3%), neoadjuvant chemotherapy (n = 4, 19.0%), and neoadjuvant immunotherapy (n = 2, 9.5%). Only 5 studies (23.8%) had a statistically significant primary outcome result with a median FI of 6 (IQR, 2.5-8.5). Non-significant primary outcomes were seen in 16 studies (76.2%) with a median RFI of 4 (IQR 1-11) and lost to followup of 0 (IQR 0-4). In the study with the highest FI (10), the FI was lower than the number of patients lost to followup (13).

CONCLUSION

Our findings demonstrate that most RCTs on esophageal cancer treatments did not report significant primary outcomes. The few studies that reported significant results had a low fragility index, suggesting a vulnerability in their findings.

Fragility index for extended prophylaxis following abdominopelvic surgery: A methodological survey

BACKGROUND

Fragility Index (FI) is increasingly used to assess robustness of statistically significant p-values reported in randomized controlled trials (RCTs). FI represents the lowest number of non-events changed to events that would make study findings non-significant. This methodological survey was designed to assess the fragility of the evidence for extended VTEp following major abdominopelvic surgery.

METHODS

MEDLINE, Embase, and CENTRAL were searched from inception to November 2023. RCTs with parallel, double-armed, superiority design comparing extended VTEp for patients undergoing major abdominopelvic surgery to controls with at least one statistically significant dichotomous outcome were included. Walsh et al.'s method of calculating FI was utilized.

RESULTS

After review of 611 citations, 6 RCTs were identified with 12 statistically significant outcomes between groups. The mean number of patients randomized per RCT was 419 (SD 176). The median FI was 1.5 (range: 1-4). The number of patients lost to follow-up was greater than the FI for 10/12 (83.3 ​%) outcomes.

CONCLUSIONS

Statistically significant differences reported in RCTs evaluating extended VTEp following major abdominopelvic surgery are not robust.

How fragile the positive results of Chinese herbal medicine randomized controlled trials on irritable bowel syndrome are?

OBJECTIVE

The fragility index (FI), which is the minimum number of changes in status from "event" to "non-event" resulting in a loss of statistical significance, serves as a significant supplementary indicator for clinical physicians in interpreting clinical trial results and aids in understanding the outcomes of randomized controlled trials (RCTs). In this systematic literature survey, we evaluated the FI for RCTs evaluating Chinese herbal medicine (CHM) for irritable bowel syndrome (IBS), and explored potential associations between study characteristics and the robustness of RCTs.

METHODS

A comprehensive search was conducted in four databases in Chinese and four databases in English from their inception to January 1, 2023. RCTs encompassed 1:1 ratio into two parallel arms and reported at least one binary outcome that demonstrated statistical significance were included. FI was calculated by the iterative reduction of a target outcome event in the treatment group and concomitant subtraction of a non-target event from that group, until positive significance (defined as P < 0.05 by Fisher's exact test) is lost. The lower the FI (minimum 1) of a trial outcome, the more fragile the positive result of the outcome was. Linear regression models were adopted to explore influence factors of the value of FI.

RESULTS

A total of 30 trials from 2 4118 potentially relevant citations were finally included. The median FI of total trials included was 1.5 (interquartile range [IQR], 1-5), and half of the trials (n = 15) had a FI equal to 1. In 12 trials (40%), the total number of participants lost to follow-up surpassed the respective FI. The study also identified that increased FI was significantly associated with no TCM syndrome differentiation for inclusion criteria of the patients, larger total sample size, low risk of bias, and larger numbers of events.

CONCLUSIONS

The majority of CHM IBS RCTs with positive results were found to be fragile. Ensuring adequate sample size, scientifically rigorous study design, proper control of confounding factors, and a quality control calibration for consistency of TCM diagnostic results among clinicians should be addressed to increase the robustness of the RCTs. We recommend reporting the FI as one of the components of sensitivity analysis in future RCTs to facilitate the assessment of the fragility of trials.

The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

BACKGROUND

Evidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.

PURPOSE

To calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).

STUDY DESIGN

Meta-analysis; Level of evidence, 2.

METHODS

A search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.

RESULTS

There were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial's sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).

CONCLUSION

More than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

The fragility index and reverse fragility index of FDA investigational device exemption trials in spinal fusion surgery: a systematic review

PURPOSE

FDA investigational device exemption (IDE) studies are considered a gold standard of assessing safety and efficacy of novel devices through RCTs. The fragility index (FI) has emerged as a means to assess robustness of statistically significant study results and inversely, the reverse fragility index (RFI) for non-significant differences. Previous authors have defined results as fragile if loss to follow up is greater than the FI or RFI. The aim of this study was to assess the FI, RFI, and robustness of data supplied by IDE studies in spinal surgery.

METHODS

This was a systematic review of the literature. Inclusion criteria included randomized controlled trials with dichotomous outcome measures conducted under IDE guidelines between 2000 and 2023. FI and RFI were calculated through successively changing events to non-events until the outcome changed to non-significance or significance, respectively. The fragility quotient (FQ) and reverse fragility quotient (RFQ) were calculated by dividing the FI and RFI, respectively, by the sample size.

RESULTS

Thirty-two studies met inclusion criteria with a total of 40 unique outcome measures; 240 outcomes were analyzed. Twenty-six studies reported 96 statistically significant results. The median FI was 6 (IQR: 3-9.25), and patients lost to follow up was greater than the FI in 99.0% (95/96) of results. The average FQ was 0.027. Thirty studies reported 144 statistically insignificant results and a median RFI of 6 (IQR: 4-8). The average RFQ extrapolated was 0.021, and loss to follow up was greater than the RFI in 98.6% (142/144) of results.

CONCLUSIONS

IDE studies in spine surgery are surprisingly fragile given their reputations, large sample sizes, and intent to establish safety in investigational devices. This study found a median FI and RFI of 6. The number of patients lost to follow-up was greater than FIand RFI in 98.8% (237/240) of reported outcomes. FQ and RFQ tell us that changes of two to three patients per hundred can flip the significance of reported outcomes. This is an important reminder of the limitations of RCTs. Analysis of fragility in future studies may help clarify the strength of the relationship between reported data and their conclusions.

Fragility Index and Fragility Quotient in Statistically Significant Randomized Controlled Trials in Plastic Breast Surgery

Background

The fragility index (FI) was conceived as an adjunct to the P value, signifying the strength of statistically significant results. The index states the minimal number of patients whose outcome must be changed from "event" to "nonevent" for the results to be statistically nonsignificant. The FI was applied in various medical specialties to assess the robustness of results presented in studies. We aim to assess the robustness of statistically significant results in studies on plastic surgery of the breast and determine factors correlated with studies deemed fragile.

Methods

A systematic literature review of PubMed databases using designated keywords was performed. Background characteristics were extracted from the studies, alongside the significance of outcomes. FI and fragility quotient were calculated for each analyzed outcome and correlated with various baseline characteristics.

Results

FI and fragility quotient were both significantly correlated only with the P value of the analyzed outcomes. However, grouping studies based on the P value into three categories did not demonstrate a difference in FI. Comparisons of fragile and robust studies did not demonstrate a statistically significant change in terms of baseline variables, except for the mean P value of the outcome.

Conclusion

Statistically significant results of randomized controlled trials in plastic surgery of the breast suffer from extensive fragility, and researchers should critically implement their conclusions in their practice.

The fragility index in randomised controlled trials of interventions for aneurysmal subarachnoid haemorrhage: A systematic review

Background

Fragility analysis supplements the p-value and risk of bias assessment in the interpretation of results of randomised controlled trials. In this systematic review we determine the fragility index (FI) and fragility quotient (FQ) of randomised trials in aneurysmal subarachnoid haemorrhage.

Methods

This is a systematic review registered with PROSPERO (ID: CRD42020173604). Randomised controlled trials in adults with aneurysmal subarachnoid haemorrhage were analysed if they reported a statistically significant primary outcome of mortality, function (e.g. modified Rankin Scale), vasospasm or delayed neurological deterioration.

Results

We identified 4825 records with 18 randomised trials selected for analysis. The median fragility index was 2.5 (inter-quartile range 0.25-5) and the median fragility quotient was 0.015 (IQR 0.02-0.039). Five of 20 trial outcomes (25%) had a fragility index of 0. In seven trials (39.0%), the number of participants lost to follow-up was greater than or equal to the fragility index. Only 16.7% of trials are at low risk of bias.

Conclusion

Randomised controlled trial evidence supporting management of aneurysmal subarachnoid haemorrhage is weaker than indicated by conventional analysis using p-values alone. Increased use of fragility analysis by clinicians and researchers could improve the translation of evidence to practice.

Robustness of Longitudinal Safety and Efficacy After Paclitaxel-Based Endovascular Therapy for Treatment of Femoro-Popliteal Artery Occlusive Disease: An Updated Systematic Review and Meta-Analysis of Randomized Controlled Trials

BACKGROUND

The aims of this study were: i) to assess fragility indices (FIs) of individual randomized controlled trials (RCTs) that compared paclitaxel-based drug-coated balloons (DCBs) or drug-eluting stents (DESs) versus standard endovascular devices, and ii) to meta-analyze mid-term and long-term safety and efficacy outcomes from available RCT data while also estimating the FI of pooled results.

METHODS

This systematic review has been registered in the PROSPERO public database (CRD42022304326 http://www.crd.york.ac.uk/PROSPERO). A query of PubMed (Medline), EMBASE (Excerpta Medical Database), Scopus, and CENTRAL (Cochrane Central Register of Controlled Trials) databases was performed to identify eligible RCTs. Rates of primary patency (PP) and target lesion revascularization (TLR) were assessed as efficacy outcomes, while lower limb amputation (LLA) consisting of major amputation that is. below or above the knee and all-cause mortality were estimated as safety outcomes. All outcomes were pooled with a random effects model to account for any clinical and study design heterogeneity. The analyses were performed by dividing the RCTs according to their maximal follow-up length (mid-term was defined as results up to 2-3 years, while long-term was defined as results up to 4-5 years). For each individual outcome, the FI and reverse fragility index (RFI) were calculated according to whether the outcome results were statistically significant or not, respectively. The fragility quotient (FQ) and reverse fragility quotient (RFQ), which are the FI or RFI divided by the sample size, were also calculated.

RESULTS

A total of 2,337 patients were included in the systematic review and meta-analysis. There were 2 RCTs examining DES devices and 14 RCTs evaluating different DCBs. For efficacy outcomes, there was evidence that paclitaxel-based endovascular therapy increased the PP rate and reduced the TLR rate at mid-term, with a calculated pooled risk ratio (RR) of 1.66 for patency (95% CI, 1.55-1.86; P < 0.001), with a corresponding number needed-to-treat (NNT) of 3 patients (95% CI, 2.9-3.8) and RR of 0.44 for TLR (95% CI, 0.35-0.54; P = 0.027), respectively. Similarly, there was evidence that paclitaxel-based endovascular therapy both increased PP and decreased TLR rates at long-term, with calculated pooled RR values of 1.73 (95% CI, 1.12-2.61; P = 0.004) and 0.53 (95% CI, 0.45-0.62; P = 0.82), respectively. For safety outcomes, there was evidence that paclitaxel-based endovascular therapy increased all-cause mortality at mid-term, with a calculated pooled RR of 2.05 (95% CI, 1.21-3.24). However, there was no difference between treatment arms in LLA at mid-term (95% CI, 0.1-2.7; P = 0.68). Similarly, neither all-cause mortality nor LLA at long-term differed between treatment arms, with a calculated pooled RR of 0.66, 1.02 (95% CI, 0.31-3.42) and 1.02 (95% CI, 0.30-5.21; P = 0.22), respectively. The pooled estimates of PP at mid-term were robust (FI = 28 and FQ = 1.9%) as were pooled rates of TLR (FI = 18 and FQ = 0.9%). However, when safety outcomes were analyzed, the robustness of the meta-analysis decreased significantly. In fact, the relationship between the use of paclitaxel-coated devices and all-cause mortality at mid-term showed very low robustness (FI = 4 and FQ = 0.2%). At 5 years, only the benefit of paclitaxel-based devices to reduce TLR remained robust, with an FI of 32 and an FQ of 3.1%.

CONCLUSIONS

The data supporting clinical efficacy endpoints of RCTs that examined paclitaxel-based devices in the treatment of femoral-popliteal arterial occlusive disease were robust; however, the pooled safety endpoints were highly fragile and prone to bias due to loss of patient follow-up in the original studies. These findings should be considered in the ongoing debate concerning the safety of paclitaxel-based devices.

Fragility index analysis for randomized controlled trials of approved biologicals and small molecule drugs in inflammatory bowel diseases

INTRODUCTION

Biologics and small molecules have been increasingly applied in Crohn's disease (CD) and ulcerative colitis (UC). But the robustness of their trials has not been evaluated.

METHODS

We initially collected all the approved biologics or small molecules for CD or UC up to December 1, 2022. Databases were then queried by keywords in chemical name and CD or UC. Randomized controlled trials (RCTs) in the two-arm, 1:1 design were included. Fragility index (FI) and fragility quotient (FQ) were subsequently calculated.

RESULTS

We included twenty-eight RCTs, including nine pivotal trials listed in approval labels, nineteen non-pivotal trials not included in the labels. The median sample size was 99 [IQR, 60-262] and the median number of loss-of-follow-up (LFU) was 14 [IQR, 8-43]. Pivotal trials in the labels had the median FI of 8 [IQR, 4-14, n = 6] that was marginally higher than non-pivotal trials (3 [IQR, 2-4], p = 0.08). The median FQ was 0.0330 [IQR, 0.1220-0.0466] and 0.0310 [IQR, 0.0129-0.0540] for pivotal and non-pivotal trials, respectively (p = 1.0). The sample size and FI were significantly correlated (Spearman correlation coefficient [r] = 0.56, 95 %CI 0.21-0.78, p = 0.003). The number of total events was also significantly correlated with FI (r = 0.53, 95 %CI 0.17-0.77, p = 0.006). Study p-values were significantly associated with FI (p = 0.01): trials with p-values < 0.001 had the highest median FI of 10 [IQR, 6-17]. No factor was found strongly correlated with FQ.

CONCLUSION

Results from trials assessing administration-approved biologics or small molecules for treating CD or UC were vulnerable to small changes by measuring FI or FQ. Pivotal studies contributing to regulatory approvals exhibited a relatively higher degree of resilience compared to non-pivotal trials.

Assessing the robustness of positive vascular surgery randomized controlled trials using their fragility index

OBJECTIVE

The fragility index (FI) measures the robustness of statistically significant findings in randomized controlled trials (RCTs) by quantifying the minimum number of event conversions required to reverse a dichotomous outcome's statistical significance. In vascular surgery, many clinical guidelines and critical decision-making points are informed by a handful of key RCTs, especially regarding open surgical versus endovascular treatment. The objective of this study is to evaluate the FI of RCTs with statistically significant primary outcomes that compared open vs endovascular surgery in vascular surgery.

METHODS

In this meta-epidemiological study and systematic review, MEDLINE, Embase, and CENTRAL were searched for RCTs comparing open versus endovascular treatments for abdominal aortic aneurysms, carotid artery stenosis, and peripheral arterial disease to December 2022. RCTs with statistically significant primary outcomes were included. Data screening and extraction were conducted in duplicate. The FI was calculated by adding an event to the group with the smaller number of events while subtracting a nonevent to the same group until Fisher's exact test produced a nonstatistically significant result. The primary outcome was the FI and proportion of outcomes where the loss to follow-up was greater than the FI. The secondary outcomes assessed the relationship of the FI to disease state, presence of commercial funding, and study design.

RESULTS

Overall, 5133 articles were captured in the initial search with 21 RCTs reporting 23 different primary outcomes being included in the final analysis. The median FI (first quartile, third quartile) was 3 (3, 20) with 16 outcomes (70%) reporting a loss to follow-up greater than its FI. Mann-Whitney U test revealed that commercially funded RCTs and composite outcomes had greater FIs (median, 20.0 [5.5, 24.5] vs median, 3.0 [2.0, 5.5], P = .035; median, 21 [8, 38] vs median, 3.0 [2.0, 8.5], P = .01, respectively). The FI did not vary between disease states (P = .285) or between index and follow-up trials (P = .147). There were significant correlations between the FI and P values (Pearson r = 0.90; 95% confidence interval, 0.77-0.96), and the number of events (r = 0.82; 95% confidence interval, 0.48-0.97).

CONCLUSIONS

A small number of event conversions (median, 3) are needed to alter the statistical significance of primary outcomes in vascular surgery RCTs evaluating open surgical and endovascular treatments. Most studies had loss to follow-up greater than its FI, which can call into question trial results, and commercially funded studies had a greater FI. The FI and these findings should be considered in future trial design in vascular surgery.

The statistical fragility of intramedullary reaming in tibial nail fixation: a systematic review

PURPOSE

To report the statistical stability of prospective clinical trials evaluating the effect of intramedullary reaming on rates of non-union in tibial fractures through calculation of the fragility metrics for non-union rates and all other dichotomous outcomes.

METHODS

Literature search was conducted for prospective clinical trials evaluating the effect of intramedullary reaming on non-union rates in tibial nailing. All dichotomous outcomes were extracted from the manuscripts. The fragility index (FI) and reverse fragility index (RFI) were calculated by determining the number of event reversals required for a statistically significant outcome to lose significance and vice-versa. The fragility quotient (FQ) and reverse fragility quotient (RFQ) were calculated by dividing the FI or RFI by the sample size, respectively. Outcomes were defined as "fragile" if the FI or RFI was found to be less than or equal to the number of patients lost to follow-up.

RESULTS

Literature search identified 579 results which produced ten studies meeting the criteria for review. There were 111 outcomes identified for analysis, of which 89 (80%) exhibited statistical fragility. For reported outcomes across the studies the median and mean FI was 2, the median FQ was 0.019, the mean FQ was 0.030, the median RFI was 4, the mean RFI was 3.95, the median RFQ was 0.045, and the mean RFQ was 0.030. Four studies reported outcomes which were found to have an FI of 0.

CONCLUSIONS

The studies evaluating the effect of intramedullary reaming on tibial nail fixation demonstrate considerable fragility. On average, two event reversals for significant findings, and four event reversals for insignificant findings are sufficient to alter statistical significance.

LEVEL OF EVIDENCE

Level II, systematic review of Level I and Level II studies.

An analysis of randomized controlled trials on anal fistula conducted between 2000 and 2020 based on the Fragility Index and Reverse Fragility Index

AIM

The aim of this work was to evaluate the robustness of randomized controlled trials (RCTs) on anal fistula management using the news tools of Fragility Index (FI), Reverse Fragility Index (RFI) and their corresponding fragility quotients.

METHOD

A systematic search was conducted based on the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines which utilized MEDLINE, EMBASE, Cochrane Library and Web of Science databases. Inclusion criteria included RCTs related to the management of anal fistula published from 2000 to 2022 with dichotomous outcomes measures and 1:1 allocation. Calculation of FI and RFI was performed by creating 2 × 2 contingency tables by successively changing one nonevent to an event for each outcome measure until the result was made nonsignificant or significant, respectively. The Fragility Quotients were calculated by dividing the FI or RFI by the total sample size. Fragile results were defined as those with a FI or RFI equal to or less than the number of patients lost to follow-up. Additionally, those with a FI or RFI less than 3 were also considered fragile. Studies were considered extremely fragile if FI was ≤1 or FQ was ≤0.01.

RESULTS

There were 36 RCTs that met our criteria, with 3223 patients. Among these, 19 (53%) were positive RCTs (p < 0.005) and 17 (47%) were negative RCTs (p > 0.05). The median FI was 2 (0-5). The analysis by categorical subgroup showed a strong correlation between FI and the p-value (p = 0.000) and the number of events (p = 0.011). The median RFI was 5 (3.5-9.5) and the subgroup analysis showed a strong correlation between RFI and the p-value (p = 0.000), sample size (0.021) and number needed to treat/number needed to harm (0.000). We considered 63.2% of positive RCTs to be fragile and 35.3% of negative RCTs.

CONCLUSIONS

In the present study we demonstrated the lack of robustness of study findings in published RCTs in the field of anal fistula.

To guide future practice, perinatal trials should be much larger, simpler and less fragile with close to 100% ascertainment of mortality and other key outcomes

The Australian Placental Transfusion Study (APTS) randomised 1,634 fetuses to delayed (≥60 s) versus immediate (≤10 s) clamping of the umbilical cord. Systematic reviews with meta-analyses, including this and similar trials, show that delaying clamping in preterm infants reduces mortality and need for blood transfusions. Amongst 1,531 infants in APTS followed up at two years, aiming to delay clamping for 60 s or more reduced the relative risk of the primary composite outcome of death or disability by 17% (p = 0.01). However, this result is fragile because nominal statistical significance (p < 0.05) would be abolished by only 2 patients switching from a non-event to an event, and the primary composite outcome was missing in 112 patients (7%). To achieve more robust evidence, any future trials should emulate the large, simple trials co-ordinated from Oxford which reliably identified moderate, incremental improvements in mortality in tens of thousands of participants, with <1% missing data. Those who fund, regulate, and conduct trials that aim to change practice should repay the trust of those who consent to participate by doing everything possible to minimise missing data for key outcomes.

Pulmonary arterial hypertension trials put to the test: Using the fragility index to assess trials robustness

BACKGROUND

Randomized clinical trials (RCTs) are considered the gold standard for evidence-based medicine. The Fragility Index (FI) is a tool to assess the robustness of RCT results. FI was validated for dichotomous outcomes and recent work expanded its use to continuous outcomes. Studying the robustness of RCTs in Pulmonary Arterial Hypertension (PAH) treatments is crucial due to the severity and mortality risks associated with this rare condition.

OBJECTIVES

Analyze FI and Fragility quotient (FQ) of significant primary outcomes in PAH RCTs and study FI correlation with sample size and journal impact factor.

METHODS

FI and FQ calculation followed by Spearman correlation to assess the correlation between FI and sample size, and FI and impact factor.

RESULTS

The median sample size of the 21 trials was 202 patients (IQR 106-267), with 6 trials reporting primary outcomes as dichotomous and 15 reporting continuous primary outcomes. The median FI was 10 (IQR 3-20), and the median FQ was 0.044 (0.026-0.097). A moderate correlation was found between FI and sample size, with r = 0.56; P = 0.008 and FI and journal impact factor (r=0.50; P=0.019). The FI for continuous outcomes was similar to that for dichotomous outcomes.

CONCLUSIONS

This study represents the first analysis of the FI and FQ of PAH treatment RCTs, and expands the use of FI to continuous outcomes in this context. The moderate correlation between FI and sample size suggests that increasing sample size alone is partially correlated to a higher FI. The similarity between FI for continuous and dichotomous outcomes supports the broader use of FI in PAH RCTs.

Assessing the robustness of COVID-19 vaccine efficacy trials: systematic review and meta-analysis, January 2023

BackgroundVaccines play a crucial role in the response to COVID-19 and their efficacy is thus of great importance.AimTo assess the robustness of COVID-19 vaccine efficacy (VE) trial results using the fragility index (FI) and fragility quotient (FQ) methodology.MethodsWe conducted a Cochrane and PRISMA-compliant systematic review and meta-analysis of COVID-19 VE trials published worldwide until 22 January 2023. We calculated the FI and FQ for all included studies and assessed their associations with selected trial characteristics using Wilcoxon rank sum tests and Kruskal-Wallis H tests. Spearman correlation coefficients and scatter plots were used to quantify the strength of correlation of FIs and FQs with trial characteristics.ResultsOf 6,032 screened records, we included 40 trials with 54 primary outcomes, comprising 909,404 participants with a median sample size per outcome of 13,993 (interquartile range (IQR): 8,534-25,519). The median FI and FQ was 62 (IQR: 22-123) and 0.50% (IQR: 0.24-0.92), respectively. FIs were positively associated with sample size (p < 0.001), and FQs were positively associated with type of blinding (p = 0.023). The Spearman correlation coefficient for FI with sample size was moderately strong (0.607), and weakly positive for FI and FQ with VE (0.138 and 0.161, respectively).ConclusionsThis was the largest study on trial robustness to date. Robustness of COVID-19 VE trials increased with sample size and varied considerably across several other important trial characteristics. The FI and FQ are valuable complementary parameters for the interpretation of trial results and should be reported alongside established trial outcome measures.

The reproducibility of interventional radiology randomized controlled trials and external validation of a classification system

PURPOSE

The fragility index (FI) measures the robustness of randomized controlled trials (RCTs). It complements the P value by taking into account the number of outcome events. In this study, the authors measured the FI for major interventional radiology RCTs.

METHODS

Interventional radiology RCTs published between January 2010 and December 2022 relating to trans-jugular intrahepatic portosystemic shunt, trans-arterial chemoembolization, needle biopsy, angiography, angioplasty, thrombolysis, and nephrostomy tube insertion were analyzed to measure the FI and robustness of the studies.

RESULTS

A total of 34 RCTs were included. The median FI of those studies was 4.5 (range 1-68). Seven trials (20.6%) had a number of patients lost to follow-up that was higher than their FI, and 15 (44.1%) had a FI of 1-3.

CONCLUSION

The median FI, and hence the reproducibility of interventional radiology RCTs, is low compared to other medical fields, with some having a FI of 1, which should be interrupted cautiously.

Assessment of 30 Years of Randomized Controlled Trials in The American Journal of Sports Medicine: 1990-2020

Background

Randomized controlled trials (RCTs) stand atop the evidence-based hierarchy of study designs for their ability to arrive at results with the lowest risk of bias. Even for RCTs, however, critical appraisal is essential before applying results to clinical practice.

Purpose

To analyze the quality of reporting of RCTs published in The American Journal of Sports Medicine (AJSM) from 1990 to 2020 and to identify trends over time and areas of improvement for future trials.

Study Design

Systematic review; Level of evidence, 1.

Methods

We queried the AJSM database for RCTs published between January 1990 and December 2020. Data pertaining to study characteristics were recorded. Quality assessments were conducted using the Detsky quality-of-reporting index and the modified Cochrane risk-of-bias (mROB) tool. Univariate and multivariable models were generated to establish factors with associations to study quality. The Fragility Index was calculated for eligible studies.

Results

A total of 277 RCTs were identified with a median sample size of 70 patients. A total of 19 RCTs were published between 1990 and 2000 (t₁); 82 RCTs between 2001 and 2010 (t₂); and 176 RCTs between 2011 and 2020 (t₃). From t₁ to t₃, significant increases were observed in the overall mean-transformed Detsky score (from 68.2% ± 9.8% to 87.4% ± 10.2%, respectively; P < .001) and mROB score (from 4.7 ± 1.6 to 6.9 ± 1.6, respectively; P < .001). Multivariable regression analysis revealed that trials with follow-up periods of <5 years clearly stated primary outcomes, and a focus on the elbow, shoulder, or knee were associated with higher mean-transformed Detsky and mROB scores. The median Fragility Index was 2 (interquartile range, 0-5) for trials with statistically significant. Studies with small sample sizes (<100 patients) were more likely to have low Fragility Index scores and less likely to have a statistically significant finding in any outcome.

Conclusion

The quantity and quality of published RCTs published in AJSM increased over the past 3 decades. However, single-center trials with small sample sizes were prone to fragile results.

Thresholds for interpreting the fragility index derived from sample of randomised controlled trials in cardiology: a meta-epidemiologic study

The fragility index (FI) was proposed as a simplified way to communicate robustness of statistically significant results and their susceptibility to a change of a handful number of events. While this index is intuitive, it is not anchored by a cut-off or a guide for interpretation. We identified cardiovascular trials published in six high impact journals from 2007 to 2021 (500 or more participants and a dichotomous statistically significant primary outcome). We estimated area under curve (AUC) to determine FI value that best predicts whether the treatment effect was precise, defined as adequately powered for a plausible relative risk reduction (RRR) of 25% or 30% or having a CI that is sufficiently narrow to exclude a risk reduction that is too small (close to the null, <0.05). The median FI of 201 included cardiovascular trials was 13 (range 1-172). FI exceeded the number of patients lost to follow-up in 46/201 (22.89%) trials. FI values of 19 and 22 predicted that trials would be precise (powered for RRR of 30% and 25%; respectively, combined with CI that excluded risk reduction <0.05). AUC for meeting these precision criteria was 0.90 (0.86-0.94). In conclusion, FI values that range 19-22 may meet various definitions of precision and can be used as a rule of thumb to suggest that a treatment effect is likely precise and less susceptible to random error. The number of patients lost to follow-up should be presented alongside FI to better illustrate fragility.

The fragility of significant results from randomized controlled trials in esophageal surgeries

While randomized controlled trials (RCTs) are regarded as one of the highest forms of clinical research, the robustness of their P values can be difficult to ascertain. Defined as the minimum number of patients in a study arm that would need to be changed from a non-event to event for the findings to lose significance, the Fragility Index is a method for evaluating results from these trials. This study aims to calculate the Fragility Index for trials evaluating perioperative esophagectomy-related interventions to determine the strength of RCTs in this field. MEDLINE and EMBASE were searched for RCTs related to esophagectomy that reported a significant dichotomous outcome. Two reviewers independently screened articles and performed the data extractions with risk of bias assessment. The Fragility Index was calculated using a two-tailed Fisher's exact test. Bivariate correlation was conducted to evaluate associations between the Fragility Index and study characteristics. 41 RCTs were included, and the median sample size was 80 patients [Interquartile range (IQR) 60-161]. Of the included outcomes, 29 (71%) were primary, and 12 (29%) were secondary. The median Fragility Index was 1 (IQR 1-3), meaning that by changing one patient from a non-event to event, the results would become non-significant. Fragility Index was correlated with P value, number of events, and journal impact factor. The RCTs related to esophagectomy did not prove to be robust, as the significance of their results could be changed by altering the outcome status of a handful of patients in one study arm.

Robustness of Significant Dichotomous Outcomes in Randomized Controlled Trials in the Treatment of Patients with COVID-19: A Systematic Analysis

Purpose

Significant results of randomized controlled trials (RCTs) should be properly weighed. This study adopted fragility index (FI) to evaluate the robustness of significant dichotomous outcomes from RCTs on coronavirus disease 2019 (COVID-19) treatment.

Materials and methods

ClinicalTrials.gov and PubMed were searched from inception to July 31, 2021. FIs were calculated and their distribution was depicted. FI's categorical influential factors were analyzed. Spearman correlation coefficient (r _s) was reported for the relationship between FI and the continuous characteristics of RCTs.

Results

Fifty RCTs with 120 outcomes in 7869 patients were included. The FI distribution was abnormal with median 3 (interquartile range 1-7, P = 0.0001). The FIs and robustness were affected by the outcomes of interest, various patient populations, and interventions (T = 18.215,16.667, 23.107; P = 0.02,0.0001, 0.001, respectively). A cubic relationship between the FIs and absolute difference of events between groups with R square of 0.848 (T = 215.828, P = 0.0001, R square = 0.865) was observed. A strong negative logarithmic relationship existed between FI and the P value with R square = - 0.834.

Conclusion

The robustness of significant dichotomous outcomes of COVID-19 treatments was fragile and affected by the outcomes of interest, patients, interventions, P value, and absolute difference of events between the groups. FI was an useful quantitative metric for the binary significant outcomes on COVID-19 treatments.

Registration

PROSPERO (CRD42021272455).

Supplementary Information

The online version contains supplementary material available at 10.1007/s44231-022-00027-y.

Left Atrial Appendage Occlusion: Current Advances and Remaining Challenges

The field of left atrial appendage occlusion is rapidly evolving. However, several issues remain including the limited randomized efficacy data, peri-device leak, device-related thrombus, and the ongoing refinement of procedural techniques. In this article, we provide a contemporary overview of left atrial appendage occlusion focusing on 4 key remaining challenges: efficacy data, peri-device leak, device-related thrombus, and procedural optimization.

Randomized Controlled Trials on Renin Angiotensin Aldosterone System Inhibitors in Chronic Kidney Disease Stages 3-5: Are They Robust? A Fragility Index Analysis

Inhibition of the renin-angiotensin-aldosterone system (RAAS) is broadly recommended in many nephrological guidelines to prevent chronic kidney disease (CKD) progression. This work aimed to analyze the robustness of randomized controlled trials (RCTs) investigating the renal and cardiovascular outcomes in CKD stages 3-5 patients treated with RAAS inhibitors (RAASi). We searched for RCTs in MEDLINE (PubMed), EMBASE databases, and the Cochrane register. Fragility indexes (FIs) for every primary and secondary outcome were calculated according to Walsh et al., who first described this novel metric, suggesting 8 as the cut-off to consider a study robust. Spearman coefficient was calculated to correlate FI to p value and sample size of statistically significant primary and secondary outcomes. Twenty-two studies met the inclusion criteria, including 80,455 patients. Sample size considerably varied among the studies (median: 1693.5, range: 73-17,276). The median follow-up was 38 months (range 24-58). The overall median of both primary and secondary outcomes was 0 (range 0-117 and range 0-55, respectively). The median of FI for primary and secondary outcomes with a p value lower than 0.05 was 6 (range: 1-117) and 7.5 (range: 1-55), respectively. The medians of the FI for primary outcomes with a p value lower than 0.05 in CKD and no CKD patients were 5.5 (range 1-117) and 22 (range 1-80), respectively. Only a few RCTs have been shown to be robust. Our analysis underlined the need for further research with appropriate sample sizes and study design to explore the real potentialities of RAASi in the progression of CKD.

Robustness of outcomes in trials evaluating sodium-glucose co-transporter 2 inhibitors for heart failure

AIMS

Recent trials have evaluated sodium-glucose co-transporter 2 inhibitors in patients with heart failure (HF). We sought to assess the robustness of findings from these trials using the fragility index (FI).

METHODS AND RESULTS

Fragility index is defined as the minimum number of patients that must be moved from the 'non-event' to the 'event' group to turn a statistically significant result to non-significant. In addition to FI, fragility quotient [(FQ); FI divided by the sample size] was calculated to assess the proportion of events that must be moved to change the significance. For statistically non-significant outcomes, reverse fragility index (RFI) and reverse fragility quotient (RFQ) were calculated. Robustness of findings after pooling data from all three trials was also assessed. A robust reduction in first HF hospitalization or cardiovascular mortality was seen with dapagliflozin (FI = 62 and FQ = 0.013), empagliflozin (FI = 50 and FQ = 0.013), and sotagliflozin (FI = 60 and FQ = 0.049). Dapagliflozin nominally improved all-cause and cardiovascular mortality, with modest FI (n = 8 and 5) and FQ (0.002 and 0.001). Empagliflozin and sotagliflozin did not demonstrate statistically significant reductions in all-cause mortality, with modest RFI (empagliflozin: RFI = 26 and RFQ = 0.007; sotagliflozin: RFI = 6 and RFQ = 0.005). A similar trend was seen with cardiovascular mortality (empagliflozin: RFI = 24 and RFQ = 0.006; sotagliflozin: RFI = 7 and RFQ = 0.006). Upon meta-analysis, the result for first HF hospitalization or cardiovascular mortality was robust (FI = 95 and FQ = 0.010). The reductions in all-cause (FI = 12 and FQ = 0.001) and cardiovascular mortality (FI = 9 and FQ = 0.001), while statistically significant, were fragile.

CONCLUSION

Improvement in the composite outcome of first HF hospitalization or cardiovascular death was highly concordant and robust across sodium-glucose co-transporter 2 inhibitor trials. In contrast, secondary endpoints of all-cause and cardiovascular mortality were statistically fragile, underscoring the need to power trials for mortality to fully understand the benefit of therapies on fatal events.

Analysis of Robustness of the Landmark Cardiovascular Outcome Trials of Antidiabetic Drugs - A Systematic Review

BACKGROUND

It is essential for randomized controlled trials (RCTs) to report results in a comprehensive manner. Hence, it is necessary to assess the robustness of the trials with statistically significant and as well as non-significant results. Robustness can be evaluated using fragility index (FI), while reverse fragility index (RFI) can be used for trials with statistically significant as well as non-significant results. The primary aim of this study was to calculate FI and RFI for cardiovascular outcome trials (CVOT).

MATERIALS AND METHODS

PubMed/MEDLINE was searched to identify all RCTs of antidiabetic drugs where the primary objective was to evaluate the cardiovascular outcomes. We recorded the trial characteristics of each CVOT trial. The FI, RFI, fragility quotient (FQ), and reverse fragility quotient (RFQ) were calculated to evaluate the robustness of the trials. Spearman rank correlation test was used for correlation.

FINDINGS

A total of 889 studies were identified and 24 RCTs were included. Among the 24 trials, 12 (50%) trials achieved statistical significance. The median FI and RFI were 29 (4-12) and 22.5 (1-37) for trials with statistically significant and non-significant results. The median FQ and RFQ were 0.0075 (0.002-0.013) and 0.0003 (0.0001-0.004) for trials with statistically significant and non-significant results. The hazard ratio, p-value, and NNT-B had a strong negative relationship with FI.

INTERPRETATION

Our study showed that half of the trials showing the superiority of cardioprotective benefits have favourable FI. The trials that failed to show superiority also have a reasonable RFI indicating the robustness of these trials. However, the results of the trials where patients lost to follow- up exceed the FI of that trial demands caution during interpretation.

The fragility of statistically significant results in gynecologic surgery: A systematic review

OBJECTIVE

To use the fragility index (FI) to evaluate the robustness of gynaecologic surgery trials that report statistically significant results. The FI defines the minimum number of patients who must have an alternative outcome to alter statistical significance.

DATA SOURCES

We searched MEDLINE, Web of Science, Embase, and ClinicalTrials.gov from 2011 to 2021 to identify gynaecologic surgery randomized controlled trials (RCTs).

STUDY SELECTION

A total of 4775 trials were screened for eligibility. All included studies evaluated benign gynaecologic surgery interventions or peri-operative medical interventions. Only two-arm RCTs with statistically significant dichotomous primary outcomes were included. Ninety-three trials were ultimately included for analysis.

DATA EXTRACTION AND SYNTHESIS

Data from the included studies, including sample size, loss to follow-up, and number of events, were recorded. The FI of each study was calculated using a predefined technique. The overall FI and FIs by subgroup (clinical subspecialty, country of origin, and statistical test used) are reported as medians alongside their interquartile ranges (IQRs). The Kruskal-Wallis test was applied to find possible statistically significant relationships between FI and the nominal subgroups. Among this cohort, the median FI was 3 (IQR 1-7). The FI was 0 in 13 trials (14%), and in 39 trials (42%), the number of patients lost to follow-up was greater than the FI. The median FI within clinical subspecialty groups (general gynaecology, anaesthesia, urogynaecology, and fertility) did not differ (P = 0.122).

CONCLUSION

Statistically significant results of RCTs in gynaecologic surgery are fragile, suggesting that clinicians should interpret results with caution. This is particularly true when the number of patients lost to follow-up is greater than the FI. The FI serves as a quality metric that can be used to evaluate robustness of results when applying the outcomes of RCTs to clinical practice or guideline development.

Fragility Index as a Measure of Randomized Clinical Trial Quality in Adult Reconstruction: A Systematic Review

Background

The Fragility Index (FI) and Reverse Fragility Index are powerful tools to supplement the P value in evaluation of randomized clinical trial (RCT) outcomes. These metrics are defined as the number of patients needed to change the significance level of an outcome. The purpose of this study was to calculate these metrics for published RCTs in total joint arthroplasty (TJA).

Methods

We performed a systematic review of RCTs in TJA over the last decade. For each study, we calculated the FI (for statistically significant outcomes) or Reverse Fragility Index (for nonstatistically significant outcomes) for all dichotomous, categorical outcomes. We also used the Pearson correlation coefficient to evaluate publication-level variables.

Results

We included 104 studies with 473 outcomes; 92 were significant, and 381 were nonstatistically significant. The median FI was 6 overall and 4 and 7 for significant and nonsignificant outcomes, respectively. There was a positive correlation between FI and sample size (R = 0.14, P = .002) and between FI and P values (R = 0.197, P = .000012).

Conclusions

This study is the largest evaluation of FI in orthopedics literature to date. We found a median FI that was comparable to or higher than FIs calculated in other orthopedic subspecialties. Although the mean and median FIs were greater than the 2 recommended by the American Academy of Orthopaedic Surgeons Clinical Practice Guidelines to demonstrate strong evidence, a large percentage of studies have an FI < 2. This suggests that the TJA literature is on par or slightly better than other subspecialties, but improvements must be made.

Level of Evidence

Level I; Systematic Review.

Fragility of randomized trials supporting cancer drug approvals stratified by approval pathway and review designations

BACKGROUND

It has been suggested that the results from fragile trials are less likely to translate into benefit in routine clinical practice.

METHODS

We searched the Food and Drug Administration (FDA) archives to identify drug approvals for solid organ malignancies between 2010 and 2019. We calculated the Fragility Index (FI) supporting each approval, using methods to account for time-to-event. We compared FI and trial and approval characteristics using Mann-Whitney U and Kruskal-Wallis test. Using logistic regression, we examined study characteristics associated with withdrawal of consent or lost to follow-up (WCLFU) exceeding the calculated FI.

RESULTS

The median FI among 125 included studies was 23 (range 1-322). The FI was ≤10 in 35 studies (28%), 11-20 in 21 (17%), and >20 in 69 (55%). The median FI/Nexp was 7.7% (range 0.1-51.7%). The median FI was significantly lower among approvals processed through the accelerated vs regular pathway (5.5 vs 25, p = 0.001), but there was no difference in median FI/Nexp. The WCLFU exceeded FI in 42% of studies. Overall survival endpoints were more likely to have a WCLFU exceeding FI (OR 3.16, p = 0.003). WCLFU exceeding FI was also associated with a lesser magnitude of effect (median HR 0.69 vs 0.55, p < 0.001). In a sensitivity analysis including only studies with 1:1 randomization, 51% of studies had WCLFU >FI.

CONCLUSION

The median FI among all trials was 23, and WCLFU exceeded FI in 42%. Comparative trials in solid tumors supporting approval through the accelerated pathway are more fragile compared to trials approved through the regular pathway, an observation likely explained by a lower sample size in the experimental arm.

Robustness and pragmatism of the evidence supporting the European Society for Medical Oncology guidelines for the diagnosis, treatment, and follow-up of follicular lymphoma

Background: Results of randomized clinical trials may not be entirely applicable to clinical practice. The present manuscript aims to explore the pragmatism and robustness of the evidence that supports the European Society for Medical Oncology (ESMO) follicular lymphoma (FL) guidelines.Methods & design: Analysis of all trials used to support positive, therapeutic, oncological recommendations in the 2020 ESMO FL guidelines. Predefined data points were extracted from each trial. Pragmatism was assessed by means of the PRECIS-2 tool, the difference in overall survival in the interventions compared and the source of funding. Robustness was assessed by means of the fragility index and the p value.Results: 28 trials were included. The full protocol or a protocol summary was provided for 12 (43%). Based on the PRECIS-2 domains, trials were considered pragmatic in organization, analysis and flexibility and explanatory in eligibility. Robustness was high, with 4/24 (17%) trials with p values between 0.05 and 0.005 and a median fragility index of 18.Conclusions: Results of trials to support ESMO recommendations in FL were robust. Pragmatism was high in some domains but modest to low in others and the pattern was similar across trials. Transparency in the publication of trial protocols was suboptimal.

Fragility index of urological literature regarding medical expulsive treatment

INTRODUCTION

The role of medical expulsive treatment (MET) is controversial. Fragility index is an additional metric to assess randomized controlled trials (RCTs) outcome validity and indicates how many patients would be required to convert a trial from being statistically significant, to not significant. The larger is the FI, the better the trial's data. The aim of this study is to assess FI of RCTs regarding MET for ureteral stones.

MATERIALS AND METHODS

A systematic literature search was performed. RCTs, reporting stone expulsion as a dichotomous outcome, showing statistical significance were eligible. FI (the number of patients needed to change from a non-event to event group, to lose statistical significance) and Fragility quotient (FI divided by total sample size), were calculated while Pearson's correlation and Mann-Whitney U test were used as appropriate.

RESULTS

Thirty-six RCTs were eligible, with median FI = 3.5 and fragility quotient = 0.042, median sample size = 81, median journal impact factor = 1.73 and median reported p value = 0.008. In 33.3% of the studies, number of patients lost during follow-up was larger than FI, while in 13.89% of the studies, FI was 0, indicating use of inappropriate statistical method. Pearson's correlation showed significant positive association between FI and sample size (r = 0.981), number of events (r = 0.982) and impact factor (r = 0.731), while no association was found with p value or publication year.

CONCLUSIONS

In this analysis, a calculated FI of 3.5 indicates that findings from RCTs on MET for ureteral stones are fragile and should be interpreted in combination with clinical thinking and expertise.

Current State of Evidence on Kidney Transplantation: How Fragile Are the Results?

BACKGROUND

The study aims is to use the fragility index (FI) to examine the strength of evidence of randomized controlled trials (RCTs) published in the last decade on kidney transplantation.

METHODS

We searched MEDLINE for studies on kidney transplantation. We included the RCTs that compared 2 groups with 1:1 randomization and reported significant P-values (<0.05) for a dichotomous outcome and were published in the top 10 transplant journals. We calculated the FI; a calculation used to determine the minimum number of subjects needed to change from a nonevent to an event to make the study results nonsignificant (P-value >0.05).

RESULTS

57 RCTs met our inclusion criteria. The median sample size was 100 participants in each arm, the median number of events was 16(IQR 8-30) in the intervention group. Among the included trials, 79% were industry-funded, 93% involved medications, and the majority were open-label. The median FI was 3 (IQR 1-11). In 43% of the trials, the number of patients reported lost to follow-up was higher than or equal to the FI. Only 4% of the RCTs imputed a value for the missing dichotomous outcome. Furthermore, the median number of subjects who discontinued the trial due to adverse effects was 21, which was greater than the FI in 60% of the RCTs.

CONCLUSION

The arbitrary classification of results into "significant" and "nonsignificant" based on p-value <0.05 should perhaps be interpreted with the help of other statistical parameters and FI is one of them.

Antithrombotic Therapy Recommendations in the European Society of Cardiology Guidelines: How Robust Are the Randomized Controlled Trials Underpinning Them?

Introduction  Criticisms have been raised against the sole use of p -value in interpreting results from randomized controlled trials (RCTs). Additional tools have been suggested, like the fragility index (FI), a measure of a trial's robustness/fragility, and derivative measures. The FI is the minimum number of patients who would have to be converted from nonevents to events, in the group with the least events, for a result to lose statistical significance. Objective  This study aimed to evaluate RCT supporting European Society of Cardiology (ESC) guidelines regarding antithrombotics, using the FI and FI-related measures. Methods  FI, fragility quotient (FQ), and FI minus LTF lost to follow-up (FI - LTF) were calculated for the RCT underpinning recommendations regarding antithrombotic therapy from the updated ESC guidelines. LTF was compared with FI. Results were calculated for the total group of studies, as per guideline and as per recommendation type. Results  Overall, 61 studies were included. The median FI was 24.5 (interquartile range [IQR]: 9.0-60.0) and median FQ was 0.0035 (IQR: 0.0019-0.0056). Median FI - LTF was 2.0 (IQR: 0.0-38.0). Twenty (32.8%) of the studies had one primary or main safety outcome with LTF exceeding FI. Peripheral arterial disease guideline and chronic coronary syndrome guideline had the lowest (2.5; IQR: 1.8-3.3) and the highest (48.5; IQR: 23.8-73.0) FI, respectively. Conclusion  The median FI suggests robustness of clinical trials evaluating antithrombotic drugs cited in the guidelines, but about one-third of them had LTF larger than FI. This emphasizes the need for assessing trials' robustness when constructing guidelines.

Survival-Inferred Fragility Index of Phase 3 Clinical Trials Evaluating Immune Checkpoint Inhibitors

IMPORTANCE

In science and medical research, extreme and dichotomous conclusions may be drawn based on whether the P value falls above or below the threshold. The fragility index (ie, the minimum number of changes from nonevents to events resulting in loss of statistical significance) captures the vulnerability of statistics in trials with binary outcomes. There are a growing number of clinical trials of immune checkpoint inhibitors (ICIs), as well as expanding eligibility for patients to receive them. The robustness of survival outcomes in randomized clinical trials (RCTs) should be evaluated using the fragility index extended to time-to-event data.

OBJECTIVE

To calculate the fragility of survival data in RCTs evaluating ICIs.

DESIGN, SETTING, AND PARTICIPANTS

In this cross-sectional study, data on phase 3 prospective RCTs investigating ICIs included in PubMed from inception until January 1, 2020, were extracted. Two- or three-group studies reporting results for overall survival were eligible for the survival-inferred fragility index (SIFI) calculation, which is the minimum number of reassignments of the best survivors from the interventional group to the control group resulting in loss of significance (defined as P < .05 by log-rank test). For nonsignificant results, a negative SIFI was calculated by reversing the direction of reassignment (from the control group to the interventional group).

MAIN OUTCOMES AND MEASURES

Survival-inferred fragility index.

RESULTS

A total of 45 phase 3 prospective RCTs (4 of which had 3 groups, for a total of 49 groups) were identified, of which 6 (13%) investigated anti-cytotoxic T-lymphocyte-associated protein 4 (CTLA-4) agents, 25 (56%) investigated anti-programmed cell death 1 (PD-1) agents, 12 (27%) investigated anti-programmed cell death 1 ligand 1 agents, and 3 (7%) investigated the combination of anti-CTLA-4 and anti-PD-1 agents. The median SIFI was 5 (interquartile range, -4 to 12) for the intention-to-treat analysis; for these trials, the SIFI was 1% or less of the total sample size in 17 of 49 populations (35%). In 25 of the 49 intention-to-treat populations (51%), the SIFI was less than the number of censored patients in the intervention group shortly after randomization (defined as <5% of the follow-up time).

CONCLUSIONS AND RELEVANCE

This study suggests that many phase 3 RCTs evaluating ICI therapies have a low SIFI for overall survival, resulting in uncertainty regarding their potential clinical benefit. Although not a definitive solution for the problems arising from dichotomization, SIFI provides an additional means of assessing and communicating the strength of statistical conclusions.