Statistical Outcomes Guiding Periprosthetic Joint Infection Prevention and Revision Are Fragile: A Systematic Review of Randomized Controlled Trials

BACKGROUND

Dichotomous outcomes are frequently reported in orthopaedic research and have substantial clinical implications. This study utilizes the fragility index (FI) and fragility quotient (FQ) metrics to determine the statistical stability of outcomes reported in total joint arthroplasty randomized controlled trials (RCTs) relating to periprosthetic joint infection (PJI).

METHODS

The RCTs that reported dichotomous data related to PJI published between January 1, 2010, and December 31, 2022, were evaluated. The FI and reverse FI (RFI) were defined as the number of outcome event reversals required to reverse the significance of significant and nonsignificant outcomes, respectively. The FQ was determined by dividing the FI or RFI by the respective sample size. There were 108 RCTs screened, and 17 studies included for analysis.

RESULTS

A total of 58 outcome events were identified, with a median FI of 4 (interquartile range [IQR] 2 to 5) and associated FQ of 0.0417 (IQR 0.0145 to 0.0602). The 13 statistically significant outcomes had a median FI of 1 (IQR 1 to 2) and FQ of 0.00935 (IQR 0.00629 to 0.01410). The 45 nonsignificant outcomes had a median RFI of 4 (IQR 3 to 5) and FQ of 0.05 (IQR 0.0361 to 0.0723). The number of patients lost to follow-up was greater than or equal to the FI in 46.6% of outcomes.

CONCLUSIONS

Statistical outcomes in RCTs analyzing PJI are fragile and may lack statistical integrity. We recommend a comprehensive fragility analysis, with the reporting of FI and FQ metrics, to aid in the interpretation of outcomes in the total joint arthroplasty literature.

Statistical Fragility in Minimally Invasive Colorectal Surgery Studies: A Review of Randomized Trials

Purpose: The P value has been used as a statistical tool in randomized controlled trials (RCTs) to establish significance but does not provide information on the robustness of a study when used alone. The fragility index (FI) provides a supplemental approach for demonstrating robustness in RCTs that report dichotomous outcomes. This study aims to determine the statistical fragility of RCTs that compare minimally invasive techniques with open techniques in managing benign and malignant colorectal diseases. Methods: Dichotomous outcomes of minimally invasive surgery versus open surgery in RCTs from 2000 to 2023 were assessed. The overall FI and fragility quotient (FQ) of each study were calculated. Results: Of the 1377 screened studies, 50 met the inclusion criteria. In total, 820 outcomes were recorded with 747 outcomes reported as not significant (P ≥ .05) and 73 as significant (P < .05). The overall FI for all studies including all outcomes was 5 (interquartile range [IQR] 4-7) with a FQ of 0.031 (IQR 0.014-0.062). Of the 50 RCTs, 6 (12%) reported a loss to follow-up that was greater than the overall FI of 5. Conclusions: As RCTs are judged increasingly beyond just the P value, practicing colorectal surgeons will benefit from using and interpreting the FI, FQ, and the P value of studies both in analyzing future RCTs and in determining whether or not to make a change in their clinical practice if there is an efficiently true discovery.

Applying the fragility index to randomized controlled trials evaluating total neoadjuvant therapy for rectal cancer: A methodological survey

BACKGROUND

Recently, there has been significant interest in, and adoption of, total neoadjuvant therapy (TNT) for locally advanced rectal cancer (LARC). We designed the present study to assess the robustness of the randomized controlled trials (RCTs) evaluating contemporary TNTs for LARC using the fragility index (FI).

MATERIALS AND METHODS

Relevant articles were identified through a review article by Johnson et al. in the Canadian Journal of Surgery. Dichotomous outcomes within these RCTs were eligible for inclusion if the reported effect size had a p-value < 0.05. The main outcome was FI for each included outcome. Walsh et al.'s method of calculating FI was utilized. Correlations between FI and research characteristics were assessed using the Spearman's rank correlation coefficients. Risk of bias was assessed using Cochrane recommended tools.

RESULTS

Ten RCTs were identified with 25 outcomes having statistically significant differences between groups. Eleven outcomes were time-to-event outcomes, while the remainder were dichotomous outcomes. Approximately half (n = 13) were oncologic outcomes. The median FI was 2 (interquartile range [IQR] 1-16). The number of patients lost to follow-up exceeded the FI in 17 outcomes (68.0 %) and thus these results were considered "fragile". Lower FI was associated with high risk of bias (rho = -0.5594) and greater loss to follow-up (rho = -0.4394), while higher FI was associated with large study size (rho = 0.5120).

CONCLUSIONS

The robustness of outcomes from trials assessing TNT for LARC was found to be questionable. Most outcomes were fragile, as determined by the FI. This survey is limited by the number of included studies.

Fragility of overactive bladder medication clinical trials: A systematic review

PURPOSE

Overactive bladder (OAB) syndrome significantly impairs quality of life, often necessitating pharmacological interventions with associated risks. The fragility of OAB trial outcomes, as measured by the fragility index (FI: smallest number of event changes to reverse statistical significance) and quotient (FQ: FI divided by total sample size expressed as a percentage), is critical yet unstudied.

MATERIALS AND METHODS

We conducted a systematic search for randomized controlled trials on OAB medications published between January 2000 and August 2023. Inclusion criteria were trials with two parallel arms reporting binary outcomes related to OAB medications. We extracted trial details, outcomes, and statistical tests employed. We calculated FI and FQ, analyzing associations with trial characteristics through linear regression.

RESULTS

We included 57 trials with a median sample size of 211 participants and a 12% median lost to follow-up. Most studies investigated anticholinergics (37/57, 65%). The median FI/FQ was 5/3.5%. Larger trials were less fragile (median FI 8; FQ 1.0%) compared to medium (FI: 4; FQ 2.5%) and small trials (FI: 4; FQ 8.3%). Double-blinded studies exhibited higher FQs (median 2.9%) than unblinded trials (6.7%). Primary and secondary outcomes had higher FIs (median 5 and 6, respectively) than adverse events (FI: 4). Each increase in 10 participants was associated with a +0.19 increase in FI (p < 0.001).

CONCLUSIONS

A change in outcome for a median of five participants, or 3.5% of the total sample size, could reverse the direction of statistical significance in OAB trials. Studies with larger sample sizes and efficacy outcomes from blinded trials were less fragile.

Fragility index analysis for randomized controlled trials of approved biologicals and small molecule drugs in inflammatory bowel diseases

INTRODUCTION

Biologics and small molecules have been increasingly applied in Crohn's disease (CD) and ulcerative colitis (UC). But the robustness of their trials has not been evaluated.

METHODS

We initially collected all the approved biologics or small molecules for CD or UC up to December 1, 2022. Databases were then queried by keywords in chemical name and CD or UC. Randomized controlled trials (RCTs) in the two-arm, 1:1 design were included. Fragility index (FI) and fragility quotient (FQ) were subsequently calculated.

RESULTS

We included twenty-eight RCTs, including nine pivotal trials listed in approval labels, nineteen non-pivotal trials not included in the labels. The median sample size was 99 [IQR, 60-262] and the median number of loss-of-follow-up (LFU) was 14 [IQR, 8-43]. Pivotal trials in the labels had the median FI of 8 [IQR, 4-14, n = 6] that was marginally higher than non-pivotal trials (3 [IQR, 2-4], p = 0.08). The median FQ was 0.0330 [IQR, 0.1220-0.0466] and 0.0310 [IQR, 0.0129-0.0540] for pivotal and non-pivotal trials, respectively (p = 1.0). The sample size and FI were significantly correlated (Spearman correlation coefficient [r] = 0.56, 95 %CI 0.21-0.78, p = 0.003). The number of total events was also significantly correlated with FI (r = 0.53, 95 %CI 0.17-0.77, p = 0.006). Study p-values were significantly associated with FI (p = 0.01): trials with p-values < 0.001 had the highest median FI of 10 [IQR, 6-17]. No factor was found strongly correlated with FQ.

CONCLUSION

Results from trials assessing administration-approved biologics or small molecules for treating CD or UC were vulnerable to small changes by measuring FI or FQ. Pivotal studies contributing to regulatory approvals exhibited a relatively higher degree of resilience compared to non-pivotal trials.

Fragility of Statistically Significant Outcomes in Colonic Diverticular Disease Randomized Trials: A Systematic Review

BACKGROUND

The p value has been criticized as an oversimplified determination of whether a treatment effect exists. One alternative is the fragility index. It is a representation of the minimum number of nonevents that would need to be converted to events to increase the p value above 0.05.

OBJECTIVE

To determine the fragility index of randomized controlled trials assessing the efficacy of interventions for patients with diverticular disease since 2010 to assess the robustness of current evidence.

DESIGN

MEDLINE, Embase, and Cochrane Central Register of Controlled Trials were searched from inception to August 2022.

SETTINGS

Articles were eligible for inclusion if they were randomized trials conducted between 2010 and 2022 with parallel, superiority designs evaluating interventions in patients with diverticular disease. Only randomized trials with dichotomous primary outcomes with an associated p value of <0.05 were considered for inclusion.

PARTICIPANTS

Any surgical or medical intervention for patients with diverticular disease.

MAIN OUTCOME MEASURES

The fragility index was determined by adding events and subtracting nonevents from the groups with the smaller number of events. Events were added until the p value exceeded 0.05. The smallest number of events required was considered the fragility index.

RESULTS

After screening 1271 citations, 15 randomized trials met the inclusion criteria. Nine of the studies evaluated surgical interventions and 6 evaluated medical interventions. The mean number of patients randomly assigned and lost to follow-up per randomized controlled trial was 92 (SD 35.3) and 9 (SD 11.4), respectively. The median fragility index was 1 (range, 0-5). The fragility indices for the included studies did not correlate significantly with any study characteristics.

LIMITATIONS

Small sample, heterogeneity, and lack of inclusion of studies with continuous outcomes.

CONCLUSIONS

The randomized trials evaluating surgical and medical interventions for diverticular disease are not robust. Changing a single-outcome event in most studies was sufficient to make a statistically significant study finding not significant. See Video Abstract .

FRAGILIDAD DE LOS RESULTADOS ESTADSTICAMENTE SIGNIFICATIVOS EN ENSAYOS ALEATORIOS DE ENFERMEDAD DIVERTICULAR DEL COLON UNA REVISIN SISTEMTICA

ANTECEDENTES:El valor p ha sido criticado por una determinación demasiado simplificada de si existe un efecto del tratamiento. Una alternativa es el Índice de Fragilidad. Es una representación del número mínimo de no eventos que deberían convertirse en eventos para aumentar el valor p por encima de 0,05.OBJETIVO:Determinar el IF de ensayos controlados aleatorios que evalúan la eficacia de las intervenciones para pacientes con enfermedad diverticular desde 2010 para evaluar la solidez de la evidencia actual.FUENTES DE DATOS:Se realizaron búsquedas en MEDLINE, Embase y CENTRAL desde el inicio hasta agosto de 2022.SELECCIÓN DE ESTUDIOS:Los artículos eran elegibles para su inclusión si eran ensayos aleatorizados realizados entre 2010 y 2022 con diseños paralelos de superioridad que evaluaran intervenciones en pacientes con enfermedad diverticular. Sólo se consideraron para su inclusión los ensayos aleatorizados con resultados primarios dicotómicos con un valor de p asociado menor que 0,05.INTERVENCIÓNES:Cualquier intervención quirúrgica o médica para pacientes con enfermedad diverticular.PRINCIPALES MEDIDAS DE VALORACIÓN:El índice de fragilidad se determinó sumando eventos y restando no eventos de los grupos con el menor número de eventos. Se agregaron eventos hasta que el valor p superó 0,05. El menor número de eventos requeridos se consideró índice de fragilidad.RESULTADOS:Después de examinar 1271 citas, 15 ensayos aleatorios cumplieron los criterios de inclusión. Nueve de los estudios evaluaron intervenciones quirúrgicas y seis evaluaron intervenciones médicas. El número medio de pacientes aleatorizados y perdidos durante el seguimiento por ECA fue 92 (DE 35,3) y 9 (DE 11,4), respectivamente. La mediana del índice de fragilidad fue 1 (rango: 0-5). Los índices de fragilidad de los estudios incluidos no se correlacionaron significativamente con ninguna característica del estudio.LIMITACIONES:Muestra pequeña, heterogeneidad y falta de inclusión de estudios con resultados continuos.CONCLUSIONES:Los ensayos aleatorios que evalúan las intervenciones quirúrgicas y médicas para la enfermedad diverticular no son sólidos. Cambiar un solo evento de resultado en la mayoría de los estudios fue suficiente para que un hallazgo estadísticamente significativo del estudio no fuera significativo. (Traducción- Dr. Ingrid Melo ).

A fragility index analysis of clinical trials evaluating low-intensity extracorporeal shockwave therapy for erectile dysfunction

Erectile dysfunction is a common sexual dysfunction that affects a significant proportion of men. Low-intensity extracorporeal shockwave therapy has been evaluated in multiple clinical trials as a therapeutic option for men with erectile dysfunction. The robustness of these clinical trials is not well defined, as the trials are hindered by inconsistent treatment protocols, small study arm size and short follow-up intervals. The fragility index is a statistical analysis which is used to evaluate the robustness of clinical trials. It is calculated by evaluating the minimum number of patients in a given trial arm that would be required to have an alternative outcome to alter the statistical significance of the results. The lowest fragility index in statistically significant trials is 1, meaning that if just one participant experienced an alternate outcome, the results would no longer achieve statistical significance. The upper limit is determined by the number of participants in a given arm of the trial. Herein, a scoping review of clinical trials evaluating the efficacy of low-intensity extracorporeal shockwave therapy in erectile dysfunction to determine the fragility index of trials with clinically significant results. We hypothesized that the fragility index would be low, indicating the results are less robust and generalizable.

What's in a P-Value? A Fragility Analysis of RCTs in the AUA Guidelines for Benign Prostatic Hyperplasia

OBJECTIVE

To use the fragility index (FI) and fragility quotient (FQ) to assess the strength of statistically significant findings for randomized controlled trials (RCTs) cited in the American Urological Association (AUA) guidelines for benign prostatic hyperplasia.

METHODS

Two investigators independently screened the AUA guidelines for management of benign prostatic hyperplasia for RCTs cited as evidence for recommendations. Investigators extracted data related to event rate per group and loss to follow-up which was compared against the FI. Stata 17.0 was used to calculate the FI and FQ which was then summarized and reported according to primary or secondary endpoints.

RESULTS

Among the 373 citations in the AUA guidelines, 24 RCTs met inclusion criteria with 29 distinct outcomes analyzed. The median fragility index was 12 (IQR = 4-38), indicating that twelve alternative events to either study arm would nullify statistical significance. Six studies had a FI of ≤2, indicating that only 1-2 outcomes would need to be changed in order to render nonsignificance of results. In 10/24 RCTs, the number of patients lost to follow-up was greater than the FI.

CONCLUSION

The AUA Clinical Practice Guidelines for management of benign prostatic hyperplasia cite RCTs with more robust findings when compared to previous studies assessing fragility in the field of Urology. While several included studies had high fragility, the median FI in our analysis was approximately 4-5 times higher than comparable studies of urologic RCTs. However, there are areas where improvement is necessary to support the highest quality of evidence-based medicine.

Fragility of cardiovascular outcome trials (CVOTs) examining nutrition interventions among patients with diabetes mellitus: a systematic review of randomized controlled trials

PURPOSE

There is controversy regarding the optimal statistical method to interpret how robust is a statistically significant result. The fragility index (FI) and the reverse fragility index (RFI) are quantitative measures that can facilitate the appraisal of a clinical trial's robustness. This study was performed to evaluate the FI and RFI of randomized controlled trials (RCTs) examining nutritional interventions in patients with diabetes mellitus, focusing on cardiovascular outcomes.

METHODS

A systematic search was conducted and relevant RCTs were identified in three databases. RCTs examining nutritional interventions (supplements or dietary patterns) in patients with DM with dichotomous primary endpoints involving cardiovascular outcomes were eligible. Data were extracted to compose 2 × 2 event tables and the FI and RFI were calculated for each comparison, using Fisher's exact test. Risk of bias (RoB) of the included RCTs was assessed with the Cochrane RoB 2.0 tool.

RESULTS

A total of 14,315 records were screened and 10 RCTs were included in the analyses. The median FI of the paired comparisons was 3 (IQR: 2-4) and the median RFI was 8 (IQR: 4.5-17). RoB and heterogeneity were low.

CONCLUSIONS

RCTs examining nutritional interventions and cardiovascular outcomes among patients with diabetes mellitus appear to be statistically fragile. Τhe FI and the RFI can be reported and interpreted as an additional perspective of a trial's robustness.

HIGHLIGHTS

• In the evidence-healthcare era, assessing how robust statistically significant results are remains a matter of controversy. • Recently, the fragility index (FI) and reverse fragility index (RFI) were proposed to assess the robustness of randomized controlled trials (RCTs) with 2 × 2 comparisons. • When applying the FI and RFI, RCTs examining nutritional interventions and cardiovascular outcomes among patients with diabetes mellitus (DM) appear to be statistically fragile. • Τhe FI and the RFI can be reported and interpreted as an additional perspective of a trial's robustness. • RCTs implementing nutrition interventions among patients with DM can improve their methodology.

Randomized Controlled Trials on Renin Angiotensin Aldosterone System Inhibitors in Chronic Kidney Disease Stages 3–5: Are They Robust? A Fragility Index Analysis

Inhibition of the renin-angiotensin-aldosterone system (RAAS) is broadly recommended in many nephrological guidelines to prevent chronic kidney disease (CKD) progression. This work aimed to analyze the robustness of randomized controlled trials (RCTs) investigating the renal and cardiovascular outcomes in CKD stages 3–5 patients treated with RAAS inhibitors (RAASi). We searched for RCTs in MEDLINE (PubMed), EMBASE databases, and the Cochrane register. Fragility indexes (FIs) for every primary and secondary outcome were calculated according to Walsh et al., who first described this novel metric, suggesting 8 as the cut-off to consider a study robust. Spearman coefficient was calculated to correlate FI to p value and sample size of statistically significant primary and secondary outcomes. Twenty-two studies met the inclusion criteria, including 80,455 patients. Sample size considerably varied among the studies (median: 1693.5, range: 73–17,276). The median follow-up was 38 months (range 24–58). The overall median of both primary and secondary outcomes was 0 (range 0–117 and range 0–55, respectively). The median of FI for primary and secondary outcomes with a p value lower than 0.05 was 6 (range: 1–117) and 7.5 (range: 1–55), respectively. The medians of the FI for primary outcomes with a p value lower than 0.05 in CKD and no CKD patients were 5.5 (range 1–117) and 22 (range 1–80), respectively. Only a few RCTs have been shown to be robust. Our analysis underlined the need for further research with appropriate sample sizes and study design to explore the real potentialities of RAASi in the progression of CKD.

Fragility of statistically significant findings from randomized clinical trials of surgical treatment of humeral shaft fractures: A systematic review

BACKGROUND

Despite recent meta-analyses of randomized controlled trials (RCTs), there remains no consensus regarding the preferred surgical treatment for humeral shaft fractures. The fragility index (FI) is an emerging tool used to evaluate the robustness of RCTs by quantifying the number of participants in a study group that would need to switch outcomes in order to reverse the study conclusions.

AIM

To investigate the fragility index of randomized control trials assessing outcomes of operative fixation in proximal humerus fractures.

METHODS

We completed a systematic review of RCTs evaluating the surgical treatment of humeral shaft fractures. Inclusion criteria included: articles published in English; patients randomized and allotted in 1:1 ratio to 2 parallel arms; and dichotomous outcome variables. The FI was calculated for total complications, each complication individually, and secondary surgeries using the Fisher exact test, as previously published.

RESULTS

Fifteen RCTs were included in the analysis comparing open reduction plate osteosynthesis with dynamic compression plate or locking compression plate, intramedullary nail, and minimally invasive plate osteosynthesis. The median FI was 0 for all parameters analyzed. Regarding individual outcomes, the FI was 0 for 81/91 (89%) of outcomes. The FI exceeded the number lost to follow up in only 2/91 (2%) outcomes.

CONCLUSION

The FI shows that data from RCTs regarding operative treatment of humeral shaft fractures are fragile and does not demonstrate superiority of any particular surgical technique.

When the p Value Doesn't Cut It: The Fragility Index Applied to Randomized Controlled Trials in Colorectal Surgery

BACKGROUND

The American Statistical Association, among others, has called for the use of statistical methods beyond p ≤ 0.05. The fragility index is a statistical metric defined as the minimum number of patients for whom if an event rather than a nonevent occurred, then the p value would increase to ≥0.05. Previous reviews have demonstrated that many randomized controlled trials have a low fragility index, suggesting they may not be robust.

OBJECTIVE

The purpose of this study was to review the fragility indices of randomized controlled trials in colorectal surgery.

DATA SOURCES

A PubMed search was performed.

STUDY SELECTION

Colorectal surgery randomized controlled trials with a dichotomous primary outcome p ≤ 0.05 and publication between 2016 and 2018 were systematically identified.

INTERVENTIONS

All procedural interventions related to colorectal surgery were included.

MAIN OUTCOME MEASURES

The main measures were the fragility index and the number of patients lost to follow-up for each trial. The percentage of trials with the number of patients lost to follow-up greater than the fragility index was calculated.

RESULTS

In total, 712 abstracts were reviewed, with 90 trials meeting the inclusion criteria. The median fragility index was 3 (interquartile range of 1 to 10). In 51 of the 90 trials (57%), the number of patients lost to follow-up was greater than the fragility index.

LIMITATIONS

The fragility index is only one measure of the robustness of a randomized clinical trial.

CONCLUSIONS

Most colorectal surgery randomized controlled trials have a low fragility index. In 57% of trials, more patients were lost to follow-up than would be required to change the outcome of the trial from "significant" to "nonsignificant" based on the p value. This emphasizes the importance of assessing the robustness of clinical trials when considering their clinical application, rather than relying solely on the p value. See Video Abstract at http://links.lww.com/DCR/B741.CUANDO EL VALOR-P ES INSUFICIENTE: ÍNDICE DE FRAGILIDAD APLICADO EN ESTUDIOS ALEATORIOS CONTROLADOS EN CIRUGÍA COLORECTAL.

ANTECEDENTES

La Sociedad Estadounidense de Estadística, entre otros, ha pedido el uso de métodos estadísticos más allá de p <0,05. El índice de fragilidad es una medida estadística definida como el número de desenlaces que podrían cambiar para revertir, o conseguir, la significación estadística, así el valor p aumentaría a ≥ 0,05. Las revisiones anteriores han demostrado que muchos estudios aleatorios controlados tienen un índice de fragilidad bajo, lo que sugiere que pueden poco sólidos.

OBJETIVO

El propósito de la présente investigación fué de revisar los índices de fragilidad de los estudios aleatorios controlados en cirugía colorrectal.

FUENTES DE DATOS

PubMed.

SELECCIN DE ESTUDIOS

Se identificaron sistemáticamente estudios aleatorios controlados de cirugía colorrectal con un resultado primario dicotómico, valor de p ≤ 0,05 y publicados entre 2016-2018.

INTERVENCIONES

Se incluyeron todas aquellas intervenciones con procedimientos relacionados con la cirugía colorrectal.

PRINCIPALES MEDIDAS DE RESULTADO

Las principales medidas fueron: el índice de fragilidad y el número de pacientes perdidos durante el seguimiento en cada estudio. Se calculó el el índice de fragilidad en porcentaje de estudios con el mayor número de pacientes perdidos durante el seguimiento mas prolongado.

RESULTADOS

En total, se revisaron 712 resúmenes con 90 ensayos que cumplieron con los criterios de inclusión. La mediana del índice de fragilidad fue de 3 (rango intercuartíl de 1 a 10). En 51 de los 90 estudios (57%), el número de pacientes perdidos durante el seguimiento fue mayor que el índice de fragilidad.

LIMITACIONES

El índice de fragilidad es solo una medida de la robustez de un estúdio clínico aleatorio.

CONCLUSIONES

La mayoría de los estudios aleatorios y controlados en cirugía colorrectal tienen un índice de fragilidad bajo. En el 57% de los estudios, se perdieron más pacientes durante el seguimiento de los que se necesitarían para cambiar el resultado del estudios de grado "significativo" a un grado "no significativo" según el valor-p. Este concepto enfatiza la importancia de evaluar la robustez de los estudios clínicos al considerar su aplicación verdadera aplicación clínica, en lugar de depender únicamente del valor-p. Consulte Video Resumen en http://links.lww.com/DCR/B741. (Traducción-Dr. Xavier Delgadillo).

The fragility of statistically significant results in gynecologic surgery: A systematic review

OBJECTIVE

To use the fragility index (FI) to evaluate the robustness of gynaecologic surgery trials that report statistically significant results. The FI defines the minimum number of patients who must have an alternative outcome to alter statistical significance.

DATA SOURCES

We searched MEDLINE, Web of Science, Embase, and ClinicalTrials.gov from 2011 to 2021 to identify gynaecologic surgery randomized controlled trials (RCTs).

STUDY SELECTION

A total of 4775 trials were screened for eligibility. All included studies evaluated benign gynaecologic surgery interventions or peri-operative medical interventions. Only two-arm RCTs with statistically significant dichotomous primary outcomes were included. Ninety-three trials were ultimately included for analysis.

DATA EXTRACTION AND SYNTHESIS

Data from the included studies, including sample size, loss to follow-up, and number of events, were recorded. The FI of each study was calculated using a predefined technique. The overall FI and FIs by subgroup (clinical subspecialty, country of origin, and statistical test used) are reported as medians alongside their interquartile ranges (IQRs). The Kruskal-Wallis test was applied to find possible statistically significant relationships between FI and the nominal subgroups. Among this cohort, the median FI was 3 (IQR 1-7). The FI was 0 in 13 trials (14%), and in 39 trials (42%), the number of patients lost to follow-up was greater than the FI. The median FI within clinical subspecialty groups (general gynaecology, anaesthesia, urogynaecology, and fertility) did not differ (P = 0.122).

CONCLUSION

Statistically significant results of RCTs in gynaecologic surgery are fragile, suggesting that clinicians should interpret results with caution. This is particularly true when the number of patients lost to follow-up is greater than the FI. The FI serves as a quality metric that can be used to evaluate robustness of results when applying the outcomes of RCTs to clinical practice or guideline development.

Statistical Fragility of Surgical Clinical Trials in Orthopaedic Trauma

The Fragility Index (FI) and the Fragility Quotient (FQ) are powerful statistical tools that can aid clinicians in assessing clinical trial results. The purpose of this study was to use the FI and FQ to evaluate the statistical robustness of widely cited surgical clinical trials in orthopaedic trauma.

The fragility of randomized placebo-controlled trials for irritable bowel syndrome

BACKGROUND

The fragility index (FI) represents the number of participants whose status in a trial would have to change from a non-event (not experiencing the primary endpoint) to an event (experiencing the primary endpoint) in order to turn a statistically significant result into a non-significant result. We sought to evaluate the fragility indices of irritable bowel syndrome [IBS-mixed (IBS-M), IBS-constipation (IBS-C), & IBS-diarrhea (IBS-D)] trials.

METHODS

Irritable bowel syndrome trials published in high-impact journals were identified from Medline. Trials had to be in adults, randomized, parallel-armed, with at least one statistically significant binary outcome, and an achieved primary endpoint of therapeutic efficacy. FI and correlation coefficients were calculated, and regression modeling used to identify predictors of a high FI.

KEY RESULTS

Twelve trials were analyzed with a median FI of 6 (range: 0-38). Median sample size in all trials was 366 (range: 44-856). Trial publication year (p = 0.71), journal impact factor (p = 0.52), duration of study (p = 0.12), and number need to treat [NNT] (p = 0.29) were not predictive of a high FI. While a lower p-value correlated with a higher FI (p = 0.039), no correlation was noted between FI and impact factor (R = -0.20, p = 0.52), trial publication year (R = 0.12, p = 0.71), duration of trial (R = -0.46, p = 0.13), NNT (R = -0.34, p = 0.29), and sample size (R = 0.23, p = 0.5). The highest FI was in a Ramosetron trial (FI = 30) for IBS-D.

CONCLUSION & INFERENCES

A median of six participants is needed to nullify results in the included IBS trials suggesting how easily statistical significance based on a threshold p-value may be overturned.

Urologic oncology randomized controlled trials are frequently fragile - A review of the urology literature

BACKGROUND

Randomized controlled trials (RCT) in urologic oncology are the basis of patient management. Considerable debate exists on the limitation of statistical reporting of randomized controlled trials. Fragility index (FI) is a measure of the number of events upon which the trials statistical results depend on. The FI is defined as the minimum number of patients whose status would have to change from a 'non-event' to 'event', in order to turn a statistically significant result to a non-significant result and vice versa. Our aim is to examine the FI of RCT's in urologic oncology published in the urology literature.

MATERIAL AND METHODS

We exhaustively searched MEDLINE and EMBASE from January 1, 2016 to December 31 2019 for RCT's in urology journals. Only studies reporting dichotomous outcomes were included and FI was calculated for each outcome. The distributions of FI across different journals and types of outcome (primary/ secondary, significant/ non-significant) were assessed. We examined the correlation of FI with sample size and P-value.

RESULTS

We identified 216 RCT's, 79 were eligible for analysis. Median FI was 3.0 (2.5, 6.0). One hundred and forty-six (89.6%) outcomes from 72 (94.7%) RCT's had a FI lower than 10 and 6 (3.7%) outcomes from 6 (7.9%) RCT's had a FI = 0. There was no statistically significant difference in FI between different types of outcomes. There was a modest correlation between the FI and the sample size (r_s 0.50, P < 0.01) and a weak correlation between the FI the P value (r_s 0.15 (P < 0.04)).

CONCLUSIONS

Randomized controlled trials in urology journals that study dichotomous outcomes often report fragile results that should be regarded with caution. Reporting the FI alongside P values may enhance the interpretation and implementation of urologic oncology RCT's.

Fragility index of urological literature regarding medical expulsive treatment

INTRODUCTION

The role of medical expulsive treatment (MET) is controversial. Fragility index is an additional metric to assess randomized controlled trials (RCTs) outcome validity and indicates how many patients would be required to convert a trial from being statistically significant, to not significant. The larger is the FI, the better the trial's data. The aim of this study is to assess FI of RCTs regarding MET for ureteral stones.

MATERIALS AND METHODS

A systematic literature search was performed. RCTs, reporting stone expulsion as a dichotomous outcome, showing statistical significance were eligible. FI (the number of patients needed to change from a non-event to event group, to lose statistical significance) and Fragility quotient (FI divided by total sample size), were calculated while Pearson's correlation and Mann-Whitney U test were used as appropriate.

RESULTS

Thirty-six RCTs were eligible, with median FI = 3.5 and fragility quotient = 0.042, median sample size = 81, median journal impact factor = 1.73 and median reported p value = 0.008. In 33.3% of the studies, number of patients lost during follow-up was larger than FI, while in 13.89% of the studies, FI was 0, indicating use of inappropriate statistical method. Pearson's correlation showed significant positive association between FI and sample size (r = 0.981), number of events (r = 0.982) and impact factor (r = 0.731), while no association was found with p value or publication year.

CONCLUSIONS

In this analysis, a calculated FI of 3.5 indicates that findings from RCTs on MET for ureteral stones are fragile and should be interpreted in combination with clinical thinking and expertise.

Current State of Evidence on Kidney Transplantation: How Fragile Are the Results?

BACKGROUND

The study aims is to use the fragility index (FI) to examine the strength of evidence of randomized controlled trials (RCTs) published in the last decade on kidney transplantation.

METHODS

We searched MEDLINE for studies on kidney transplantation. We included the RCTs that compared 2 groups with 1:1 randomization and reported significant P-values (<0.05) for a dichotomous outcome and were published in the top 10 transplant journals. We calculated the FI; a calculation used to determine the minimum number of subjects needed to change from a nonevent to an event to make the study results nonsignificant (P-value >0.05).

RESULTS

57 RCTs met our inclusion criteria. The median sample size was 100 participants in each arm, the median number of events was 16(IQR 8-30) in the intervention group. Among the included trials, 79% were industry-funded, 93% involved medications, and the majority were open-label. The median FI was 3 (IQR 1-11). In 43% of the trials, the number of patients reported lost to follow-up was higher than or equal to the FI. Only 4% of the RCTs imputed a value for the missing dichotomous outcome. Furthermore, the median number of subjects who discontinued the trial due to adverse effects was 21, which was greater than the FI in 60% of the RCTs.

CONCLUSION

The arbitrary classification of results into "significant" and "nonsignificant" based on p-value <0.05 should perhaps be interpreted with the help of other statistical parameters and FI is one of them.

Fragility index of network meta-analysis with application to smoking cessation data

BACKGROUND AND OBJECTIVES

The network meta-analysis (NMA) is frequently used to synthesize evidence for multiple treatment comparisons, but its complexity may affect the robustness (or fragility) of the results. The fragility index (FI) is recently proposed to assess the fragility of the results from clinical studies and from pairwise meta-analyses. We extend the FI to NMAs with binary outcomes.

METHODS

We define the FI for each treatment comparison in NMAs. It quantifies the minimal number of events necessary to be modified for altering the comparison's statistical significance. We introduce an algorithm to derive the FI and visualizations of the process. A worked example of smoking cessation data is used to illustrate the proposed methods.

RESULTS

Some treatment comparisons had small FIs; their significance (or nonsignificance) could be altered by modifying a few events' status. They were related to various factors, such as P-values, event counts, and sample sizes, in the original NMA. After modifying event status, treatment ranking measures were also changed to different extents.

CONCLUSION

Many NMAs include insufficiently compared treatments, small event counts, or small sample sizes; their results are potentially fragile. The FI offers a useful tool to evaluate treatment comparisons' robustness and reliability.

Statistical Fragility of Surgical and Procedural Clinical Trials in Orthopaedic Oncology

The fragility index (FI) is a powerful tool that can be used to assess the statistical strength of a study outcome. This metric is defined as the number of patients who would need to have an alternative outcome to convert a clinical trial result from statistically significant to not statistically significant, or vice versa. No studies to date have used the FI to evaluate surgical and procedural clinical trials in the orthopaedic oncology literature. The primary purpose of this study was to use the FI to evaluate the statistical strength of widely cited surgical and procedural clinical trials in orthopaedic oncology.

Methods

We performed a PubMed search for orthopaedic oncology clinical trials in high impact orthopaedics-focused, oncology-focused, and general medicine journals. For each study included in this analysis, we calculated the FI for all identified dichotomous, categorical outcomes.

Results

We identified 23 studies with 48 outcomes. Twelve of these outcomes were statistically significant, with a median FI of two. Nine studies addressed the number of patients lost to follow up, and the FI was less than the number of patients lost to follow up for most outcomes (60%) in these studies.

Conclusions

The orthopaedic oncology literature has substantial statistical fragility, likely explained by a high number of patients lost to follow up and small sample sizes. More multicenter, cooperative studies are necessary to increases the robustness of clinical research in orthopaedic oncology.

Fragility index: how fragile is the data that support the American College of Gastroenterology guidelines for the management of Crohn's disease?

Randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine. However, recent literature has drawn attention to the limitations of using P-value to report statistical significance of outcomes in the clinical trials. We performed this analysis to analyze the strength of the data that supported the American College of Gastroenterology (ACG) guidelines for 'Management of Crohn's disease in adults' using fragility index (FI). We screened all the RCTs referenced in the 2018 ACG guidelines 'Management of Crohn's disease in adults'. We calculated the FI and the fragility quotient (FQ) and its correlation with P-value. Data were also collected on the patients lost to follow up, year of publication, sample size, number needed to treat (NNT), science citation index (SCI), presence of blinding and the number of centers in these studies. Of the 91 RCTs cited in this guideline, 32 RCTs met the inclusion criteria. The median values for FI for 32 trials were 3 [interquartile range (IQR) 2-6], FQ 0.026 (IQR 0.012-0.413), P-value 0.010 (IQR 0.001-0.03), lost to follow up 17 (IQR 10-39.5) and sample size 133 (IQR 74.5-281.5). There was statistically significant correlation between FI and P-value (rs -0.86, P <0.001) and sample size (rs 0.56, P = 0.002). There was no correlation found with number lost to follow up, NNT, SCI, year of publication, blinding and number of centers. The majority of the RCTs conducted in the field of Crohn's disease rely on small number of superior events for statistical significance, thus rendering the validity of their conclusion questionable. At least 18 out of 60 ACG recommendations are based on RCTs in which, number of patients lost to follow up exceeds FI, thus making reported outcomes of the trial weak. We suggest that FI and FQ should be included in clinical trials to better understand if the data are meaningful, beyond a P-value.

Fragility Index in Cardiovascular Randomized Controlled Trials

BACKGROUND

Efficacy of an intervention is commonly evaluated using P values, in addition to effect size measures such as absolute risk reduction, relative risk reduction, and numbers needed to treat. However, these measures are not always intuitive to clinicians. The fragility index (FI) is a more intuitive number that can facilitate interpretation but can only be used with binary outcomes. FI is the minimum number of patients who must be moved from the nonevent group to the event group to turn a significant result nonsignificant. In this retrospective analysis, we assessed the robustness of cardiovascular randomized controlled trials (RCTs), which report a positive (statistically significant) primary outcome by using the FI.

METHODS AND RESULTS

We searched Medline from 2007 to 2017 to identify cardiovascular RCTs published in 6 high impact journals (The Lancet, New England Journal of Medicine, Journal of the American Medical Association, Circulation, Journal of the American College of Cardiology and European Heart Journal). Only RCTs with sample sizes >500 and a 2-by-2 factorial design or dichotomous primary outcomes were selected. FI was calculated using a defined approach. Among the cohort of 123 RCTs that met inclusion criteria, median FI was 13 (interquartile range, 5-26). In 28 trials (22.8%), FI ranged between 1 and 4. In 37 trials (30.1%), number of patients lost to follow-up was higher than the FI. Pharmaceutical interventions had higher FI compared with other interventions, FI=19 (7-52; P=0.002). Median FI varied according to subspecialty (electrophysiology=2; heart failure=11; interventional cardiology=8; P=0.020) and multiregional RCTs had higher FI=22 (12-53.25; P=0.023). FI did not differ based on risk of bias indicators, funding, or publication year.

CONCLUSIONS

Considerable variations in FI were observed among cardiovascular trials, suggesting the need for careful interpretation of results, particularly when number of patients lost to follow-up exceeds FI.

Reexamining the data used in the 2012 guidelines of the American Association for the Study of Liver Diseases for the management of adult patients with ascites due to cirrhosis

Background:

In 2012, the American Association for the Study of Liver Diseases published practice guidelines for the management of patients with ascites caused by cirrhosis, using data from randomized controlled trials (RCTs) and observational studies. We reexamined the strength of these RCTs by calculating the fragility index (FI), a novel metric proposed for evaluating the robustness of RCTs.

Methods:

We screened all RCTs referenced in the guidelines for specific criteria. We calculated the FI and fragility quotient (FQ), and analyzed the correlation between FI and several variables.

Results:

Twenty-one RCTs were included. The median (25^th, 75^th) FI and FQ were 1 (interquartile range [IQR] 0.5-6) and 0.070 (IQR 0.008-0.166), respectively. For studies that reported the number of patients lost to follow up (12 RCTs), the median of patients lost was 2 (IQR 0-6.5). There was no significant correlation between FI and sample size (r_s=0.357), P-value (r_s=-0.299), number lost to follow up (r_s=0.355), Science Citation Index (r_s=0.347), year of publication (r_s=-0.085), blinding (r_pb=-0.18) or number of centers (r_pb=0.10). However, a significant correlation was seen between FI and number needed to treat (r_s=-0.549; P=0.015).

Conclusions:

RCTs in the field of cirrhosis-related ascites are fragile. Of the 21 trials analyzed, 13 had an FI of 3 or below and these trials influenced 13 of the 49 recommendations in the guidelines. We recommend the incorporation of FI and FQ in addition to P-value to better understand the meaning of the results in gastroenterological studies.

The fragility of statistically significant results from clinical nutrition randomized controlled trials

BACKGROUND & AIMS

Recently, a parameter called "Fragility index" (FI) has been proposed, which measures how many events the statistical significance relies on. The lower the FI the more "fragile" the results, and thus more care should be taken when interpreting the results. Our aim in this study was to check FI of nutritional trials.

METHODS

We conducted a systematic review of human clinical nutrition RCTs that report statistically significant dichotomous primary outcomes. We searched the EMBASE, MEDLINE, and Scopus databases. The FI of primary outcomes using the Fisher exact test was calculated and checked the correlations of FI with the number of randomised trials, the p-value of primary outcomes, the publication date, the journal impact factor and the number of patients lost to follow-up.

RESULTS

The initial database search revealed 5790 articles, 37 of which were included in qualitative synthesis. The median (IQR) FI for all studies was 1 (1-3). 28 studies (75.7%) had an FI lower or equal to 2, and in 12 (32.43%) articles, the FI was lower than the number of patients lost to follow-up. No correlations were found between FI and the study characteristics (number of randomized patients, p value of primary outcome, event ratio in experimental group, event ratio in control group, publication date, journal impact factor, lost to follow-up).

CONCLUSION

The results of RCTs in nutritional research often rely on a small number of events or patients. The number of patients lost to follow-up is frequently higher than the FI calculation. Formulating recommendations based on RCTs should be done with caution and FI may be used as auxiliary parameter when assessing the robustness of their findings.

The Fragility of Statistically Significant Findings in Pediatric Critical Care Randomized Controlled Trials

OBJECTIVES

The Fragility Index measures the number of events on which the statistical significance of a result depends and has been suggested as an adjunct statistical assessment for interpretation of trial results. This study aimed to assess the robustness of statistically significant results from pediatric critical care randomized controlled trials with dichotomous outcomes.

DATA SOURCES

A previously published scoping review of pediatric critical care randomized controlled trials (www.PICUtrials.net).

STUDY SELECTION

A total of 342 trials were screened for inclusion. After applying inclusion/exclusion criteria, 43 fulfilled eligibility criteria and were included in the analysis.

DATA EXTRACTION

Calculation of Fragility Index for trials reporting a statistically significant dichotomous outcome, and analysis of the relationship between trial characteristics and Fragility Index.

DATA SYNTHESIS

The median Fragility Index was 2 (interquartile range, 1-6). The median sample size was 98 (interquartile range, 50-148) and sample size demonstrated a strong correlation with the Fragility Index (r = 0.729; n = 43; p < 0.001). The median number of outcome events was 8 (interquartile range, 4-15) and the total number of outcome events also showed a strong correlation with the Fragility Index (r = 0.728; n = 43; p < 0.001).

CONCLUSIONS

Results from pediatric critical care randomized controlled trials with dichotomous outcomes reporting statistically significant findings often hinge on a small number of outcome events. Clinicians should exercise caution when interpreting results of trials with a low Fragility Index.