When the p Value Doesn't Cut It: The Fragility Index Applied to Randomized Controlled Trials in Colorectal Surgery

Statistical Fragility in Minimally Invasive Colorectal Surgery Studies: A Review of Randomized Trials

Purpose: The P value has been used as a statistical tool in randomized controlled trials (RCTs) to establish significance but does not provide information on the robustness of a study when used alone. The fragility index (FI) provides a supplemental approach for demonstrating robustness in RCTs that report dichotomous outcomes. This study aims to determine the statistical fragility of RCTs that compare minimally invasive techniques with open techniques in managing benign and malignant colorectal diseases. Methods: Dichotomous outcomes of minimally invasive surgery versus open surgery in RCTs from 2000 to 2023 were assessed. The overall FI and fragility quotient (FQ) of each study were calculated. Results: Of the 1377 screened studies, 50 met the inclusion criteria. In total, 820 outcomes were recorded with 747 outcomes reported as not significant (P ≥ .05) and 73 as significant (P < .05). The overall FI for all studies including all outcomes was 5 (interquartile range [IQR] 4-7) with a FQ of 0.031 (IQR 0.014-0.062). Of the 50 RCTs, 6 (12%) reported a loss to follow-up that was greater than the overall FI of 5. Conclusions: As RCTs are judged increasingly beyond just the P value, practicing colorectal surgeons will benefit from using and interpreting the FI, FQ, and the P value of studies both in analyzing future RCTs and in determining whether or not to make a change in their clinical practice if there is an efficiently true discovery.

Applying the fragility index to randomized controlled trials evaluating total neoadjuvant therapy for rectal cancer: A methodological survey

BACKGROUND

Recently, there has been significant interest in, and adoption of, total neoadjuvant therapy (TNT) for locally advanced rectal cancer (LARC). We designed the present study to assess the robustness of the randomized controlled trials (RCTs) evaluating contemporary TNTs for LARC using the fragility index (FI).

MATERIALS AND METHODS

Relevant articles were identified through a review article by Johnson et al. in the Canadian Journal of Surgery. Dichotomous outcomes within these RCTs were eligible for inclusion if the reported effect size had a p-value < 0.05. The main outcome was FI for each included outcome. Walsh et al.'s method of calculating FI was utilized. Correlations between FI and research characteristics were assessed using the Spearman's rank correlation coefficients. Risk of bias was assessed using Cochrane recommended tools.

RESULTS

Ten RCTs were identified with 25 outcomes having statistically significant differences between groups. Eleven outcomes were time-to-event outcomes, while the remainder were dichotomous outcomes. Approximately half (n = 13) were oncologic outcomes. The median FI was 2 (interquartile range [IQR] 1-16). The number of patients lost to follow-up exceeded the FI in 17 outcomes (68.0 %) and thus these results were considered "fragile". Lower FI was associated with high risk of bias (rho = -0.5594) and greater loss to follow-up (rho = -0.4394), while higher FI was associated with large study size (rho = 0.5120).

CONCLUSIONS

The robustness of outcomes from trials assessing TNT for LARC was found to be questionable. Most outcomes were fragile, as determined by the FI. This survey is limited by the number of included studies.

Statistical fragility of outcomes in acellular dermal matrix literature: A systematic review of randomized controlled trials

BACKGROUND

Acellular dermal matrix (ADM) is commonly used in plastic and reconstructive surgery. With the abundance of randomized controlled trials (RCTs) reporting P-values for ADM outcomes, this study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to evaluate the statistical stability of the outcomes in ADM RCTs.

METHODS

PubMed, Embase, SCOPUS, Medline, and Cochrane databases were reviewed for ADM RCTs (2003-present) reporting a dichotomous, categorical outcome. FI and rFI (event reversals influencing outcome significance) and FQ (standardized fragility) were calculated and reported as median. Subgroup analysis was performed based on intervention types.

RESULTS

Among the 127 studies screened, 56 RCTs with 579 outcomes were included. The median FI stood at 4 (3-5) and FQ was 0.04 (0.03-0.07). Only 101 outcomes were statistically significant with a median FI of 3 (1-6) and FQ of 0.04 (0.02-0.08). The nonsignificant outcomes had a median FI of 4 (3-5) and FQ of 0.04 (0.03-0.07). Notably, 26% of the outcomes had several patients lost to follow up equal to or surpassing the FI. Based on the intervention type, the median FIs showed minor fluctuations but remained low.

CONCLUSIONS

Outcomes from ADM-related RCTs were statistically fragile. Slight outcome reversals or maintenance of patient follow-up can alter the significance of results. Therefore, future researchers are recommended to jointly report FI, FQ, and P-values to offer a comprehensive view of the robustness in ADM literature.

Fragility index analysis for randomized controlled trials of approved biologicals and small molecule drugs in inflammatory bowel diseases

INTRODUCTION

Biologics and small molecules have been increasingly applied in Crohn's disease (CD) and ulcerative colitis (UC). But the robustness of their trials has not been evaluated.

METHODS

We initially collected all the approved biologics or small molecules for CD or UC up to December 1, 2022. Databases were then queried by keywords in chemical name and CD or UC. Randomized controlled trials (RCTs) in the two-arm, 1:1 design were included. Fragility index (FI) and fragility quotient (FQ) were subsequently calculated.

RESULTS

We included twenty-eight RCTs, including nine pivotal trials listed in approval labels, nineteen non-pivotal trials not included in the labels. The median sample size was 99 [IQR, 60-262] and the median number of loss-of-follow-up (LFU) was 14 [IQR, 8-43]. Pivotal trials in the labels had the median FI of 8 [IQR, 4-14, n = 6] that was marginally higher than non-pivotal trials (3 [IQR, 2-4], p = 0.08). The median FQ was 0.0330 [IQR, 0.1220-0.0466] and 0.0310 [IQR, 0.0129-0.0540] for pivotal and non-pivotal trials, respectively (p = 1.0). The sample size and FI were significantly correlated (Spearman correlation coefficient [r] = 0.56, 95 %CI 0.21-0.78, p = 0.003). The number of total events was also significantly correlated with FI (r = 0.53, 95 %CI 0.17-0.77, p = 0.006). Study p-values were significantly associated with FI (p = 0.01): trials with p-values < 0.001 had the highest median FI of 10 [IQR, 6-17]. No factor was found strongly correlated with FQ.

CONCLUSION

Results from trials assessing administration-approved biologics or small molecules for treating CD or UC were vulnerable to small changes by measuring FI or FQ. Pivotal studies contributing to regulatory approvals exhibited a relatively higher degree of resilience compared to non-pivotal trials.

Fragility of Statistically Significant Outcomes in Colonic Diverticular Disease Randomized Trials: A Systematic Review

BACKGROUND

The p value has been criticized as an oversimplified determination of whether a treatment effect exists. One alternative is the fragility index. It is a representation of the minimum number of nonevents that would need to be converted to events to increase the p value above 0.05.

OBJECTIVE

To determine the fragility index of randomized controlled trials assessing the efficacy of interventions for patients with diverticular disease since 2010 to assess the robustness of current evidence.

DESIGN

MEDLINE, Embase, and Cochrane Central Register of Controlled Trials were searched from inception to August 2022.

SETTINGS

Articles were eligible for inclusion if they were randomized trials conducted between 2010 and 2022 with parallel, superiority designs evaluating interventions in patients with diverticular disease. Only randomized trials with dichotomous primary outcomes with an associated p value of <0.05 were considered for inclusion.

PARTICIPANTS

Any surgical or medical intervention for patients with diverticular disease.

MAIN OUTCOME MEASURES

The fragility index was determined by adding events and subtracting nonevents from the groups with the smaller number of events. Events were added until the p value exceeded 0.05. The smallest number of events required was considered the fragility index.

RESULTS

After screening 1271 citations, 15 randomized trials met the inclusion criteria. Nine of the studies evaluated surgical interventions and 6 evaluated medical interventions. The mean number of patients randomly assigned and lost to follow-up per randomized controlled trial was 92 (SD 35.3) and 9 (SD 11.4), respectively. The median fragility index was 1 (range, 0-5). The fragility indices for the included studies did not correlate significantly with any study characteristics.

LIMITATIONS

Small sample, heterogeneity, and lack of inclusion of studies with continuous outcomes.

CONCLUSIONS

The randomized trials evaluating surgical and medical interventions for diverticular disease are not robust. Changing a single-outcome event in most studies was sufficient to make a statistically significant study finding not significant. See Video Abstract .

FRAGILIDAD DE LOS RESULTADOS ESTADSTICAMENTE SIGNIFICATIVOS EN ENSAYOS ALEATORIOS DE ENFERMEDAD DIVERTICULAR DEL COLON UNA REVISIN SISTEMTICA

ANTECEDENTES:El valor p ha sido criticado por una determinación demasiado simplificada de si existe un efecto del tratamiento. Una alternativa es el Índice de Fragilidad. Es una representación del número mínimo de no eventos que deberían convertirse en eventos para aumentar el valor p por encima de 0,05.OBJETIVO:Determinar el IF de ensayos controlados aleatorios que evalúan la eficacia de las intervenciones para pacientes con enfermedad diverticular desde 2010 para evaluar la solidez de la evidencia actual.FUENTES DE DATOS:Se realizaron búsquedas en MEDLINE, Embase y CENTRAL desde el inicio hasta agosto de 2022.SELECCIÓN DE ESTUDIOS:Los artículos eran elegibles para su inclusión si eran ensayos aleatorizados realizados entre 2010 y 2022 con diseños paralelos de superioridad que evaluaran intervenciones en pacientes con enfermedad diverticular. Sólo se consideraron para su inclusión los ensayos aleatorizados con resultados primarios dicotómicos con un valor de p asociado menor que 0,05.INTERVENCIÓNES:Cualquier intervención quirúrgica o médica para pacientes con enfermedad diverticular.PRINCIPALES MEDIDAS DE VALORACIÓN:El índice de fragilidad se determinó sumando eventos y restando no eventos de los grupos con el menor número de eventos. Se agregaron eventos hasta que el valor p superó 0,05. El menor número de eventos requeridos se consideró índice de fragilidad.RESULTADOS:Después de examinar 1271 citas, 15 ensayos aleatorios cumplieron los criterios de inclusión. Nueve de los estudios evaluaron intervenciones quirúrgicas y seis evaluaron intervenciones médicas. El número medio de pacientes aleatorizados y perdidos durante el seguimiento por ECA fue 92 (DE 35,3) y 9 (DE 11,4), respectivamente. La mediana del índice de fragilidad fue 1 (rango: 0-5). Los índices de fragilidad de los estudios incluidos no se correlacionaron significativamente con ninguna característica del estudio.LIMITACIONES:Muestra pequeña, heterogeneidad y falta de inclusión de estudios con resultados continuos.CONCLUSIONES:Los ensayos aleatorios que evalúan las intervenciones quirúrgicas y médicas para la enfermedad diverticular no son sólidos. Cambiar un solo evento de resultado en la mayoría de los estudios fue suficiente para que un hallazgo estadísticamente significativo del estudio no fuera significativo. (Traducción- Dr. Ingrid Melo ).

Tranexamic Acid Is Safe in Patients with a History of Venous Thromboembolism Undergoing Total Joint Arthroplasty

BACKGROUND

Tranexamic acid (TXA) is increasingly utilized during total knee arthroplasty (TKA) and total hip arthroplasty (THA) to decrease blood loss; however, there are concerns with regard to potential thromboembolic complications, particularly in high-risk patients. This study sought to define a subset of patients at elevated risk for thromboembolic complications following total joint arthroplasty (TJA) and to compare postoperative outcomes between patients who received TXA and those who did not.

METHODS

Patients who underwent primary, elective TJA from 2015 to 2021 were identified in the Premier Healthcare Database. Patients with a history of venous thromboembolism, defined as a history of pulmonary embolism or deep vein thrombosis, were identified and formed the high-risk cohort. Patient demographic characteristics, hospital factors, patient comorbidities, antithrombotic medication use, perioperative blood transfusion, and 90-day complications were assessed and compared between patients who received TXA and those who did not. Univariate regression and multivariable regression were performed to account for potential confounders.

RESULTS

The high-risk cohort comprised 70,759 patients who underwent TJA, of whom 46,074 (65.1%) received TXA and 24,685 (34.9%) did not. After controlling for confounding factors, patients in the TXA cohort had similar risks of pulmonary embolism (adjusted odds ratio [OR], 0.90 [95% confidence interval (CI), 0.79 to 1.02]; p = 0.097), stroke (adjusted OR, 0.97 [95% CI, 0.69 to 1.37]; p = 0.867), and myocardial infarction (adjusted OR, 0.93 [95% CI, 0.69 to 1.24]; p = 0.614) compared with patients who did not receive TXA. Patients who received TXA demonstrated decreased risks of transfusion (adjusted OR, 0.42 [95% CI, 0.38 to 0.46]; p < 0.001) and 90-day readmission (adjusted OR, 0.87 [95% CI, 0.80 to 0.94]; p < 0.001).

CONCLUSIONS

TXA utilization was not associated with an increased risk of postoperative pulmonary embolism, stroke, or myocardial infarction in patients with a history of venous thromboembolism. Furthermore, patients who received TXA had a decreased risk of transfusion and readmission. This evidence suggests that TXA may be safely utilized among select high-risk patients.

LEVEL OF EVIDENCE

Therapeutic Level III . See Instructions for Authors for a complete description of levels of evidence.

The Fragility of Landmark Randomized Controlled Trials in the Plastic Surgery Literature

Background

Randomized controlled trials (RCTs) are integral to the progress of evidenced-based medicine and help guide changes in the standards of care. Although results are traditionally evaluated according to their corresponding P value, the universal utility of this statistical metric has been called into question. The fragility index (FI) has been developed as an adjunct method to provide additional statistical perspective. In this study, we aimed to determine the fragility of 25 highly cited RCTs in the plastic surgery literature.

Methods

A PubMed search was used to identify the 25 highest cited RCTs with statistically significant dichotomous outcomes across 24 plastic surgery journals. Article characteristics were extracted, and the FI of each article was calculated. Additionally, Altmetric scores were determined for each study to determine article attention across internet platforms.

Results

The median FI score across included studies was 4 (2-7.5, interquartile range). The two highest FI scores were 208 and 58, respectively. Four studies (16%) had scores of 0 or 1. Three studies (12%) had scores of 2. All other studies (72%) had FI scores of 3 or higher. The median Altmetric score was 0 (0-3).

Conclusion

The FI can provide additional perspective on the robustness of study results, but like the P value, it should be interpreted in the greater context of other study elements.

An analysis of randomized controlled trials on anal fistula conducted between 2000 and 2020 based on the Fragility Index and Reverse Fragility Index

AIM

The aim of this work was to evaluate the robustness of randomized controlled trials (RCTs) on anal fistula management using the news tools of Fragility Index (FI), Reverse Fragility Index (RFI) and their corresponding fragility quotients.

METHOD

A systematic search was conducted based on the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines which utilized MEDLINE, EMBASE, Cochrane Library and Web of Science databases. Inclusion criteria included RCTs related to the management of anal fistula published from 2000 to 2022 with dichotomous outcomes measures and 1:1 allocation. Calculation of FI and RFI was performed by creating 2 × 2 contingency tables by successively changing one nonevent to an event for each outcome measure until the result was made nonsignificant or significant, respectively. The Fragility Quotients were calculated by dividing the FI or RFI by the total sample size. Fragile results were defined as those with a FI or RFI equal to or less than the number of patients lost to follow-up. Additionally, those with a FI or RFI less than 3 were also considered fragile. Studies were considered extremely fragile if FI was ≤1 or FQ was ≤0.01.

RESULTS

There were 36 RCTs that met our criteria, with 3223 patients. Among these, 19 (53%) were positive RCTs (p < 0.005) and 17 (47%) were negative RCTs (p > 0.05). The median FI was 2 (0-5). The analysis by categorical subgroup showed a strong correlation between FI and the p-value (p = 0.000) and the number of events (p = 0.011). The median RFI was 5 (3.5-9.5) and the subgroup analysis showed a strong correlation between RFI and the p-value (p = 0.000), sample size (0.021) and number needed to treat/number needed to harm (0.000). We considered 63.2% of positive RCTs to be fragile and 35.3% of negative RCTs.

CONCLUSIONS

In the present study we demonstrated the lack of robustness of study findings in published RCTs in the field of anal fistula.