Fragility of overactive bladder medication clinical trials: A systematic review

PURPOSE

Overactive bladder (OAB) syndrome significantly impairs quality of life, often necessitating pharmacological interventions with associated risks. The fragility of OAB trial outcomes, as measured by the fragility index (FI: smallest number of event changes to reverse statistical significance) and quotient (FQ: FI divided by total sample size expressed as a percentage), is critical yet unstudied.

MATERIALS AND METHODS

We conducted a systematic search for randomized controlled trials on OAB medications published between January 2000 and August 2023. Inclusion criteria were trials with two parallel arms reporting binary outcomes related to OAB medications. We extracted trial details, outcomes, and statistical tests employed. We calculated FI and FQ, analyzing associations with trial characteristics through linear regression.

RESULTS

We included 57 trials with a median sample size of 211 participants and a 12% median lost to follow-up. Most studies investigated anticholinergics (37/57, 65%). The median FI/FQ was 5/3.5%. Larger trials were less fragile (median FI 8; FQ 1.0%) compared to medium (FI: 4; FQ 2.5%) and small trials (FI: 4; FQ 8.3%). Double-blinded studies exhibited higher FQs (median 2.9%) than unblinded trials (6.7%). Primary and secondary outcomes had higher FIs (median 5 and 6, respectively) than adverse events (FI: 4). Each increase in 10 participants was associated with a +0.19 increase in FI (p < 0.001).

CONCLUSIONS

A change in outcome for a median of five participants, or 3.5% of the total sample size, could reverse the direction of statistical significance in OAB trials. Studies with larger sample sizes and efficacy outcomes from blinded trials were less fragile.

Statistical Fragility Analysis of Open Reduction Internal Fixation vs Primary Arthrodesis to Treat Lisfranc Injuries: A Systematic Review

BACKGROUND

There is a lack of consensus in the use of open reduction internal fixation (ORIF) vs primary arthrodesis (PA) in the management of Lisfranc injuries. Statistical fragility represents the number of events needed to flip statistical significance and provides context to interpret P values of outcomes from conflicting studies. The current study evaluates the statistical fragility of existing research with an outcome-specific approach to provide statistical clarity to the ORIF vs PA discussion. We hypothesized that statistical fragility analysis would offer clinically relevant insight when interpreting conflicting outcomes regarding ORIF vs PA management of Lisfranc injuries.

METHODS

All comparative studies, RCTs, and case-series investigating ORIF vs PA management of Lisfranc injuries published through October 5, 2023, were identified. Descriptive characteristics, dichotomous outcomes, and continuous outcomes were extracted. Fragility index and continuous fragility index were calculated by the number of event reversals needed to alter significance. Outcomes were categorized by clinical relevance, and median FI and CFI were reported.

RESULTS

A total of 244 studies were screened. Ten studies and 67 outcomes (44 dichotomous, 23 continuous) were included in the fragility analysis. Of the 10 studies, 4 studies claimed PA to correlate with superior outcomes compared to ORIF with regard to functional scores and return to function outcomes. Of these 4 studies, 3 were statistically robust. Six studies claimed PA and ORIF to have no differences in outcomes, in which only 2 studies were statistically robust.

CONCLUSION

The overall research regarding ORIF vs PA is relatively robust compared with other orthopaedic areas of controversy. Although the full statistical context of each article must be considered, studies supporting PA superiority with regard to functional scores and return to function metrics were found to be statistically robust. Outcome-specific analysis revealed moderate fragility in several clinically relevant outcomes such as functional score, return to function, and wound complications.

Water birth: a systematic review and meta-analysis of maternal and neonatal outcomes

OBJECTIVE

This systematic review and meta-analysis aimed to conduct a thorough and contemporary assessment of maternal and neonatal outcomes associated with water birth in comparison with land-based birth.

DATA SOURCES

We conducted a comprehensive search of PubMed, EMBASE, CINAHL, and gray literature sources, from inception to February 28, 2023.

STUDY ELIGIBILITY CRITERIA

We included randomized and nonrandomized studies that assessed maternal and neonatal outcomes in patients who delivered either conventionally or while submerged in water.

METHODS

Pooled unadjusted odds ratios with 95% confidence intervals were calculated using a random-effects model (restricted maximum likelihood method). We assessed the 95% prediction intervals to estimate the likely range of future study results. To evaluate the robustness of the results, we calculated fragility indices. Maternal infection was designated as the primary outcome, whereas postpartum hemorrhage, perineal lacerations, obstetrical anal sphincter injury, umbilical cord avulsion, low Apgar scores, neonatal aspiration requiring resuscitation, neonatal infection, neonatal mortality within 30 days of birth, and neonatal intensive care unit admission were considered secondary outcomes.

RESULTS

Of the 20,642 articles identified, 52 were included in the meta-analyses. Based on data from observational studies, water birth was not associated with increased probability of maternal infection compared with land birth (10 articles, 113,395 pregnancies; odds ratio, 0.93; 95% confidence interval, 0.76-1.14). Patients undergoing water birth had decreased odds of postpartum hemorrhage (21 articles, 149,732 pregnancies; odds ratio, 0.80; 95% confidence interval, 0.68-0.94). Neonates delivered while submerged in water had increased odds of cord avulsion (10 articles, 91,504 pregnancies; odds ratio, 1.75; 95% confidence interval, 1.38-2.24) and decreased odds of low Apgar scores (21 articles, 165,917 pregnancies; odds ratio, 0.69; 95% confidence interval, 0.58-0.82), neonatal infection (15 articles, 53,635 pregnancies; odds ratio, 0.64; 95% confidence interval, 0.42-0.97), neonatal aspiration requiring resuscitation (19 articles, 181,001 pregnancies; odds ratio, 0.60; 95% confidence interval, 0.43-0.84), and neonatal intensive care unit admission (30 articles, 287,698 pregnancies; odds ratio, 0.56; 95% confidence interval, 0.45-0.70).

CONCLUSION

When compared with land birth, water birth does not appear to increase the risk of most maternal and neonatal complications. Like any other delivery method, water birth has its unique considerations and potential risks, which health care providers and expectant parents should evaluate thoroughly. However, with proper precautions in place, water birth can be a reasonable choice for mothers and newborns, in facilities equipped to conduct water births safely.

Platelet-Rich Plasma Versus Alternative Injections for Osteoarthritis of the Knee: A Systematic Review and Statistical Fragility Index-Based Meta-analysis of Randomized Controlled Trials

BACKGROUND

Based in part on the results of randomized controlled trials (RCTs) that suggest a beneficial effect over alternative treatment options, the use of platelet-rich plasma (PRP) for the management of knee osteoarthritis (OA) is widespread and increasing. However, the extent to which these studies are vulnerable to slight variations in the outcomes of patients remains unknown.

PURPOSE

To evaluate the statistical fragility of conclusions from RCTs that reported outcomes of patients with knee OA who were treated with PRP versus alternative nonoperative management strategies.

STUDY DESIGN

Systematic review and meta-analysis; Level of evidence, 2.

METHODS

All RCTs comparing PRP with alternative nonoperative treatment options for knee OA were identified. The fragility index (FI) and reverse FI were applied to assess the robustness of conclusions regarding the efficacy of PRP for knee OA. Meta-analyses were performed to determine the minimum number of patients from ≥1 trials included in the meta-analysis for which a modification on the event status would change the statistical significance of the pooled treatment effect.

RESULTS

In total, this analysis included outcomes from 1993 patients with a mean ± SD age of 58.0 ± 3.8 years. The mean number of events required to reverse significance of individual RCTs (FI) was 4.57 ± 5.85. Based on random-effects meta-analyses, PRP demonstrated a significantly higher rate of successful outcomes when compared with hyaluronic acid (P = .002; odds ratio [OR], 2.19; 95% CI, 1.33-3.62), as well as higher rates of patient-reported symptom relief (P = .019; OR, 1.55; 95% CI, 1.07-2.24), not requiring a reintervention after the initial injection treatment (P = .002; OR, 2.17; 95% CI, 1.33-3.53), and achieving the minimal clinically important difference (MCID) for pain improvement (P = .007; OR, 6.19; 95% CI, 1.63-23.42) when compared with all alternative nonoperative treatments. Overall, the mean number of events per meta-analysis required to change the statistical significance of the pooled treatment effect was 8.67 ± 4.50.

CONCLUSION

Conclusions drawn from individual RCTs evaluating PRP for knee OA demonstrated slight robustness. On meta-analysis, PRP demonstrated a significant advantage over hyaluronic acid as well as improved symptom relief, lower rates of reintervention, and more frequent achievement of the MCID for pain improvement when compared with alternative nonoperative treatment options. Statistically significant pooled treatment effects evaluating PRP for knee OA are more robust than approximately half of all comparable meta-analyses in medicine and health care. Future RCTs and meta-analyses should consider reporting FIs and fragility quotients to facilitate interpretation of results in their proper context.

Interpreting the Current Literature on Outcomes of Robotic-Assisted Versus Conventional Total Knee Arthroplasty Using Fragility Analysis: A Systematic Review and Cross-Sectional Study of Randomized Controlled Trials

BACKGROUND

Fragility analysis is a method of further characterizing outcomes in terms of the stability of statistical findings. This study assesses the statistical fragility of recent randomized controlled trials (RCTs) evaluating robotic-assisted versus conventional total knee arthroplasty (RA-TKA versus C-TKA).

METHODS

We queried PubMed for RCTs comparing alignment, function, and outcomes between RA-TKA and C-TKA. Fragility index (FI) and reverse fragility index (RFI) (collectively, "FI") were calculated for dichotomous outcomes as the number of outcome reversals needed to change statistical significance. Fragility quotient (FQ) was calculated by dividing the FI by the sample size for that outcome event. Median FI and FQ were calculated for all outcomes collectively as well as for each individual outcome. Subanalyses were performed to assess FI and FQ based on outcome event type and statistical significance, as well as study loss to follow-up and year of publication.

RESULTS

The overall median FI was 3.0 (interquartile range, [IQR] 1.0 to 6.3) and the median reverse fragility index was 3.0 (IQR 2.0 to 4.0). The overall median FQ was 0.027 (IQR 0.012 to 0.050). Loss to follow-up was greater than FI for 23 of the 38 outcomes assessed.

CONCLUSIONS

A small number of alternative outcomes is often enough to reverse the statistical significance of findings in RCTs evaluating dichotomous outcomes in RA-TKA versus C-TKA. We recommend reporting FI and FQ alongside P values to improve the interpretability of RCT results.

The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

BACKGROUND

Evidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.

PURPOSE

To calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).

STUDY DESIGN

Meta-analysis; Level of evidence, 2.

METHODS

A search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.

RESULTS

There were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial's sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).

CONCLUSION

More than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

Statistical Significance Versus Clinical Relevance: A Head-to-Head Comparison of the Fragility Index and Relative Risk Index

Background In biostatistics, assessing the fragility of research findings is crucial for understanding their clinical significance. This study focuses on the fragility index, unit fragility index, and relative risk index as measures to evaluate statistical fragility. The fragility indices assess the susceptibility of p-values to change significance with minor alterations in outcomes within a 2x2 contingency table. In contrast, the relative risk index quantifies the deviation of observed findings from therapeutic equivalence, the point at which the relative risk equals 1. While the fragility indices have intuitive appeal and have been widely applied, their behavior across a wide range of contingency tables has not been rigorously evaluated. Methods Using a Python software program, a simulation approach was employed to generate random 2x2 contingency tables. All tables under consideration exhibited p-values < 0.05 according to Fisher's exact test. Subsequently, the fragility indices and the relative risk index were calculated. To account for sample size variations, the indices were divided by the sample size to give fragility and risk quotients. A correlation matrix assessed the collinearity between each metric and the p-value. Results The analysis included 2,000 contingency tables with cell counts ranging from 20 to 480. Notably, the formulas for calculating the fragility indices encountered limitations when cell counts approached zero or duplicate cell counts hindered standardized application. The correlation coefficients with p-values were as follows: unit fragility index (-0.806), fragility index (-0.802), fragility quotient (-0.715), unit fragility quotient (-0.695), relative risk index (-0.403), and risk quotient (-0.261). Conclusion The fragility indices and fragility quotients demonstrated a strong correlation with p-values below 0.05, while the relative risk index and relative risk quotient exhibited a weak association with p-values below this threshold. This implies that the fragility indices offer limited additional information beyond the p-value alone. In contrast, the relative risk index and risk quotient exhibit independence from the p-value, indicating that they may provide important additional information about statistical fragility by evaluating the divergence of observed results from therapeutic equivalence, irrespective of the p-value-based statistical significance.

Randomized Controlled Trials in ICU in the Four Highest-Impact General Medicine Journals

OBJECTIVE

To study ICU trials published in the four highest-impact general medicine journals by comparing them with concurrently published non-ICU trials in the same journals.

DATA SOURCES

PubMed was searched for randomized controlled trials (RCTs) published between January 2014 and October 2021 in the New England Journal of Medicine , The Lancet , the Journal of the American Medical Association , and the British Medical Journal.

STUDY SELECTION

Original RCT publications investigating any type of intervention in any patient population.

DATA EXTRACTION

ICU RCTs were defined as RCTs exclusively including patients admitted to the ICU. Year and journal of publication, sample size, study design, funding source, study outcome, type of intervention, Fragility Index (FI), and Fragility Quotient were collected.

DATA SYNTHESIS

A total of 2,770 publications were screened. Of 2,431 original RCTs, 132 (5.4%) were ICU RCTs, gradually rising from 4% in 2014 to 7.5% in 2021. ICU RCTs and non-ICU RCTs included a comparable number of patients (634 vs 584, p = 0.528). Notable differences for ICU RCTs were the low occurrence of commercial funding (5% vs 36%, p < 0.001), the low number of RCTs that reached statistical significance (29% vs 65%, p < 0.001), and the low FI when they did reach significance (3 vs 12, p = 0.008).

CONCLUSIONS

In the last 8 years, RCTs in ICU medicine made up a meaningful, and growing, portion of RCTs published in high-impact general medicine journals. In comparison with concurrently published RCTs in non-ICU disciplines, statistical significance was rare and often hinged on the outcome events of just a few patients. Increased attention should be paid to realistic expectations of treatment effects when designing ICU RCTs to detect differences in treatment effects that are reliable and clinically relevant.

An analysis of randomized controlled trials on anal fistula conducted between 2000 and 2020 based on the Fragility Index and Reverse Fragility Index

AIM

The aim of this work was to evaluate the robustness of randomized controlled trials (RCTs) on anal fistula management using the news tools of Fragility Index (FI), Reverse Fragility Index (RFI) and their corresponding fragility quotients.

METHOD

A systematic search was conducted based on the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines which utilized MEDLINE, EMBASE, Cochrane Library and Web of Science databases. Inclusion criteria included RCTs related to the management of anal fistula published from 2000 to 2022 with dichotomous outcomes measures and 1:1 allocation. Calculation of FI and RFI was performed by creating 2 × 2 contingency tables by successively changing one nonevent to an event for each outcome measure until the result was made nonsignificant or significant, respectively. The Fragility Quotients were calculated by dividing the FI or RFI by the total sample size. Fragile results were defined as those with a FI or RFI equal to or less than the number of patients lost to follow-up. Additionally, those with a FI or RFI less than 3 were also considered fragile. Studies were considered extremely fragile if FI was ≤1 or FQ was ≤0.01.

RESULTS

There were 36 RCTs that met our criteria, with 3223 patients. Among these, 19 (53%) were positive RCTs (p < 0.005) and 17 (47%) were negative RCTs (p > 0.05). The median FI was 2 (0-5). The analysis by categorical subgroup showed a strong correlation between FI and the p-value (p = 0.000) and the number of events (p = 0.011). The median RFI was 5 (3.5-9.5) and the subgroup analysis showed a strong correlation between RFI and the p-value (p = 0.000), sample size (0.021) and number needed to treat/number needed to harm (0.000). We considered 63.2% of positive RCTs to be fragile and 35.3% of negative RCTs.

CONCLUSIONS

In the present study we demonstrated the lack of robustness of study findings in published RCTs in the field of anal fistula.

Statistical Fragility of Randomized Controlled Trials Evaluating Rehabilitation After Arthroscopic Rotator Cuff Repair

Background

Clinical decision-making often relies on evidence-based medicine, derived from objective data with conventional and rigorous statistical tests to evaluate significance. The literature surrounding rehabilitation after rotator cuff repair (RCR) is conflicting, with no defined standard of practice.

Purpose

To determine the fragility index (FI) and the fragility quotient (FQ) of randomized controlled trials (RCTs) evaluating rehabilitation protocols after RCR.

Study Design

Systematic review.

Methods

A systematic review was performed according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines by searching the PubMed, Cochrane Library, and Embase databases for RCTs evaluating rehabilitation protocols after arthroscopic RCRs from 2000 to June 1, 2022. The FI was determined by manipulating the dichotomous outcome events from each article until a reversal of significance with 2 × 2 contingency tables was achieved. The FQ was determined by dividing the FI by the sample size.

Results

Fourteen RCTs with 48 dichotomous outcomes were ultimately included for analysis. The mean FI for the included dichotomous outcomes was 4 (interquartile range, 3-6), suggesting that the reversal of 4 events is required to change study significance. The mean FQ was 0.048. Of the RCTs that reported data regarding loss to follow-up, most studies (58.5%) indicated that >4 patients had been lost to follow-up.

Conclusion

The results of RCT studies of RCR rehabilitation protocols are moderately fragile, something clinicians should be aware of when implementing study results into practice. We recommend the inclusion of FI and FQ in addition to standard P values when reporting statistical results in future RCTs with dichotomous outcome variables on this topic.

Assessing the robustness of COVID-19 vaccine efficacy trials: systematic review and meta-analysis, January 2023

BackgroundVaccines play a crucial role in the response to COVID-19 and their efficacy is thus of great importance.AimTo assess the robustness of COVID-19 vaccine efficacy (VE) trial results using the fragility index (FI) and fragility quotient (FQ) methodology.MethodsWe conducted a Cochrane and PRISMA-compliant systematic review and meta-analysis of COVID-19 VE trials published worldwide until 22 January 2023. We calculated the FI and FQ for all included studies and assessed their associations with selected trial characteristics using Wilcoxon rank sum tests and Kruskal-Wallis H tests. Spearman correlation coefficients and scatter plots were used to quantify the strength of correlation of FIs and FQs with trial characteristics.ResultsOf 6,032 screened records, we included 40 trials with 54 primary outcomes, comprising 909,404 participants with a median sample size per outcome of 13,993 (interquartile range (IQR): 8,534-25,519). The median FI and FQ was 62 (IQR: 22-123) and 0.50% (IQR: 0.24-0.92), respectively. FIs were positively associated with sample size (p < 0.001), and FQs were positively associated with type of blinding (p = 0.023). The Spearman correlation coefficient for FI with sample size was moderately strong (0.607), and weakly positive for FI and FQ with VE (0.138 and 0.161, respectively).ConclusionsThis was the largest study on trial robustness to date. Robustness of COVID-19 VE trials increased with sample size and varied considerably across several other important trial characteristics. The FI and FQ are valuable complementary parameters for the interpretation of trial results and should be reported alongside established trial outcome measures.

Introducing the fragility index-A case study using the Term Breech Trial

The fragility index (FI) is a sensitivity analysis of the statistically significant result of a clinical study. It is the number of hypothetical changes in the primary event of one of the two cohorts in a 1-to-1 comparative trial to render the statistically significant result non-significant (ie, to alter the P-value from ≤0.05 to >0.05). The FI can be compared with the patient drop-out rates and protocol violations, which, if much higher than the FI, may arguably suggest less robustness/stability of the trial's results. To illustrate the concept, we have chosen the Term Breech Trial (TBT) as a case study. The TBT results favor planned cesarean birth, as opposed to planned vaginal delivery, in the term singleton fetus with breech presentation. Our analysis shows that the FI of the TBT is 21, which is small in comparison to the number (hundreds) of protocol violations present. Some experts have suggested the inclusion of the FI in data analysis and subsequent discussion of clinical trial data. Routine use of such a metric may be valuable in encouraging readers to maintain a healthy degree of skepticism, especially when interpreting trial results which may directly influence clinical practice.

Empirical assessment of fragility index based on a large database of clinical studies in the Cochrane Library

RATIONALE AIMS AND OBJECTIVES

The fragility index (FI) and fragility quotient (FQ) are increasingly used measures for assessing the robustness of clinical studies with binary outcomes in terms of statistical significance. The FI is the minimum number of event status modifications that can alter a study result's statistical significance (or nonsignificance), and the FQ is calculated as the FI divided by the study's total sample size. The literature has no widely recognized criteria for interpreting the fragility measures' magnitudes. This article aims to provide an empirical assessment for the FI and FQ based on a large database of clinical studies in the Cochrane Library.

METHODS

We explored the overall empirical distributions of the FI and FQ based on five common methods (Fisher's exact test, χ2 test, risk difference, odds ratio, and relative risk) for determining statistical significance of binary outcomes in clinical research. We also considered three different scenarios for the FI calculation and evaluated the relationship between p values and FIs or FQs using Spearman'sρ $\rho $ . Finally, we summarized empirical thresholds based on the overall distributions of the FI and FQ to facilitate their interpretations in future research.

RESULTS

For about 20% of studies with significant results, the statistical significance was changed after modifying the event status of only one participant. Studies with significant results were considered slightly fragile if the significance hinged on the statuses of about five events. Studies were extremely fragile if FI≤ $\le $ 1 or FQ≤ $\le $ 0.01. The FIs were strongly correlated with p values for significant studies, while Spearman'sρ $\rho $ varied according to the total sample sizes of studies.

CONCLUSIONS

The statistical significance of clinical studies could be changed after modifying a few events' statuses. Many studies' findings are fairly fragile. The distributions of the FI and FQ provide insights for appraising the robustness of evidence in clinical decision-making.

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.

Spin and fragility in randomised controlled trials in the anaesthesia literature: a systematic review

BACKGROUND

Given variable frequency of misleading reports and the potential for spin (a way of describing results that can mislead readers) to influence interpretation of randomised controlled trials (RCTs), we have undertaken a spin reassessment. We evaluated the quality of recent literature in anaesthesia journals by assessing the presence of spin and calculating the fragility index.

METHODS

This systematic review of randomised trials was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We searched via PubMed® from January 1, 2019 to January 1, 2021 to identify all RCTs published in one of the 20 anaesthesia journals with the highest journal impact factors during this time. Four pairs of reviewers assessed articles independently for eligibility using a piloted electronic data extraction form. They assessed the presence of spin in statistically negative RCTs and calculated the fragility index for statistically positive RCTs.

RESULTS

Of the 802 screened records, 162 (20%) articles were analysed for spin, and 65 (8%) trials were analysed for fragility index. For the statistically negative studies, 66 articles (40%) presented spin; 89% of these occurrences of spin were described in the conclusion of the abstract. The primary type of spin was the highlight of secondary outcomes (67%). For statistically positive trials, the median fragility index was 4 [1-8].

CONCLUSIONS

This systematic review showed that 40% of statistically negative trials in high-impact anaesthesia journals could mislead readers. For statistically positive RCTs, the results relied on few subjects, with a median fragility index of 4 [1-8]. Efforts must be continued to reduce spin and fragility in the medical literature.

The Statistical Fragility of Foot and Ankle Surgery Randomized Controlled Trials

Fragility index (FI) is a metric used to interpret the results of randomized controlled trials (RCTs), and describes the number of subjects that would need to be switched from event to non-event for a result to no longer be significant. Studies that analyze FI of RCTs in various orthopedic subspecialties have shown the RCTs to be largely underpowered and highly fragile. However, FI has not been assessed in foot and ankle RCTs. The MEDLINE and Embase online databases were searched from 1/1/2011 through 11/19/2021 for RCTs involving foot and ankle conditions. FI, fragility quotient (FQ), and difference between the FI and number of subjects lost to follow-up was calculated. Spearman correlation was performed to determine the relationship between sample size and FI. Overall, 1262 studies were identified of which 18 were included in the final analysis. The median sample size was 65 (interquartile range [IQR] 57-95.5), the median FI was 2 (IQR 1-2.5), and the median FQ was 0.026 (IQR 0.012-0.033). Ten of 15 (67%) studies with non-zero FI values had FI values less than the number of subjects lost to follow-up. There was linear association between FI and sample size (R² = 0.495, p-value: .031). This study demonstrates that RCTs in the field of foot and ankle surgery are highly fragile, similar to other orthopedic subspecialties.

The Statistical Fragility of Platelet-Rich Plasma as Treatment for Plantar Fasciitis: A Systematic Review and Simulated Fragility Analysis

Background

Plantar fasciitis (PF) is the most common cause of heel pain and can be a source of extensive physical disability and financial burden. Platelet-rich plasma (PRP) offers a potentially definitive, regenerative treatment modality that, if effective, could change the current paradigm of PF care. However, randomized controlled trials (RCTs) on the clinical benefits of PRP for refractory PF offer inconsistent conclusions, potentially because of the broader limitations of using P value thresholds to declare statistical and clinical significance. In this study, we use the Continuous Fragility Index (CFI) and Quotient (CFQ) to appraise the statistical robustness of data from RCTs evaluating PRP for treatment of PF.

Methods

RCTs comparing outcomes after PRP injection vs alternative treatment in patients with chronic PF were evaluated. Representative simulated data sets were generated for each reported outcome event using summary statistics. The CFI was determined by manipulating each data set until reversal of significance (α=0.05) was achieved. The corresponding CFQ was calculated by dividing the CFI by the sample size.

Results

Of 259 studies screened, 20 studies (59 outcome events) were included in this analysis. From these simulations, the median CFI for all events was 9, suggesting that varying the treatment of 9 patients would be required to reverse trial significance. The corresponding CFQ was 0.177. Studies with reported P value <.05 were more statistically fragile (CFI=10, CFQ=0.122) than studies with reported P value >.05 (CFI=5, CFQ=0.179). Of 36 outcome events reporting lost to follow-up data, 10 events (27.8%) lost ≥9 patients.

Conclusion

Our findings suggest that, on average, the statistical fragility of RCTs evaluating PRP for nonoperative PF therapy is at least comparable to that of the sports medicine literature. However, several included studies had concerningly low simulated fragility scores. Orthopaedic surgeons may benefit from preferentially relying on studies with higher CFI and CFQ values when evaluating the utility of PRP for chronic PF in their own clinical practice. Given the importance of RCT data in clinical decision making, fragility indices could help give context to the stability of statistical findings.

Level of Evidence

Level I, systematic review.

The Fragility of Statistical Significance in Patellofemoral Instability Research: A Systematic Review

BACKGROUND

Fragility analysis is increasingly utilized to evaluate the robustness of results within the orthopaedic literature and has frequently revealed instability of reported outcomes.

PURPOSE/HYPOTHESIS

The purpose of this investigation was to utilize a fragility analysis to evaluate the stability of reported results in the patellofemoral instability (PFI) literature. We hypothesized the demonstration of significant fragility in patellofemoral research to be similar to that identified throughout other areas of the orthopaedic literature.

STUDY DESIGN

Systematic review; Level of evidence, 4.

METHODS

The PubMed database was queried from January 1, 2000, to October 10, 2020 for comparative trials in 10 prominent orthopaedic journals that reported dichotomous outcomes related to the management of PFI. The fragility index (FI) and the fragility quotient (FQ) were calculated for each individual outcome event, and the overall FI and FQ were determined for all included studies.

RESULTS

A total of 22 comparative studies comprising 11 randomized controlled trials and 11 nonrandomized trials were included for the analysis. A total of 75 outcome events underwent a fragility analysis and revealed a median FI and FQ of 3 (interquartile range [IQR], 1-5) and 0.043 (IQR, 0.018-0.081), respectively. Also 27% of included studies reported loss to follow-up greater than the overall FI, therefore suggesting the maintenance of the follow-up may have resulted in the reversal of significance.

CONCLUSION

The result of the comprehensive fragility analysis demonstrated a lack of robustness in PFI research with the alteration of only a few outcome events required to reverse statistical significance. We therefore recommend the triple reporting of the P value, the FI, and the FQ to aid in the interpretation of the statistical integrity of future comparative trials in the PFI literature.

The Statistical Fragility of Operative vs Nonoperative Management for Achilles Tendon Rupture: A Systematic Review of Comparative Studies

BACKGROUND

The statistical significance of randomized controlled trials (RCTs) and comparative studies is often conveyed utilizing the P value. However, P values are an imperfect measure and may be vulnerable to a small number of outcome reversals to alter statistical significance. The interpretation of the statistical strength of these studies may be aided by the inclusion of a Fragility Index (FI) and Fragility Quotient (FQ). This study examines the statistical stability of studies comparing operative vs nonoperative management for Achilles tendon rupture.

METHODS

A systematic search was performed of 10 orthopaedic journals between 2000 and 2021 for comparative studies focusing on management of Achilles tendon rupture reporting dichotomous outcome measures. FI for each outcome was determined by the number of event reversals necessary to alter significance (P < .05). FQ was calculated by dividing the FI by the respective sample size. Additional subgroup analyses were performed.

RESULTS

Of 8020 studies screened, 1062 met initial search criteria with 17 comparative studies ultimately included for analysis, 10 of which were RCTs. A total of 40 outcomes were examined. Overall, the median FI was 2.5 (interquartile range [IQR] 2-4), the mean FI was 2.90 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). The FI was less than the number of patients lost to follow-up for 78% of outcomes.

CONCLUSION

Studies examining the efficacy of operative vs nonoperative management of Achilles tendon rupture may not be as statistically stable as previously thought. The average number of outcome reversals needed to alter the significance of a given study was 2.90. Future analyses may benefit from the inclusion of a fragility index and a fragility quotient in their statistical analyses.

Fragility of randomized controlled trials on treatment of proximal humeral fracture

BACKGROUND

Proximal humeral fracture represents an increasingly common pathology with evaluation and treatment often guided by evidence from randomized controlled trials (RCTs), but the strength of an RCT must be considered in this process. The purpose of this study was to evaluate the strength of outcomes in RCTs on the management of proximal humeral fractures using the fragility index (FI), a method used with statistically significant dichotomous outcomes to assess the number of patients that would change an outcome measure from significant (P ≤ .05) to nonsignificant if the patient outcome changed. We also aimed to correlate the FI with other measures of study strength.

METHODS

A systematic review was performed using Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines to evaluate RCTs on the management of proximal humeral fractures. The PubMed, Ovid MEDLINE, Web of Science, and Embase databases were searched from database inception to May 2021. RCTs with at least 1 statistically significant (P ≤ .05) dichotomous outcome were included. The FI was calculated for each included trial using the Fisher exact test. The FI was correlated with the study sample size and journal impact factor.

RESULTS

Ten RCTs reporting on 656 patients and published between 2011 and 2020 were included. The median patient sample size was 67 (mean, 65.6; range, 40-86). Complications were the most commonly reported dichotomous statistically significant outcome. The median FI was 1 (mean, 2.6; range, 0-18), with 4 studies having an FI of 0. A median FI of 1 indicates that 1 patient experiencing an alternative outcome or having not been lost to follow-up could have changed the pertinent conclusions of the trial for a given outcome. The median number of patients lost to follow-up was 3 (mean, 4.9; range, 0-16) and exceeded the FI in 50% of studies. There was no correlation between the FI and sample size (Spearman coefficient = 0.0592, P = .865) or between the FI and journal impact factor (Spearman coefficient = -0.0229, P = .522).

CONCLUSION

In most studies of proximal humeral fractures, only 1 or 2 patients experiencing an alternative outcome or lost to follow-up would change the conclusions for the dichotomous outcome studied. Although the FI cannot be used to assess continuous variables, which are often the primary outcome variables of RCTs, it does offer an additional unique measure of study strength that surgeons should consider when evaluating RCTs.

The Fragility Index of Statistically Significant Findings From Randomized Controlled Trials Comparing the Management Strategies of Anterior Shoulder Instability

BACKGROUND

Debate centering on the management of anterior shoulder instability (ASI) in recent years has led to many randomized controlled trials (RCTs) being published on the topic. The fragility index (FI) has subsequently emerged as a novel method of assessing significant findings reported in RCTs, particularly those with small sample sizes.

PURPOSE

To evaluate the FI of statistically significant findings in RCTs that reported the outcomes of management strategies of patients with ASI.

STUDY DESIGN

Systematic review; Level of evidence, 1.

METHODS

Using PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, 2 independent reviewers performed a systematic review of RCTs focusing on the outcomes of management strategies of patients with ASI. There were 3 main categories of RCTs included: (1) nonoperative management in internal rotation (IR) versus external rotation (ER), (2) nonoperative management versus a surgical intervention, and (3) surgical management with arthroscopic Bankart repair versus open Bankart repair. The Fisher exact test was utilized to calculate the FI for the reversal of statistical significance in all statistically significant dichotomous outcomes.

RESULTS

A total of 21 RCTs were included, including 1589 shoulders (mean age, 29.4 years) with a mean follow-up of 26.8 months. There were 10 RCTs (831 shoulders) that reported outcomes after the nonoperative management of ASI in IR versus ER, with a mean FI of 6.8. There were 5 RCTs (324 shoulders) that reported outcomes comparing the nonoperative and operative management of ASI, with a mean FI of 3.5. There were 6 RCTs (434 shoulders) that reported outcomes after the operative management of ASI with either arthroscopic Bankart repair or open Bankart repair, with a mean FI of 9.6.

CONCLUSION

The overall FI of RCTs reporting the outcomes of management strategies for patients with ASI was high, suggesting a moderate fragility of statistically significant outcomes including recurrence, revision stabilization, and return to play.

The Statistical Fragility of Single-Bundle vs Double-Bundle Autografts for ACL Reconstruction: A Systematic Review of Comparative Studies

Background:

The statistical significance of a given study outcome can be liable to small changes in findings. P values are common, but imperfect statistical methods to convey significance, and inclusion of the fragility index (FI) and fragility quotient (FQ) may provide a clearer perception of statistical strength.

Purpose/Hypothesis:

The purpose was to examine the statistical stability of studies comparing primary single-bundle to double-bundle anterior cruciate ligament reconstruction (ACLR) utilizing autograft and independent tunnel drilling. It was hypothesized that the study findings would be vulnerable to a small number of outcome event reversals, often less than the number of patients lost to follow-up.

Study Design:

Systematic review; Level of evidence, 2.

Methods:

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, the authors searched PubMed for comparative studies and randomized controlled trials (RCTs) published in select journals, based on impact factor, between 2005 and 2020. Risk-of-bias assessment and methodology scoring were conducted for the included studies. A total of 48 dichotomous outcome measures were examined for possible event reversals. The FI for each outcome was determined by the number of event reversals necessary to alter significance. The FQ was calculated by dividing the FI by the respective sample size.

Results:

Of the 1794 studies screened, 15 comparative studies were included for analysis; 13 studies were RCTs. Overall, the mean FI and FQ were 3.14 (IQR, 2-4) and 0.050 (IQR, 0.032-0.062), respectively. For 72.9% of outcomes, the FI was less than the number of patients lost to follow-up.

Conclusion:

Studies comparing single-bundle versus double-bundle ACLR may not be as statistically stable as previously thought. Comparative studies and RCTs are at substantial risk for statistical fragility, with few event reversals required to alter significance. The reversal of fewer than 4 outcome events in a treatment group can alter the statistical significance of a given result; this is commonly less than the number of patients lost to follow-up. Future comparative study analyses might consider including FI and FQ with P values in their statistical analysis.

The Fragility Index of Randomized Controlled Trials for Preterm Neonates

BACKGROUND

As a metric to determine the robustness of trial results, the fragility index (FI) is the number indicating how many patients would be required to reverse the significant results. This study aimed to calculate the FI in randomized controlled trials (RCTs) involving premature.

METHODS

Trials were included if they had a 1:1 study design, reported statistically significant dichotomous outcomes, and had an explicitly stated sample size or power calculation. The FI was calculated for binary outcomes using Fisher's exact test, and the FIs of subgroups were compared. Spearman's correlation was applied to determine correlations between the FI and study characteristics.

RESULTS

Finally, 66 RCTs were included in the analyses. The median FI for these trials was 3.00 (interquartile range [IQR]: 1.00-5.00), with a median fragility quotient of 0.014 (IQR: 0.008-0.028). FI was ≤ 3 in 42 of these 66 RCTs (63.6%), and in 42.4% (28/66) of the studies, the number of patients lost to follow-up was greater than that of the FI. Significant differences were found in the FI among journals (p = 0.011). We observed that FI was associated with the sample size, total number of events, and reported p-values (r _s = 0.437, 0.495, and -0.857, respectively; all p < 0.001).

CONCLUSION

For RCTs in the premature population, a median of only three events was needed to change from a "non-event" to "event" to render a significant result non-significant, indicating that the significance may hinge on a small number of events.

Fragility indices for only sufficiently likely modifications

The fragility index is a clinically meaningful metric based on modifying patient outcomes that is increasingly used to interpret the robustness of clinical trial results. The fragility index relies on a concept that explores alternative realizations of the same clinical trial by modifying patient measurements. In this article, we propose to generalize the fragility index to a family of fragility indices called the incidence fragility indices that permit only outcome modifications that are sufficiently likely and provide an exact algorithm to calculate the incidence fragility indices. Additionally, we introduce a far-reaching generalization of the fragility index to any data type and explain how to permit only sufficiently likely modifications for nondichotomous outcomes. All of the proposed methodologies follow the fragility index concept.

The Fragility of Statistical Significance in Cartilage Restoration of the Knee: A Systematic Review of Randomized Controlled Trials

OBJECTIVE

The purpose of this study was to utilize fragility analysis to assess the robustness of randomized controlled trials (RCTs) evaluating the management of articular cartilage defects of the knee. We hypothesize that the cartilage restorative literature will be fragile with the reversal of only a few outcome events required to change statistical significance.

DESIGN

RCTs from 11 orthopedic journals indexed on PubMed from 2000 to 2020 reporting dichotomous outcome measures relating to the management of articular cartilage defects of the knee were included. The Fragility Index (FI) for each outcome was calculated through the iterative reversal of a single outcome event until significance was reversed. The Fragility Quotient (FQ) was calculated by dividing each FI by study sample size. Additional statistical analysis was performed to provide median FI and FQ across subgroups.

RESULTS

Nineteen RCTs containing 60 dichotomous outcomes were included for analysis. The FI and FQ of all outcomes was 4 (IQR 2-7) and 0.067 (IQR 0.034-0.096), respectively. The average number of patients lost to follow-up (LTF) was 3.9 patients with 15.8% of the included studies reporting LTF greater than or equal to 4, the FI of all included outcomes.

CONCLUSIONS

The orthopedic literature evaluating articular cartilage defects of the knee is fragile as the reversal of relatively few outcome events may alter the significance of statistical findings. We therefore recommend comprehensive fragility analysis and triple reporting of the P value, FI, and FQ to aid in the interpretation and contextualization of clinical findings reported in the cartilage restoration literature.

Fragility Index and Fragility Quotient in Randomized Controlled Trials on Corticosteroids in ARDS Due to COVID-19 and Non-COVID-19 Etiology

Background: The effectiveness of corticosteroids in acute respiratory distress syndrome (ARDS) and COVID-19 still remains uncertain. Since ARDS is due to a hyperinflammatory response to a direct injury, we decided to perform a meta-analysis and an evaluation of robustness of randomised clinical trials (RCTs) investigating the impact of corticosteroids on mortality in ARDS in both COVID-19 and non-COVID-19 patients. We conducted a systematic search of the literature from inception up to 30 October 2020, using the MEDLINE database and the PubMed interface. We evaluated the fragility index (FI) of the included RCTs using a two-by-two contingency table and the p-value produced by the Fisher exact test; the fragility quotient (FQ) was calculated by dividing the FI score by the total sample size of the trial. Results: Thirteen RCTs were included in the analysis; five of them were conducted in COVID-19 ARDS, including 7692 patients, while 8 RCTS were performed in non-COVID ARDS with 1091 patients evaluated. Three out of eight RCTs in ARDS had a FI > 0 while 2 RCTs out of five in COVID-19 had FI > 0. The median of FI for ARDS was 0.625 (0.47) while the median of FQ was 0.03 (0.014). The median of FI for COVID-19 was 6 (2) while the median of FQ was 0.059 (0.055). In this systematic review, we found that FI and FQ of RCTs evaluating the use of corticosteroids in ARDS and COVID-19 were low.

The Statistical Fragility of Platelet-Rich Plasma in Rotator Cuff Surgery: A Systematic Review and Meta-analysis

BACKGROUND

The practice of evidence-based medicine relies on objective data to guide clinical decision-making with specific statistical thresholds conveying study significance.

PURPOSE

To determine the utility of applying the fragility index (FI) and the fragility quotient (FQ) analysis to randomized controlled trials (RCTs) evaluating the utilization of platelet-rich plasma (PRP) in rotator cuff repairs (RCRs).

STUDY DESIGN

Systematic review and meta-analysis.

METHODS

RCTs pertaining to the utilization of PRP in surgical RCRs published in 13 peer-reviewed journals from 2000 to 2020 were evaluated. The FI was determined by manipulating each reported outcome event until a reversal of significance was appreciated. The associated FQ was determined by dividing the FI by the sample size.

RESULTS

Of the 9746 studies screened, 19 RCTs were ultimately included for analysis. The overall FI incorporating all 19 RCTs was only 4, suggesting that the reversal of only 4 events is required to change study significance. The associated FQ was determined as 0.092. Of the 43 outcome events reporting lost to follow-up data, 13 (30.2%) represented lost to follow-up >4.

CONCLUSION

Our analysis suggests that RCTs evaluating PRP for surgical RCRs may lack statistical stability with only a few outcome events required to alter trial significance. Therefore, we recommend the reporting of an FI and an FQ in conjunction with P value analysis to carefully interpret the integrity of statistical stability in future comparative trials.

CLINICAL RELEVANCE

Clinical decisions are often informed by statistically significant results. Thus, a true understanding of the robustness of the statistical findings informing clinical decision-making is of critical importance.

Pitfalls in Study Interpretation

Background: The goal of a randomized or observational study is to develop an unbiased and reliable answer to a therapeutic question. However, there are multiple pitfalls in the reporting and interpretation of data that can compromise our ability to evaluate the pragmatism and the effectiveness of the intervention being studied. Researchers must be conscious of these biases when designing their studies, just as readers must be aware of these potential pitfalls when interpreting results. Results: The purpose of this review is to highlight some of the more common sources of bias in clinical research, including internal and external validity, type 1 and type 2 error, reporting of secondary outcomes, the use of subgroup analyses, and multiple comparisons. This article also discusses potential solutions to these issues, including using the fragility index to understand the robustness of study conclusions, and generating an E value to determine the degree of unmeasured confounding in a study. Conclusions: With an understanding of these pitfalls, readers can critically review scientific literature and ascertain the validity of the conclusions.

The Statistical Fragility of Hamstring Versus Patellar Tendon Autografts for Anterior Cruciate Ligament Reconstruction: A Systematic Review of Comparative Studies

BACKGROUND

Evidence-based medicine utilizes data to inform clinical decision making, despite the ability of a small number of outcome reversals to change statistical significance. P values are common measurements of statistical significance that possess inherent flaws. The inclusion of the fragility index (FI) and fragility quotient (FQ) may provide a clearer conveyance of statistical strength.

PURPOSE/HYPOTHESIS

The purpose was to examine the statistical stability of studies comparing hamstring tendon and bone-patellar tendon-bone autografts in primary single-bundle anterior cruciate ligament reconstruction with independent tunnel drilling. We hypothesized that the findings of these studies are vulnerable to a small number of outcome event reversals, often fewer than the number of patients lost to follow-up.

STUDY DESIGN

Systematic review.

METHODS

Comparative studies and randomized controlled trials (RCTs) published in 10 leading orthopaedic journals between 2000 and 2020 were analyzed. Statistical significance was defined as a P value ≤.05. FI for each outcome was determined by the number of event reversals necessary to alter significance. FQ was calculated by dividing the FI by the respective sample size.

RESULTS

Of the 1803 studies screened, 643 met initial search criteria, with 18 comparative studies ultimately included for analysis, 8 of which were RCTs. A total of 114 outcomes were examined. Overall, the mean (interquartile range) FI and FQ were 3.77 (2-4) and 0.040 (0.016-0.055), respectively. The FI was less than the number of patients lost to follow-up for 76.3% of outcomes.

CONCLUSION

Studies examining graft choice for anterior cruciate ligament reconstruction may not be as statistically stable as previously thought. Comparative studies and RCTs are at substantial risk for statistical fragility, with few event reversals required to alter significance. The reversal of <4 outcome events in a treatment group can alter the statistical significance of a given result; this is commonly fewer than the number of patients lost to follow-up. Future comparative study analyses might consider including FI and FQ with P values in their statistical analysis.

Fragility of randomized trials supporting cancer drug approvals stratified by approval pathway and review designations

BACKGROUND

It has been suggested that the results from fragile trials are less likely to translate into benefit in routine clinical practice.

METHODS

We searched the Food and Drug Administration (FDA) archives to identify drug approvals for solid organ malignancies between 2010 and 2019. We calculated the Fragility Index (FI) supporting each approval, using methods to account for time-to-event. We compared FI and trial and approval characteristics using Mann-Whitney U and Kruskal-Wallis test. Using logistic regression, we examined study characteristics associated with withdrawal of consent or lost to follow-up (WCLFU) exceeding the calculated FI.

RESULTS

The median FI among 125 included studies was 23 (range 1-322). The FI was ≤10 in 35 studies (28%), 11-20 in 21 (17%), and >20 in 69 (55%). The median FI/Nexp was 7.7% (range 0.1-51.7%). The median FI was significantly lower among approvals processed through the accelerated vs regular pathway (5.5 vs 25, p = 0.001), but there was no difference in median FI/Nexp. The WCLFU exceeded FI in 42% of studies. Overall survival endpoints were more likely to have a WCLFU exceeding FI (OR 3.16, p = 0.003). WCLFU exceeding FI was also associated with a lesser magnitude of effect (median HR 0.69 vs 0.55, p < 0.001). In a sensitivity analysis including only studies with 1:1 randomization, 51% of studies had WCLFU >FI.

CONCLUSION

The median FI among all trials was 23, and WCLFU exceeded FI in 42%. Comparative trials in solid tumors supporting approval through the accelerated pathway are more fragile compared to trials approved through the regular pathway, an observation likely explained by a lower sample size in the experimental arm.

Robustness and pragmatism of the evidence supporting the European Society for Medical Oncology guidelines for the diagnosis, treatment, and follow-up of follicular lymphoma

Background: Results of randomized clinical trials may not be entirely applicable to clinical practice. The present manuscript aims to explore the pragmatism and robustness of the evidence that supports the European Society for Medical Oncology (ESMO) follicular lymphoma (FL) guidelines.Methods & design: Analysis of all trials used to support positive, therapeutic, oncological recommendations in the 2020 ESMO FL guidelines. Predefined data points were extracted from each trial. Pragmatism was assessed by means of the PRECIS-2 tool, the difference in overall survival in the interventions compared and the source of funding. Robustness was assessed by means of the fragility index and the p value.Results: 28 trials were included. The full protocol or a protocol summary was provided for 12 (43%). Based on the PRECIS-2 domains, trials were considered pragmatic in organization, analysis and flexibility and explanatory in eligibility. Robustness was high, with 4/24 (17%) trials with p values between 0.05 and 0.005 and a median fragility index of 18.Conclusions: Results of trials to support ESMO recommendations in FL were robust. Pragmatism was high in some domains but modest to low in others and the pattern was similar across trials. Transparency in the publication of trial protocols was suboptimal.

Fragility Index of Recently Published Meta-Analyses in Pediatric Urology: A Striking Observation

Background and objective

Fragility Index (FI) of meta-analyses determines their stability in terms of the level of confidence and strength behind the results depicted by them. The present study was conducted to estimate the FI of recently published meta-analyses in the Journal of Pediatric Urology (JPUrol).

Method

Twenty recently published articles on meta-analyses were screened to identify the eligible ones. The baseline data of each meta-analysis including the details of the author, number of included studies, total sample size, the total number of events, the status of the overall outcome (significant or non-significant), type of effect measure, type of method used for pooling the estimates, and type of effects model were recorded. FI was calculated by doing each single status modification. The 95% CI of the treatment effect was re-calculated until the statistical significance of the meta-analysis was reversed.

Results

A total of seven articles incorporating 22 meta-analyses were included. Seven (32%) of them had a statistically significant outcome prior to FI estimation. The risk ratio (17/22; 77%) was the most commonly used effect measure. The random-effects model (15/22; 68%) and the Mantel-Haenszel method (20/22; 91%) of pooling the estimates were utilized in the majority of meta-analyses. The median (Q1-Q3; range) FI of statistically significant, non-significant, and total meta-analyses were 5 (3-19.5; 2-39), 5 (3.5-6; 1-17), and 5 (3-13; 1-39) respectively. FI of ≤5 was noticed in four out of seven (57%), 9/15 (60%), and 13/22 (59%) of these meta-analyses respectively.

Conclusion

Based on our findings, the majority of the recently published meta-analyses in the field of pediatric urology are fragile and depend upon the event status of ≤5 participants.

Effect of Metal-Ligand Coordination Complexes on Molecular Dynamics and Structure of Cross-Linked Poly(dimethylosiloxane)

Poly(dimethylosiloxane) (PDMS) cross-linked by metal-ligand coordination has a potential functionality for electronic devices applications. In this work, the molecular dynamics of bipyridine (bpy)–PDMS-MeCl₂ (Me: Mn²⁺, Fe²⁺, Ni²⁺, and Zn²⁺) are investigated by means of broadband dielectric spectroscopy and supported by differential scanning calorimetry and density functional theory calculations. The study of molecular motions covered a broad range of temperatures and frequencies and was performed for the first time for metal-ligand cross-linked PDMS. It was found that the incorporation of bpy moieties into PDMS chain prevents its crystallization. The dielectric permittivity of studied organometallic systems was elevated and almost two times higher (ε′ ~4 at 1 MHz) than in neat PDMS. BpyPDMS-MeCl₂ complexes exhibit slightly higher glass transition temperature and fragility as compared to a neat PDMS. Two segmental type relaxations (α and α_ac) were observed in dielectric studies, and their origin was discussed in relation to the molecular structure of investigated complexes. The α_ac relaxation was observed for the first time in amorphous metal-ligand complexes. It originates from the lower mobility of PDMS polymer chains, which are immobilized by metal-ligand coordination centers via bipyridine moieties.

Evaluation of the fragility of pivotal trials used to support US Food and Drug Administration approval for plaque psoriasis

BACKGROUND

Over the last 5 years, there has been a rapid growth in the number of clinical trials used to support a US Food and Drug Administration (FDA) approval for systemic therapies with labeled indications for plaque psoriasis.

OBJECTIVE

We aim to evaluate the fragility of clinical trial data used to support FDA approval of therapies for psoriasis.

METHODS

We reviewed the primary endpoints of the pivotal trials of all systemic medications with a labeled indication for plaque psoriasis available from Drugs@FDA.

RESULTS

Sixty-nine clinical trial primary endpoints met inclusion criteria and were assessed for robustness, yielding a median fragility index of 72 and a median fragility quotient of 0.19.

LIMITATIONS

Efficacy and statistical analysis data for several approved medications were not available on the product label or on Drugs@FDA.

CONCLUSIONS

When compared with randomized controlled trials for FDA approval across various diseases, pivotal trials in psoriasis appear quite robust to changes in outcomes.

Fragility Index in Cardiovascular Randomized Controlled Trials

BACKGROUND

Efficacy of an intervention is commonly evaluated using P values, in addition to effect size measures such as absolute risk reduction, relative risk reduction, and numbers needed to treat. However, these measures are not always intuitive to clinicians. The fragility index (FI) is a more intuitive number that can facilitate interpretation but can only be used with binary outcomes. FI is the minimum number of patients who must be moved from the nonevent group to the event group to turn a significant result nonsignificant. In this retrospective analysis, we assessed the robustness of cardiovascular randomized controlled trials (RCTs), which report a positive (statistically significant) primary outcome by using the FI.

METHODS AND RESULTS

We searched Medline from 2007 to 2017 to identify cardiovascular RCTs published in 6 high impact journals (The Lancet, New England Journal of Medicine, Journal of the American Medical Association, Circulation, Journal of the American College of Cardiology and European Heart Journal). Only RCTs with sample sizes >500 and a 2-by-2 factorial design or dichotomous primary outcomes were selected. FI was calculated using a defined approach. Among the cohort of 123 RCTs that met inclusion criteria, median FI was 13 (interquartile range, 5-26). In 28 trials (22.8%), FI ranged between 1 and 4. In 37 trials (30.1%), number of patients lost to follow-up was higher than the FI. Pharmaceutical interventions had higher FI compared with other interventions, FI=19 (7-52; P=0.002). Median FI varied according to subspecialty (electrophysiology=2; heart failure=11; interventional cardiology=8; P=0.020) and multiregional RCTs had higher FI=22 (12-53.25; P=0.023). FI did not differ based on risk of bias indicators, funding, or publication year.

CONCLUSIONS

Considerable variations in FI were observed among cardiovascular trials, suggesting the need for careful interpretation of results, particularly when number of patients lost to follow-up exceeds FI.

Reexamining the data used in the 2012 guidelines of the American Association for the Study of Liver Diseases for the management of adult patients with ascites due to cirrhosis

Background:

In 2012, the American Association for the Study of Liver Diseases published practice guidelines for the management of patients with ascites caused by cirrhosis, using data from randomized controlled trials (RCTs) and observational studies. We reexamined the strength of these RCTs by calculating the fragility index (FI), a novel metric proposed for evaluating the robustness of RCTs.

Methods:

We screened all RCTs referenced in the guidelines for specific criteria. We calculated the FI and fragility quotient (FQ), and analyzed the correlation between FI and several variables.

Results:

Twenty-one RCTs were included. The median (25^th, 75^th) FI and FQ were 1 (interquartile range [IQR] 0.5-6) and 0.070 (IQR 0.008-0.166), respectively. For studies that reported the number of patients lost to follow up (12 RCTs), the median of patients lost was 2 (IQR 0-6.5). There was no significant correlation between FI and sample size (r_s=0.357), P-value (r_s=-0.299), number lost to follow up (r_s=0.355), Science Citation Index (r_s=0.347), year of publication (r_s=-0.085), blinding (r_pb=-0.18) or number of centers (r_pb=0.10). However, a significant correlation was seen between FI and number needed to treat (r_s=-0.549; P=0.015).

Conclusions:

RCTs in the field of cirrhosis-related ascites are fragile. Of the 21 trials analyzed, 13 had an FI of 3 or below and these trials influenced 13 of the 49 recommendations in the guidelines. We recommend the incorporation of FI and FQ in addition to P-value to better understand the meaning of the results in gastroenterological studies.

The Fragility Index in Hand Surgery Randomized Controlled Trials

PURPOSE

Randomized controlled trials (RCTs) are the gold standard for comparing clinical interventions. Statistical significance as reported via a P value has been used to determine if a difference between clinical interventions exists in an RCT. However, P values do not clearly convey information about the robustness of a study's conclusions. An emerging metric, called the fragility index (the number of subjects who would need to change outcome category to raise the P value above the .05 threshold), is an indirect measure of how likely a repeat of the trial would reach the same conclusions. This study addressed the fragility of RCTs using dichotomous outcomes in hand surgery.

METHODS

Using systematic searching of the MEDLINE database, we identified hand surgery RCTs published in 11 high-impact journals published in the last decade (2007-2017). Studies were identified that involved 2 parallel arms, allocated patients to treatment and control in a 1:1 ratio, and reported statistical significance for a dichotomous variable. The fragility index was calculated using Fisher's exact test, using previously published methods.

RESULTS

Five hand surgery RCTs were identified for inclusion reporting a range of fragility indices from 0 to 26. Two of the trials (40%) had a fragility index of 2 or less. Two of the trials (40%) reported that the number of patients lost to follow-up exceeded the fragility index, meaning that results of the patients lost to follow-up could theoretically completely reverse the study conclusions.

CONCLUSIONS

The range of fragility indices reported in the recent hand surgery literature is consistent with previous reporting within orthopedic surgery.

CLINICAL RELEVANCE

The fragility index is a useful metric to analyze the robustness of the study conclusions that should complement other methods of critical evaluation including the P value or effect sizes. Our results emphasize the need for future efforts to strengthen the robustness of RCT conclusions.

The Fragility Index in a Cohort of HIV/AIDS Randomized Controlled Trials

HIV/AIDS is associated with significant morbidity, mortality, and financial burden. For these reasons, robust clinical evidence is critical. We aim to investigate the fragility index, fragility quotient, and risk of bias of clinical trial endpoints in HIV medicine. The fragility index represents the minimum amount of trial endpoint "nonevents" changed to "events" in one trial arm required to nullify statistical significance. The fragility quotient contextualized the fragility index by dividing the index by the total trial sample size. We selected eligible trials from the Department of Health and Human Services guideline for the use of antiretroviral agents in HIV-1-infected adults and adolescents. We calculated the fragility index and fragility quotient for all included trials. The Cochrane "risk of bias" Tool 2.0 was used to evaluate the likelihood and sources of bias in the included trials. Thirty-nine RCTs were included for our analysis of fragility. Thirty-six were included for our analysis of the risk of bias. The median fragility index was 5. Three RCTs were at high risk of bias, all due to the selection of the endpoint or statistical test. Twenty had some concerns for risk of bias. The analyzed HIV medicine RCT endpoints were fragile, overall. This indicates that a median of 5 patients across all included studies would nullify the statistical significance of the endpoints. Furthermore, we found evidence that concerns for bias are present at a high rate.

Fragility of randomized clinical trials of treatment of clavicular fractures

BACKGROUND

Statistical significance, as reported by the P value, has traditionally been the most commonly reported way to determine whether a difference exists between clinical interventions. Unfortunately, P values alone confer little about the robustness of a study's conclusions. An emerging metric, the fragility index (FI), helps to address this challenge by quantifying the number of events per outcome group that would need to be reversed to the alternative outcome in order to raise the P value above the 0.05 threshold.

METHODS

Using systematic search strategy, we identified randomized controlled trials (RCTs) pertaining to clavicular fractures published in the last decade (2007-2017). Studies included for analysis involved 2 parallel arms, were published in English, allocated patients to treatment and control arms in a 1:1 ratio, and reported statistical significance (P < .05) for dichotomous variables. The FI was determined based on the Fisher exact test, using previously published methods.

RESULTS

Fifteen RCTs were included. The median FI was 2 (range, 0-17). Eleven studies (73.3%) had an FI of 2 or less. Seven of the trials (46.7%) reported that the number of patients lost to follow-up exceeded the FI.

CONCLUSIONS

The median FI reported in the recent literature on clavicular fractures is only 2. The FI is a useful metric to analyze the robustness of study conclusions that should complement other methods of critical data evaluation, including the P value or effect sizes. Future efforts are needed to increase institutional collaboration and patient recruitment to strengthen the robustness of RCT conclusions, especially in the realm of clavicular fracture management.

How Fragile Are Clinical Trial Outcomes That Support the CHEST Clinical Practice Guidelines for VTE?

BACKGROUND

VTE remains a health concern for global populations. Clinical practice guidelines are necessary to guide physicians in the prophylaxis and treatment of VTE.

METHODS

Our investigation assessed the robustness of the underlying evidence in 21 randomized controlled trials (RCTs) used to support treatment recommendations in the 2016 update of the CHEST Guideline and Expert Panel Report on Antithrombotic Therapy for VTE Disease. We calculated the fragility index and fragility quotient for qualifying outcomes within RCTs.

RESULTS

The median fragility index for all studies was 5 (interquartile range, 1-9), with a median fragility quotient of 0.012 (interquartile range, 0.002-0.032).

CONCLUSIONS

Our conclusions parallel those of previous investigations of the fragility of RCT outcomes; we found that some outcomes used to support recommendations in AT10 are fragile. We recommend that the fragility index and fragility quotient be adopted as measures of robustness of clinical trial outcomes.