Paediatric surgical trials, their fragility index, and why to avoid using it to evaluate results

The fragility of statistical findings in the intertrochanteric fracture fixation literature: a systematic review of randomized controlled trials

INTRODUCTION

Intertrochanteric fractures are common and can lead to significant disability and morality, particularly in the elderly. Utilizing the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ), this study evaluates the statistical fragility of outcomes reported in intertrochanteric fracture fixation randomized controlled trials (RCTs).

MATERIALS AND METHODS

Data sources: Pubmed, Embase, and MEDLINE were queried for RCTs published between 2010-present.

STUDY SELECTION

RCTs reporting 1:1 categorical, dichotomous outcomes were included. Articles were excluded if they were not RCTs, had over two treatment groups, included in vitro/animal/cadaveric data, and did not feature intertrochanteric fractures.

DATA EXTRACTION

Publication and individual outcome data were collected by three independent reviewers.

DATA SYNTHESIS

FI and rFI were calculated as the number of event reversals required to reverse the statistical significance for each outcome. The FQ was calculated by dividing FI by the study sample size. Subgroup analysis was performed based on outcome types.

RESULTS

Two hundred thirty-two articles were screened, and 52 articles with a total of 370 outcomes were included for analysis. The median FI was 5 (IQR 4-6) with a FQ of 0.05 (IQR 0.032-0.078). 57/370 outcomes were statistically significant with a median FI of 3 (IQR 1-8). 313 outcomes were statistically nonsignificant with a median rFI of 5 (IQR 4-6). The number of patients lost to follow-up was greater than or equal to the FI in 127/370 outcomes (34.32%). Outcomes relating to malunion/nonunion were the most fragile, encompassing 11 outcomes with a median FI of 3 (IQR 2.5-5).

CONCLUSION

Outcomes in intertrochanteric fracture fixation RCTs are fragile as reversal of a few outcomes or maintaining follow-up may alter the significance of study findings. Thus, P-values are recommended to be routinely reported with FI and FQ metrics in order to provide a comprehensive understanding of the statistical robustness of outcomes in orthopedic trauma literature.

LEVEL OF EVIDENCE

I.

Challenging the Concept of Statistical Fragility: Is There Any Value Added?

ABSTRACT

Today, well-designed randomized clinical trials (RCTs) are considered the pinnacle of clinical research, and they inform many practices in orthopaedics. When designing these studies, researchers conduct a power analysis, which allows researchers to strike a balance between (1) enrolling enough patients to detect a clinically important treatment effect (i.e., researchers can be confident that the effect is unlikely due to chance) and (2) cost, time, and risk to patients, which come with enrolling an excessive number of patients. Because researchers will have a desire to conduct resource-efficient RCTs and protect patients from harm, many studies report a p value that is close to the threshold for significance. The concept of the fragility index (FI) was introduced as a simple way to interpret RCT findings, but it does not account for RCT design. The adoption of the FI conflicts with researchers' goals of designing efficient RCTs that conserve resources and limit ineffective or harmful treatments to patients. The use of the FI may reflect many clinicians' lack of familiarity with interpreting p values beyond "significant" or "nonsignificant." Instead of inventing new metrics to convey the same information provided by the p value, greater emphasis should be placed on educating clinicians on how to interpret p values and, more broadly, statistics, when reading scientific studies.

Statistical fragility of outcomes in acellular dermal matrix literature: A systematic review of randomized controlled trials

BACKGROUND

Acellular dermal matrix (ADM) is commonly used in plastic and reconstructive surgery. With the abundance of randomized controlled trials (RCTs) reporting P-values for ADM outcomes, this study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to evaluate the statistical stability of the outcomes in ADM RCTs.

METHODS

PubMed, Embase, SCOPUS, Medline, and Cochrane databases were reviewed for ADM RCTs (2003-present) reporting a dichotomous, categorical outcome. FI and rFI (event reversals influencing outcome significance) and FQ (standardized fragility) were calculated and reported as median. Subgroup analysis was performed based on intervention types.

RESULTS

Among the 127 studies screened, 56 RCTs with 579 outcomes were included. The median FI stood at 4 (3-5) and FQ was 0.04 (0.03-0.07). Only 101 outcomes were statistically significant with a median FI of 3 (1-6) and FQ of 0.04 (0.02-0.08). The nonsignificant outcomes had a median FI of 4 (3-5) and FQ of 0.04 (0.03-0.07). Notably, 26% of the outcomes had several patients lost to follow up equal to or surpassing the FI. Based on the intervention type, the median FIs showed minor fluctuations but remained low.

CONCLUSIONS

Outcomes from ADM-related RCTs were statistically fragile. Slight outcome reversals or maintenance of patient follow-up can alter the significance of results. Therefore, future researchers are recommended to jointly report FI, FQ, and P-values to offer a comprehensive view of the robustness in ADM literature.

The Statistical Fragility of Orbital Fractures: A Systematic Review of Randomized Controlled Trials

BACKGROUND

The P value has often been used as a tool to determine the statistical significance and evaluate the statistical robustness of study findings in orthopedic literature. The purpose of this study is to apply both the fragility index (FI) and the fragility quotient (FQ) to evaluate the degree of statistical fragility in orbital fracture literature. We hypothesized that the dichotomous outcomes within the orbital fracture literature will be vulnerable to a small number of outcome event reversals and will be statistically fragile.

METHODS

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), the authors identified all dichotomous data for randomized controlled trials (RCTs) in orbital fracture literature and performed a PubMed search from 2000 to 2022. The FI of each outcome was calculated through the reversal of a single outcome event until significance was reversed. The FQ was calculated by dividing each FI by study sample size. The interquartile range (IQR) was also calculated for the FI and FQ.

RESULTS

Of the 3,329 studies screened, 28 met the criteria with 10 RCTs evaluating orbital fractures included for analysis. A total of 58 outcome events with 22 significant (P < .05) outcomes and 36 nonsignificant (P ≥ .05) outcomes were identified. The overall FI and FQ for all 58 outcomes was 5 (IQR: 4 to 5) and 0.140 (IQR: 0.075 to 0.250), respectively. Fragility analysis of statistical significant outcomes and nonsignificant outcomes had an FI of 3.5 with no IQR and 5 (IQR 4-5), respectively. All of the studies reported a loss to follow-up data, where 20% (2) was greater than the overall FI of 5.

CONCLUSION

The orbital fracture literature provides treatment guidance by relying on statistical significant results from RCTs. However, the RCTs in the orbital fracture peer-reviewed literature may not be statistically stable as previously thought. The sole reliance of the P value may depict misleading results. Thus, we recommend standardizing the reporting of the P value, FI, and FQ in the orbital fracture literature to aid readers in reliably drawing conclusions based on fragility outcome measures impacting clinical decision-making.

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.