On clinical trial fragility due to patients lost to follow up

Reverse Fragility Index: Comparing Revision Rates Between Direct Anterior and Other Approaches in Total Hip Arthroplasty. A Systematic Review of Randomized Controlled Trials

BACKGROUND

Despite increasing adoption of the direct anterior (DA) approach in total hip arthroplasty (THA), uncertainty persists regarding its outcomes beyond the 1-year mark in comparison to other approaches. We used the reverse fragility index (RFI) to evaluate the robustness of reported findings in the literature.

METHODS

We conducted a systematic review of randomized controlled trials (RCTs) comparing implant revision rates between DA and other approaches in THA, defined as all those different from DA. Our primary outcome was the RFI, which gauges the number of events needed for a nonsignificant result to become significant, in the revision rate between DA and other approaches. We also calculated the reverse fragility quotient by dividing the RFI by each study's sample size. Median values and interquartile ranges (IQRs) were displayed.

RESULTS

A total of 10 RCTs with a total of 971 patients were included. The median RFI was 5 (IQR, 4 to 5), indicating the study's results would be statistically significant if the outcomes of 5 patients in 1 treatment arm were reversed. The median reverse fragility quotient was 0.049 (IQR, 0.04 to 0.057), indicating that a change of outcome in 4.9% of patients would render the revision rate significant. The median number of patients lost to follow-up was 4 (IQR, 0 to 7). Of the 10 RCTs, 6 had more patients lost to follow-up than their respective RFI values.

CONCLUSIONS

Notable fragility was evidenced in most studies comparing DA to other approaches for THA. Surgeons should not solely rely on the P value to determine clinical significance and instead use multiple metrics.

LEVEL OF EVIDENCE

II.

Effects of closed loop ventilation on ventilator settings, patient outcomes and ICU staff workloads - a systematic review

BACKGROUND

Lung protective ventilation is considered standard of care in the intensive care unit. However, modifying the ventilator settings can be challenging and is time consuming. Closed loop modes of ventilation are increasingly attractive for use in critically ill patients. With closed loop ventilation, settings that are typically managed by the ICU professionals are under control of the ventilator's algorithms.

OBJECTIVES

To describe the effectiveness, safety, efficacy and workload with currently available closed loop ventilation modes.

DESIGN

Systematic review of randomised clinical trials.

DATA SOURCES

A comprehensive systematic search in PubMed, Embase and the Cochrane Central register of Controlled Trials search was performed in January 2023.

ELIGIBILITY CRITERIA

Randomised clinical trials that compared closed loop ventilation with conventional ventilation modes and reported on effectiveness, safety, efficacy or workload.

RESULTS

The search identified 51 studies that met the inclusion criteria. Closed loop ventilation, when compared with conventional ventilation, demonstrates enhanced management of crucial ventilator variables and parameters essential for lung protection across diverse patient cohorts. Adverse events were seldom reported. Several studies indicate potential improvements in patient outcomes with closed loop ventilation; however, it is worth noting that these studies might have been underpowered to conclusively demonstrate such benefits. Closed loop ventilation resulted in a reduction of various aspects associated with the workload of ICU professionals but there have been no studies that studied workload in sufficient detail.

CONCLUSIONS

Closed loop ventilation modes are at least as effective in choosing correct ventilator settings as ventilation performed by ICU professionals and have the potential to reduce the workload related to ventilation. Nevertheless, there is a lack of sufficient research to comprehensively assess the overall impact of these modes on patient outcomes, and on the workload of ICU staff.

The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

PURPOSE

To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature.

METHODS

All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF.

RESULTS

Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal.

CONCLUSIONS

The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT.

CLINICAL RELEVANCE

Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.

Fragility of Statistically Significant Outcomes in Colonic Diverticular Disease Randomized Trials: A Systematic Review

BACKGROUND

The p value has been criticized as an oversimplified determination of whether a treatment effect exists. One alternative is the fragility index. It is a representation of the minimum number of nonevents that would need to be converted to events to increase the p value above 0.05.

OBJECTIVE

To determine the fragility index of randomized controlled trials assessing the efficacy of interventions for patients with diverticular disease since 2010 to assess the robustness of current evidence.

DESIGN

MEDLINE, Embase, and Cochrane Central Register of Controlled Trials were searched from inception to August 2022.

SETTINGS

Articles were eligible for inclusion if they were randomized trials conducted between 2010 and 2022 with parallel, superiority designs evaluating interventions in patients with diverticular disease. Only randomized trials with dichotomous primary outcomes with an associated p value of <0.05 were considered for inclusion.

PARTICIPANTS

Any surgical or medical intervention for patients with diverticular disease.

MAIN OUTCOME MEASURES

The fragility index was determined by adding events and subtracting nonevents from the groups with the smaller number of events. Events were added until the p value exceeded 0.05. The smallest number of events required was considered the fragility index.

RESULTS

After screening 1271 citations, 15 randomized trials met the inclusion criteria. Nine of the studies evaluated surgical interventions and 6 evaluated medical interventions. The mean number of patients randomly assigned and lost to follow-up per randomized controlled trial was 92 (SD 35.3) and 9 (SD 11.4), respectively. The median fragility index was 1 (range, 0-5). The fragility indices for the included studies did not correlate significantly with any study characteristics.

LIMITATIONS

Small sample, heterogeneity, and lack of inclusion of studies with continuous outcomes.

CONCLUSIONS

The randomized trials evaluating surgical and medical interventions for diverticular disease are not robust. Changing a single-outcome event in most studies was sufficient to make a statistically significant study finding not significant. See Video Abstract .

FRAGILIDAD DE LOS RESULTADOS ESTADSTICAMENTE SIGNIFICATIVOS EN ENSAYOS ALEATORIOS DE ENFERMEDAD DIVERTICULAR DEL COLON UNA REVISIN SISTEMTICA

ANTECEDENTES:El valor p ha sido criticado por una determinación demasiado simplificada de si existe un efecto del tratamiento. Una alternativa es el Índice de Fragilidad. Es una representación del número mínimo de no eventos que deberían convertirse en eventos para aumentar el valor p por encima de 0,05.OBJETIVO:Determinar el IF de ensayos controlados aleatorios que evalúan la eficacia de las intervenciones para pacientes con enfermedad diverticular desde 2010 para evaluar la solidez de la evidencia actual.FUENTES DE DATOS:Se realizaron búsquedas en MEDLINE, Embase y CENTRAL desde el inicio hasta agosto de 2022.SELECCIÓN DE ESTUDIOS:Los artículos eran elegibles para su inclusión si eran ensayos aleatorizados realizados entre 2010 y 2022 con diseños paralelos de superioridad que evaluaran intervenciones en pacientes con enfermedad diverticular. Sólo se consideraron para su inclusión los ensayos aleatorizados con resultados primarios dicotómicos con un valor de p asociado menor que 0,05.INTERVENCIÓNES:Cualquier intervención quirúrgica o médica para pacientes con enfermedad diverticular.PRINCIPALES MEDIDAS DE VALORACIÓN:El índice de fragilidad se determinó sumando eventos y restando no eventos de los grupos con el menor número de eventos. Se agregaron eventos hasta que el valor p superó 0,05. El menor número de eventos requeridos se consideró índice de fragilidad.RESULTADOS:Después de examinar 1271 citas, 15 ensayos aleatorios cumplieron los criterios de inclusión. Nueve de los estudios evaluaron intervenciones quirúrgicas y seis evaluaron intervenciones médicas. El número medio de pacientes aleatorizados y perdidos durante el seguimiento por ECA fue 92 (DE 35,3) y 9 (DE 11,4), respectivamente. La mediana del índice de fragilidad fue 1 (rango: 0-5). Los índices de fragilidad de los estudios incluidos no se correlacionaron significativamente con ninguna característica del estudio.LIMITACIONES:Muestra pequeña, heterogeneidad y falta de inclusión de estudios con resultados continuos.CONCLUSIONES:Los ensayos aleatorios que evalúan las intervenciones quirúrgicas y médicas para la enfermedad diverticular no son sólidos. Cambiar un solo evento de resultado en la mayoría de los estudios fue suficiente para que un hallazgo estadísticamente significativo del estudio no fuera significativo. (Traducción- Dr. Ingrid Melo ).

The Statistical Fragility of Marrow Stimulation for Cartilage Defects of the Knee: A Systematic Review of Randomized Controlled Trials

OBJECTIVE

Marrow stimulation is used to address knee cartilage defects. In this study, we used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to evaluate statistical fragility of outcomes reported in randomized controlled trials (RCTs) evaluating marrow stimulation.

DESIGN

PubMed, Embase, and MEDLINE were queried for recent RCTs (January 1, 2010-September 5, 2023) assessing marrow stimulation for cartilage defects of the knee. The FI and rFI were calculated as the number of outcome event reversals required to alter statistical significance for significant and nonsignificant outcomes, respectively. The FQ was determined by dividing the FI by the study sample size.

RESULTS

Across 155 total outcomes from 21 RCTs, the median FI was 3 (interquartile range [IQR], 2-5), with an associated median FQ of 0.067 (IQR, 0.033-0.010). Thirty-two outcomes were statistically significant, with a median FI of 2 (IQR, 1-3.25) and FQ of 0.050 (IQR, 0.025-0.069). Ten of the 32 (31.3%) outcomes reported as statistically significant had an FI of 1. In total, 123 outcomes were nonsignificant, with a median rFI of 3 (IQR, 2-5). Studies assessing stem cell augments were the most fragile, with a median FI of 2. In 55.5% of outcomes, the number of patients lost to follow-up was greater than or equal to the FI.

CONCLUSION

Statistical findings in RCTs evaluating marrow stimulation for cartilage defects of the knee are statistically fragile. We recommend combined reporting of P-values with FI and FQ metrics to aid in the interpretation of clinical findings in comparative trials assessing cartilage restoration.

Empirical assessment of fragility index based on a large database of clinical studies in the Cochrane Library

RATIONALE AIMS AND OBJECTIVES

The fragility index (FI) and fragility quotient (FQ) are increasingly used measures for assessing the robustness of clinical studies with binary outcomes in terms of statistical significance. The FI is the minimum number of event status modifications that can alter a study result's statistical significance (or nonsignificance), and the FQ is calculated as the FI divided by the study's total sample size. The literature has no widely recognized criteria for interpreting the fragility measures' magnitudes. This article aims to provide an empirical assessment for the FI and FQ based on a large database of clinical studies in the Cochrane Library.

METHODS

We explored the overall empirical distributions of the FI and FQ based on five common methods (Fisher's exact test, χ2 test, risk difference, odds ratio, and relative risk) for determining statistical significance of binary outcomes in clinical research. We also considered three different scenarios for the FI calculation and evaluated the relationship between p values and FIs or FQs using Spearman'sρ $\rho $ . Finally, we summarized empirical thresholds based on the overall distributions of the FI and FQ to facilitate their interpretations in future research.

RESULTS

For about 20% of studies with significant results, the statistical significance was changed after modifying the event status of only one participant. Studies with significant results were considered slightly fragile if the significance hinged on the statuses of about five events. Studies were extremely fragile if FI≤ $\le $ 1 or FQ≤ $\le $ 0.01. The FIs were strongly correlated with p values for significant studies, while Spearman'sρ $\rho $ varied according to the total sample sizes of studies.

CONCLUSIONS

The statistical significance of clinical studies could be changed after modifying a few events' statuses. Many studies' findings are fairly fragile. The distributions of the FI and FQ provide insights for appraising the robustness of evidence in clinical decision-making.

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.