1
|
Oeding JF, Krych AJ, Camp CL, Varady NH. The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model. Arthroscopy 2024:S0749-8063(24)00366-9. [PMID: 38777001 DOI: 10.1016/j.arthro.2024.05.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Revised: 04/21/2024] [Accepted: 05/02/2024] [Indexed: 05/25/2024]
Abstract
PURPOSE To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature. METHODS All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF. RESULTS Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal. CONCLUSIONS The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT. CLINICAL RELEVANCE Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.
Collapse
Affiliation(s)
- Jacob F Oeding
- School of Medicine, Mayo Clinic Alix School of Medicine, Rochester, Minnesota, U.S.A.; Department of Orthopaedics, Institute of Clinical Sciences, The Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.
| | - Aaron J Krych
- Department of Orthopaedic Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A
| | - Christopher L Camp
- Department of Orthopaedic Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A
| | - Nathan H Varady
- Department of Orthopaedic Surgery, Hospital for Special Surgery, New York, New York, U.S.A
| |
Collapse
|
2
|
Stern BZ, Poeran J. Statistics in Brief: The Fragility Index. Clin Orthop Relat Res 2023; 481:1288-1291. [PMID: 36862056 PMCID: PMC10263243 DOI: 10.1097/corr.0000000000002622] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/19/2022] [Accepted: 02/10/2023] [Indexed: 03/03/2023]
Affiliation(s)
- Brocha Z. Stern
- Leni and Peter W. May Department of Orthopaedics, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Institute for Healthcare Delivery Science, Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Jashvant Poeran
- Leni and Peter W. May Department of Orthopaedics, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Institute for Healthcare Delivery Science, Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
3
|
Ruelos VCB, Masood R, Puzzitiello RN, Moverman MA, Pagani NR, Menendez ME, Salzler MJ. The reverse fragility index: RCTs reporting non-significant differences in failure rates between hamstring and bone-patellar tendon-bone autografts have fragile results. Knee Surg Sports Traumatol Arthrosc 2023:10.1007/s00167-023-07420-0. [PMID: 37093236 DOI: 10.1007/s00167-023-07420-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 04/14/2023] [Indexed: 04/25/2023]
Abstract
PURPOSE The reverse fragility index (RFI) is a novel metric to appraise the results of studies reporting statistically non-significant results. The purpose of this study was to determine the statistical robustness of randomized controlled trials (RCTs) reporting non-significant differences in anterior cruciate ligament reconstruction (ACLR) graft failure rates, defined as re-rupture/revision ACLR rate, between hamstring tendon (HT) and bone-patellar tendon-bone (BTB) autografts by calculating RFIs. METHODS A systematic review was performed to identify RCTs that compared HT to BTB grafts for ACLR through January 2022. Studies reporting non-significant differences in graft re-rupture and revision ACLR rate (n.s.) were included. The RFI, defined as the fewest number of event reversals needed to change the non-significant graft re-rupture/revision outcome to statistically significant (P < 0.05), was recorded for each study. In addition, the number of studies in which the loss to follow-up exceeded the RFI was recorded. RESULTS Among the 16 included RCTs, the median (interquartile range [IQR]) sample size was 71 (64-114), and the median (IQR) total number of graft re-rupture/revision ACLR events was 4 (4-6). The median (IQR) graft re-rupture/revision ACLR rate was 4.3% (3.0-6.4) overall, 4.1% (2.6-6.7) in the BTB group, and 5.4% (3.0-6.3) in the HT group. The median (IQR) RFI was 3 (3-4), signifying that a reversal of the outcome in 3 patients in one arm was needed to flip the studies' result from non-significant to statistically significant (P < 0.05). The median (IQR) number of participants lost to follow-up was 11 (3-13), and 13 (81.3%) of the included studies had a loss to follow-up greater than the studies' RFI. CONCLUSION The results of RCTs reporting statistically non-significant re-rupture/revision ACLR rates between HT and BTB autografts would become significant if the outcome were reversed in a small number of patients-a number that was less than the loss to follow-up in the majority of studies. Thus, the neutrality of these studies is fragile, and a true statistically significant difference in re-rupture/revision rates may have been undetected. LEVEL OF EVIDENCE Level I.
Collapse
Affiliation(s)
| | - Raisa Masood
- Department of Orthopaedic Surgery, Tufts Medical Center, 270 Tremont St Biewend Building, Boston, MA, 02111, USA
| | - Richard N Puzzitiello
- Department of Orthopaedic Surgery, Tufts Medical Center, 270 Tremont St Biewend Building, Boston, MA, 02111, USA
| | - Michael A Moverman
- Department of Orthopaedic Surgery, Tufts Medical Center, 270 Tremont St Biewend Building, Boston, MA, 02111, USA
| | - Nicholas R Pagani
- Department of Orthopaedic Surgery, Tufts Medical Center, 270 Tremont St Biewend Building, Boston, MA, 02111, USA
| | - Mariano E Menendez
- Midwest Orthopaedics at Rush, Rush University Medical Center, Chicago, IL, USA
| | - Matthew J Salzler
- Department of Orthopaedic Surgery, Tufts Medical Center, 270 Tremont St Biewend Building, Boston, MA, 02111, USA.
| |
Collapse
|
4
|
Milto AJ, Negri CE, Baker J, Thuppal S. The Statistical Fragility of Foot and Ankle Surgery Randomized Controlled Trials. J Foot Ankle Surg 2022; 62:191-196. [PMID: 36182644 DOI: 10.1053/j.jfas.2022.08.014] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 08/16/2022] [Accepted: 08/27/2022] [Indexed: 02/03/2023]
Abstract
Fragility index (FI) is a metric used to interpret the results of randomized controlled trials (RCTs), and describes the number of subjects that would need to be switched from event to non-event for a result to no longer be significant. Studies that analyze FI of RCTs in various orthopedic subspecialties have shown the RCTs to be largely underpowered and highly fragile. However, FI has not been assessed in foot and ankle RCTs. The MEDLINE and Embase online databases were searched from 1/1/2011 through 11/19/2021 for RCTs involving foot and ankle conditions. FI, fragility quotient (FQ), and difference between the FI and number of subjects lost to follow-up was calculated. Spearman correlation was performed to determine the relationship between sample size and FI. Overall, 1262 studies were identified of which 18 were included in the final analysis. The median sample size was 65 (interquartile range [IQR] 57-95.5), the median FI was 2 (IQR 1-2.5), and the median FQ was 0.026 (IQR 0.012-0.033). Ten of 15 (67%) studies with non-zero FI values had FI values less than the number of subjects lost to follow-up. There was linear association between FI and sample size (R2 = 0.495, p-value: .031). This study demonstrates that RCTs in the field of foot and ankle surgery are highly fragile, similar to other orthopedic subspecialties.
Collapse
Affiliation(s)
- Anthony J Milto
- Division of Orthopedics and Rehabilitation, Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL; Center for Clinical Research, Southern Illinois University School of Medicine, Springfield, IL
| | - Cecily E Negri
- Division of Orthopedics and Rehabilitation, Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL
| | - Jeffrey Baker
- Division of Orthopedics and Rehabilitation, Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL
| | - Sowmyanarayanan Thuppal
- Division of Orthopedics and Rehabilitation, Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL; Center for Clinical Research, Southern Illinois University School of Medicine, Springfield, IL.
| |
Collapse
|
5
|
Hopper GP, Pioger C, Philippe C, El Helou A, Campos JP, Gousopoulos L, Carrozzo A, Vieira TD, Sonnery-Cottet B. Risk Factors for Anterior Cruciate Ligament Graft Failure in Professional Athletes: An Analysis of 342 Patients With a Mean Follow-up of 100 Months From the SANTI Study Group. Am J Sports Med 2022; 50:3218-3227. [PMID: 36177758 DOI: 10.1177/03635465221119186] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/31/2023]
Abstract
BACKGROUND Anterior cruciate ligament (ACL) injuries are among the most common knee injuries sustained in elite sport, and athletes generally undergo ACL reconstruction (ACLR) to facilitate their return to sport. ACL graft rupture is a career-threatening event for elite athletes. PURPOSE/HYPOTHESIS The purpose of this study was to determine the risk factors for graft failure in professional athletes undergoing ACLR. It was hypothesized that athletes who underwent combined ACLR with a lateral extra-articular procedure (LEAP) would experience significantly lower rates of graft rupture in comparison with those who underwent isolated ACLR. STUDY DESIGN Cohort study; Level of evidence, 3. METHODS Professional athletes who underwent primary ACLR with a minimum follow-up of 2 years were identified from the Santy database. Patients were excluded if they underwent major concomitant procedures, including multiligament reconstruction surgery or osteotomy. Further ipsilateral knee injury, contralateral knee injury, and any other reoperations or complications after the index procedure were identified by interrogation of the database and review of the medical notes. RESULTS A total of 342 athletes with a mean follow-up of 100.2 ± 51.9 months (range, 24-215 months) were analyzed. Graft failures totaling 31 (9.1%) were reported, requiring revision surgery because of symptomatic instability. The rate of graft failure was significantly higher when ACLR was not combined with a LEAP (15.5% vs 6.0%; P = .0105) and in athletes aged 21 years or younger (13.8% vs 6.6%; P = .0290). A multivariate analysis was performed using the Cox model and demonstrated that athletes undergoing an isolated ACLR were at >2-fold risk of ACL graft rupture (hazard ratio [HR], 2.678 [1.173; 4.837], P = .0164) when compared with those undergoing a combined ACLR with a LEAP. Additionally, athletes aged ≤21 years were also at >2-fold risk of graft failure (HR, 2.381 [1.313; 5.463]; P = .0068) than those aged >21 years. Sex, sport, and graft type were not found to be significant risk factors for graft failure. CONCLUSION Professional athletes undergoing isolated ACLR and aged ≤21 years are at >2-fold greater risk of graft failure. Orthopaedic surgeons treating elite athletes should combine an ACLR with a LEAP to improve ACL graft survivorship.
Collapse
Affiliation(s)
- Graeme P Hopper
- NHS Glasgow and Clyde South Glasgow University Hospitals NHS Trust Glasgow, Glasgow, UK.,Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France
| | - Charles Pioger
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France.,Hôpital Ambroise-Paré, Department of Orthopedic Surgery, Boulogne-Billancourt, Île-de-France, France
| | - Corentin Philippe
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France
| | - Abdo El Helou
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France
| | - Joao Pedro Campos
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France
| | - Lampros Gousopoulos
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France
| | - Alessandro Carrozzo
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France.,University of Rome La Sapienza, Orthopaedics and Traumatology Rome, Lazio, Italy
| | - Thais Dutra Vieira
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France
| | - Bertrand Sonnery-Cottet
- Centre Orthopédique Santy, Lyon, France; Hôpital Privé Jean Mermoz, Ramsay-Générale de Santé, Lyon, France
| |
Collapse
|
6
|
Kaarre J, Zsidai B, Narup E, Horvath A, Svantesson E, Hamrin Senorski E, Grassi A, Musahl V, Samuelsson K. Scoping Review on ACL Surgery and Registry Data. Curr Rev Musculoskelet Med 2022; 15:385-393. [PMID: 35829892 PMCID: PMC9463418 DOI: 10.1007/s12178-022-09775-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 06/15/2022] [Indexed: 11/30/2022]
Abstract
Purpose of Review To present an overview of registry-based anterior cruciate ligament (ACL) research, as well as provide insight into the future of ACL registries. Recent Findings During the past decades, the ACL registries have had an important role in increasing our understanding of patients with ACL injuries and their treatment. The registry data has deepened our understanding of factors that have been associated with an increased risk of sustaining an ACL injury and for evaluation of treatment factors and their impact on patient-related outcomes. Recently, registry-based ACL research using artificial intelligence (AI) and machine learning (ML) has shown potential to create clinical decision-making tools and analyzing outcomes. Thus, standardization of collected data between the registries is needed to facilitate the further collaboration between registries and to facilitate the interpretation of results and subsequently improve the possibilities for implementation of AI and ML in the registry-based research. Summary Several studies have been based on the current ACL registries providing an insight into the epidemiology of ACL injuries as well as outcomes following ACL reconstruction. However, the current ACL registries are facing future challenges, and thus, new methods and techniques are needed to ensure further good quality and clinical applicability of study findings based on ACL registry data.
Collapse
Affiliation(s)
- Janina Kaarre
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden. .,Sahlgrenska Sports Medicine Center (SSMC), Gothenburg, Sweden. .,Department of Orthopaedic Surgery, UPMC Freddie Fu Sports Medicine Center, University of Pittsburgh, Pittsburgh, PA, USA.
| | - Bálint Zsidai
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.,Sahlgrenska Sports Medicine Center (SSMC), Gothenburg, Sweden.,Department of Orthopaedic Surgery, UPMC Freddie Fu Sports Medicine Center, University of Pittsburgh, Pittsburgh, PA, USA
| | - Eric Narup
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.,Sahlgrenska Sports Medicine Center (SSMC), Gothenburg, Sweden
| | - Alexandra Horvath
- Sahlgrenska Sports Medicine Center (SSMC), Gothenburg, Sweden.,Department of Internal Medicine and Clinical Nutrition, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Eleonor Svantesson
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.,Sahlgrenska Sports Medicine Center (SSMC), Gothenburg, Sweden
| | - Eric Hamrin Senorski
- Sahlgrenska Sports Medicine Center (SSMC), Gothenburg, Sweden.,Unit of Physiotherapy, Department of Health and Rehabilitation, Institute of Neuroscience and Physiology, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Alberto Grassi
- IIa Clinica Ortopedica e Traumatologica, IRCCS Istituto Ortopedico Rizzoli, Bologna, Italy
| | - Volker Musahl
- Department of Orthopaedic Surgery, UPMC Freddie Fu Sports Medicine Center, University of Pittsburgh, Pittsburgh, PA, USA
| | - Kristian Samuelsson
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.,Sahlgrenska Sports Medicine Center (SSMC), Gothenburg, Sweden.,Department of Orthopaedics, Sahlgrenska University Hospital, Mölndal, Sweden
| |
Collapse
|
7
|
Fackler NP, Ehlers CB, Callan KT, Amirhekmat A, Smith EJ, Parisien RL, Wang D. Statistical Fragility of Single-Row Versus Double-Row Anchoring for Rotator Cuff Repair: A Systematic Review of Comparative Studies. Orthop J Sports Med 2022; 10:23259671221093391. [PMID: 35571970 PMCID: PMC9096204 DOI: 10.1177/23259671221093391] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/06/2022] [Accepted: 02/17/2022] [Indexed: 01/08/2023] Open
Abstract
Background: Comparative studies and randomized controlled trials (RCTs) often use the P (probability) value to convey the statistical significance of their findings. P values are an imperfect measure, however, and are vulnerable to a small number of outcome reversals to alter statistical significance. The inclusion of a fragility index (FI) and fragility quotient (FQ) may aid in the interpretation of a study’s statistical strength. Purpose/Hypothesis: The purpose of this study was to examine the statistical stability of studies comparing single-row to double-row rotator cuff repair. It was hypothesized that the findings of these studies would be vulnerable to a small number of outcome event reversals, often fewer than the number of patients lost to follow-up. Study Design: Systematic review; Level of evidence, 3. Methods: We analyzed comparative studies and RCTs on primary single-row versus double-row rotator cuff repair that were published between 2000 and 2021 in 10 leading orthopaedic journals. Statistical significance was defined as a P < .05. The FI for each outcome was determined by the number of event reversals necessary to alter significance. The FQ was calculated by dividing the FI by the respective sample size. Results: Of 4896 studies screened, 22 comparative studies, 10 of which were RCTs, were ultimately included for analysis. A total of 74 outcomes were examined. Overall, the median FI was 2 (interquartile range [IQR], 1-3), and the median FQ was 0.035 (IQR, 0.020-0.057). The mean FI was 2.55 ± 1.29, and the mean FQ was 0.043 ± 0.027. In 64% of outcomes, the FI was less than the number of patients lost to follow-up.) Additionally, 81% of significant outcomes needed just a single outcome reversal to lose their significance. Conclusion: Over half of the studies currently used to guide clinical practice have a number of patients lost to follow-up greater than their FI. The results of these studies should be interpreted within the context of these limitations. Future analyses may benefit from the inclusion of the FI and the FQ in their statistical analyses.
Collapse
Affiliation(s)
- Nathan P. Fackler
- Department of Orthopaedic Surgery, University of California, Irvine, Irvine, California, USA
- Georgetown University School of Medicine, Washington, DC, USA
| | - Cooper B. Ehlers
- Department of Orthopaedic Surgery, University of California, San Diego, San Diego, California, USA
| | - Kylie T. Callan
- Department of Orthopaedic Surgery, University of California, Irvine, Irvine, California, USA
| | - Arya Amirhekmat
- Department of Orthopaedic Surgery, University of California, Irvine, Irvine, California, USA
| | - Eric J. Smith
- Department of Orthopaedic Surgery, University of California, Irvine, Irvine, California, USA
| | | | - Dean Wang
- Department of Orthopaedic Surgery, University of California, Irvine, Irvine, California, USA
| |
Collapse
|
8
|
Ehlers CB, Curley AJ, Fackler NP, Minhas A, Rodriguez AN, Pasko K, Chang ES. The Statistical Fragility of Single-Bundle vs Double-Bundle Autografts for ACL Reconstruction: A Systematic Review of Comparative Studies. Orthop J Sports Med 2022; 9:23259671211064626. [PMID: 34988239 PMCID: PMC8721389 DOI: 10.1177/23259671211064626] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Accepted: 09/10/2021] [Indexed: 11/17/2022] Open
Abstract
Background: The statistical significance of a given study outcome can be liable to small changes in findings. P values are common, but imperfect statistical methods to convey significance, and inclusion of the fragility index (FI) and fragility quotient (FQ) may provide a clearer perception of statistical strength. Purpose/Hypothesis: The purpose was to examine the statistical stability of studies comparing primary single-bundle to double-bundle anterior cruciate ligament reconstruction (ACLR) utilizing autograft and independent tunnel drilling. It was hypothesized that the study findings would be vulnerable to a small number of outcome event reversals, often less than the number of patients lost to follow-up. Study Design: Systematic review; Level of evidence, 2. Methods: Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, the authors searched PubMed for comparative studies and randomized controlled trials (RCTs) published in select journals, based on impact factor, between 2005 and 2020. Risk-of-bias assessment and methodology scoring were conducted for the included studies. A total of 48 dichotomous outcome measures were examined for possible event reversals. The FI for each outcome was determined by the number of event reversals necessary to alter significance. The FQ was calculated by dividing the FI by the respective sample size. Results: Of the 1794 studies screened, 15 comparative studies were included for analysis; 13 studies were RCTs. Overall, the mean FI and FQ were 3.14 (IQR, 2-4) and 0.050 (IQR, 0.032-0.062), respectively. For 72.9% of outcomes, the FI was less than the number of patients lost to follow-up. Conclusion: Studies comparing single-bundle versus double-bundle ACLR may not be as statistically stable as previously thought. Comparative studies and RCTs are at substantial risk for statistical fragility, with few event reversals required to alter significance. The reversal of fewer than 4 outcome events in a treatment group can alter the statistical significance of a given result; this is commonly less than the number of patients lost to follow-up. Future comparative study analyses might consider including FI and FQ with P values in their statistical analysis.
Collapse
Affiliation(s)
- Cooper B Ehlers
- Georgetown University School of Medicine, El Segundo, California, USA
| | | | - Nathan P Fackler
- Georgetown University School of Medicine, El Segundo, California, USA
| | - Arjun Minhas
- Georgetown University School of Medicine, El Segundo, California, USA
| | - Ariel N Rodriguez
- Georgetown University School of Medicine, El Segundo, California, USA
| | - Kory Pasko
- Georgetown University School of Medicine, El Segundo, California, USA
| | - Edward S Chang
- Inova Medical Group Orthopedics and Sports Medicine, Fairfax, Virginia, USA
| |
Collapse
|
9
|
Vargas M, Marra A, Buonanno P, Coviello A, Iacovazzo C, Servillo G. Fragility Index and Fragility Quotient in Randomized Controlled Trials on Corticosteroids in ARDS Due to COVID-19 and Non-COVID-19 Etiology. J Clin Med 2021; 10:jcm10225287. [PMID: 34830569 PMCID: PMC8624335 DOI: 10.3390/jcm10225287] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Revised: 10/20/2021] [Accepted: 11/12/2021] [Indexed: 12/15/2022] Open
Abstract
Background: The effectiveness of corticosteroids in acute respiratory distress syndrome (ARDS) and COVID-19 still remains uncertain. Since ARDS is due to a hyperinflammatory response to a direct injury, we decided to perform a meta-analysis and an evaluation of robustness of randomised clinical trials (RCTs) investigating the impact of corticosteroids on mortality in ARDS in both COVID-19 and non-COVID-19 patients. We conducted a systematic search of the literature from inception up to 30 October 2020, using the MEDLINE database and the PubMed interface. We evaluated the fragility index (FI) of the included RCTs using a two-by-two contingency table and the p-value produced by the Fisher exact test; the fragility quotient (FQ) was calculated by dividing the FI score by the total sample size of the trial. Results: Thirteen RCTs were included in the analysis; five of them were conducted in COVID-19 ARDS, including 7692 patients, while 8 RCTS were performed in non-COVID ARDS with 1091 patients evaluated. Three out of eight RCTs in ARDS had a FI > 0 while 2 RCTs out of five in COVID-19 had FI > 0. The median of FI for ARDS was 0.625 (0.47) while the median of FQ was 0.03 (0.014). The median of FI for COVID-19 was 6 (2) while the median of FQ was 0.059 (0.055). In this systematic review, we found that FI and FQ of RCTs evaluating the use of corticosteroids in ARDS and COVID-19 were low.
Collapse
|
10
|
Parisien RL, Trofa DP, O’Connor M, Knapp B, Curry EJ, Tornetta P, Lynch TS, Li X. The Fragility of Significance in the Hip Arthroscopy Literature: A Systematic Review. JB JS Open Access 2021; 6:e21.00035. [PMID: 34703967 PMCID: PMC8542173 DOI: 10.2106/jbjs.oa.21.00035] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/14/2022] Open
Abstract
The purpose of the present study was to perform the first examination of the utility of p values and the degree of statistical fragility in the hip arthroscopy literature by applying both the Fragility Index (FI) and the Fragility Quotient (FQ) to dichotomous comparative trials. We hypothesized that dichotomous comparative trials evaluating categorical outcomes in the hip arthroscopy literature are statistically fragile. METHODS The PubMed and MEDLINE databases were queried from 2008-2018 for comparative studies evaluating dichotomous data in the hip arthroscopy literature. The present analysis included both randomized controlled trials (RCTs) and non-RCTs in which dichotomous data and associated p values were reported. Fragility analysis was performed with use of the Fisher exact test until an alteration of significance was determined. RESULTS Of the 5,836 studies screened, 4,156 met the search criteria, with 52 comparative studies included for analysis. One hundred and fifty total outcome events with 33 significant (p < 0.05) outcomes and 117 nonsignificant (p ≥ 0.05) outcomes were identified. The final FI incorporating all 150 outcome events from 52 comparative studies was only 3.5 (interquartile range, 2 to 6), with an associated FQ of 0.032 (interquartile range, 0.017 to 0.063). Twenty-two studies (42.3%) either failed to report loss to follow-up (LTF) data or reported LTF greater than the overall FI of 3.5. CONCLUSIONS The peer-reviewed hip arthroscopy literature may not be as stable as previously thought, as the sole reliance on a threshold p value has proven misleading. We therefore recommend reporting of the FI and FQ, in conjunction with p values, to aid in the evaluation and interpretation of statistical robustness and quantitative significance in future comparative hip arthroscopy studies.
Collapse
Affiliation(s)
| | | | | | - Brock Knapp
- Boston University School of Medicine, Boston, Massachusetts
| | - Emily J. Curry
- Boston University School of Medicine, Boston, Massachusetts
| | - Paul Tornetta
- Boston University School of Medicine, Boston, Massachusetts
| | - T. Sean Lynch
- Columbia University Irving Medical Center, New York, NY
| | - Xinning Li
- Boston University School of Medicine, Boston, Massachusetts
| |
Collapse
|
11
|
Williams MO, Sedarous M, Dennis B, Dlamini V, Nwaiwu O, Nguyen L, Okafor PN. The fragility of randomized placebo-controlled trials for irritable bowel syndrome. Neurogastroenterol Motil 2021; 33:e14166. [PMID: 34399023 DOI: 10.1111/nmo.14166] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/26/2020] [Revised: 02/21/2021] [Indexed: 12/20/2022]
Abstract
BACKGROUND The fragility index (FI) represents the number of participants whose status in a trial would have to change from a non-event (not experiencing the primary endpoint) to an event (experiencing the primary endpoint) in order to turn a statistically significant result into a non-significant result. We sought to evaluate the fragility indices of irritable bowel syndrome [IBS-mixed (IBS-M), IBS-constipation (IBS-C), & IBS-diarrhea (IBS-D)] trials. METHODS Irritable bowel syndrome trials published in high-impact journals were identified from Medline. Trials had to be in adults, randomized, parallel-armed, with at least one statistically significant binary outcome, and an achieved primary endpoint of therapeutic efficacy. FI and correlation coefficients were calculated, and regression modeling used to identify predictors of a high FI. KEY RESULTS Twelve trials were analyzed with a median FI of 6 (range: 0-38). Median sample size in all trials was 366 (range: 44-856). Trial publication year (p = 0.71), journal impact factor (p = 0.52), duration of study (p = 0.12), and number need to treat [NNT] (p = 0.29) were not predictive of a high FI. While a lower p-value correlated with a higher FI (p = 0.039), no correlation was noted between FI and impact factor (R = -0.20, p = 0.52), trial publication year (R = 0.12, p = 0.71), duration of trial (R = -0.46, p = 0.13), NNT (R = -0.34, p = 0.29), and sample size (R = 0.23, p = 0.5). The highest FI was in a Ramosetron trial (FI = 30) for IBS-D. CONCLUSION & INFERENCES A median of six participants is needed to nullify results in the included IBS trials suggesting how easily statistical significance based on a threshold p-value may be overturned.
Collapse
Affiliation(s)
- MaryJane O Williams
- Division of Gastroenterology, Hepatology and Nutrition, Vidant Medical Center, East Carolina University, Greenville, North Carolina, USA
| | - Mary Sedarous
- Department of Medicine, McMaster University, Hamilton, Ontorio, Canada
| | - Brittany Dennis
- Department of Medicine, McMaster University, Hamilton, Ontorio, Canada
| | - Vanessa Dlamini
- Meharry Medical College School of Medicine, Nashville, Tennessee, USA
| | - Obioma Nwaiwu
- Department of Medicine, University of Arkansas School of Medical Sciences, Fayetteville, Arkansas, USA
| | - Linda Nguyen
- Division of Gastroenterology and Hepatology, Stanford University School of Medicine, Stanford, California, USA
| | - Philip N Okafor
- Division of Gastroenterology and Hepatology, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
12
|
Ehlers CB, Curley AJ, Fackler NP, Minhas A, Chang ES. The Statistical Fragility of Hamstring Versus Patellar Tendon Autografts for Anterior Cruciate Ligament Reconstruction: A Systematic Review of Comparative Studies. Am J Sports Med 2021; 49:2827-2833. [PMID: 33211555 DOI: 10.1177/0363546520969973] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/31/2023]
Abstract
BACKGROUND Evidence-based medicine utilizes data to inform clinical decision making, despite the ability of a small number of outcome reversals to change statistical significance. P values are common measurements of statistical significance that possess inherent flaws. The inclusion of the fragility index (FI) and fragility quotient (FQ) may provide a clearer conveyance of statistical strength. PURPOSE/HYPOTHESIS The purpose was to examine the statistical stability of studies comparing hamstring tendon and bone-patellar tendon-bone autografts in primary single-bundle anterior cruciate ligament reconstruction with independent tunnel drilling. We hypothesized that the findings of these studies are vulnerable to a small number of outcome event reversals, often fewer than the number of patients lost to follow-up. STUDY DESIGN Systematic review. METHODS Comparative studies and randomized controlled trials (RCTs) published in 10 leading orthopaedic journals between 2000 and 2020 were analyzed. Statistical significance was defined as a P value ≤.05. FI for each outcome was determined by the number of event reversals necessary to alter significance. FQ was calculated by dividing the FI by the respective sample size. RESULTS Of the 1803 studies screened, 643 met initial search criteria, with 18 comparative studies ultimately included for analysis, 8 of which were RCTs. A total of 114 outcomes were examined. Overall, the mean (interquartile range) FI and FQ were 3.77 (2-4) and 0.040 (0.016-0.055), respectively. The FI was less than the number of patients lost to follow-up for 76.3% of outcomes. CONCLUSION Studies examining graft choice for anterior cruciate ligament reconstruction may not be as statistically stable as previously thought. Comparative studies and RCTs are at substantial risk for statistical fragility, with few event reversals required to alter significance. The reversal of <4 outcome events in a treatment group can alter the statistical significance of a given result; this is commonly fewer than the number of patients lost to follow-up. Future comparative study analyses might consider including FI and FQ with P values in their statistical analysis.
Collapse
Affiliation(s)
- Cooper B Ehlers
- School of Medicine, Georgetown University, Washington, DC, USA
| | - Andrew J Curley
- Department of Orthopaedic Surgery, MedStar Georgetown University Hospital, Washington, DC, USA
| | | | - Arjun Minhas
- School of Medicine, Georgetown University, Washington, DC, USA
| | - Edward S Chang
- INOVA Orthopaedics and Sports Medicine, Fairfax, Virginia, USA
| |
Collapse
|
13
|
Chertin L, Burman I, Haifler M. Urologic oncology randomized controlled trials are frequently fragile - A review of the urology literature. Urol Oncol 2021; 39:735.e1-735.e8. [PMID: 34103225 DOI: 10.1016/j.urolonc.2021.04.037] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2021] [Revised: 04/05/2021] [Accepted: 04/25/2021] [Indexed: 10/21/2022]
Abstract
BACKGROUND Randomized controlled trials (RCT) in urologic oncology are the basis of patient management. Considerable debate exists on the limitation of statistical reporting of randomized controlled trials. Fragility index (FI) is a measure of the number of events upon which the trials statistical results depend on. The FI is defined as the minimum number of patients whose status would have to change from a 'non-event' to 'event', in order to turn a statistically significant result to a non-significant result and vice versa. Our aim is to examine the FI of RCT's in urologic oncology published in the urology literature. MATERIAL AND METHODS We exhaustively searched MEDLINE and EMBASE from January 1, 2016 to December 31 2019 for RCT's in urology journals. Only studies reporting dichotomous outcomes were included and FI was calculated for each outcome. The distributions of FI across different journals and types of outcome (primary/ secondary, significant/ non-significant) were assessed. We examined the correlation of FI with sample size and P-value. RESULTS We identified 216 RCT's, 79 were eligible for analysis. Median FI was 3.0 (2.5, 6.0). One hundred and forty-six (89.6%) outcomes from 72 (94.7%) RCT's had a FI lower than 10 and 6 (3.7%) outcomes from 6 (7.9%) RCT's had a FI = 0. There was no statistically significant difference in FI between different types of outcomes. There was a modest correlation between the FI and the sample size (rs 0.50, P < 0.01) and a weak correlation between the FI the P value (rs 0.15 (P < 0.04)). CONCLUSIONS Randomized controlled trials in urology journals that study dichotomous outcomes often report fragile results that should be regarded with caution. Reporting the FI alongside P values may enhance the interpretation and implementation of urologic oncology RCT's.
Collapse
Affiliation(s)
- Leon Chertin
- Department of urology, Shamir Medical Center, IL
| | | | - Miki Haifler
- Department of urology, Chaim Sheba Medical Center, IL.
| |
Collapse
|
14
|
Grammatikopoulou MG, Nigdelis MP, Theodoridis X, Gkiouras K, Tranidou A, Papamitsou T, Bogdanos DP, Goulis DG. How fragile are Mediterranean diet interventions? A research-on-research study of randomised controlled trials. BMJ Nutr Prev Health 2021; 4:115-131. [PMID: 34308119 PMCID: PMC8258081 DOI: 10.1136/bmjnph-2020-000188] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2020] [Revised: 02/04/2021] [Accepted: 02/13/2021] [Indexed: 11/25/2022] Open
Abstract
INTRODUCTION The Mediterranean diet (MD) is a traditional regional dietary pattern and a healthy diet recommended for the primary and secondary prevention of various diseases and health conditions. Results from the higher level of primary evidence, namely randomised controlled trials (RCTs), are often used to produce dietary recommendations; however, the robustness of RCTs with MD interventions is unknown. METHODS A systematic search was conducted and all MD RCTs with dichotomous primary outcomes were extracted from PubMed. The fragility (FI) and the reverse fragility index (RFI) were calculated for the trials with significant and non-significant comparisons, respectively. RESULTS Out of 27 RCTs of parallel design, the majority failed to present a significant primary outcome, exhibiting an FI equal to 0. The median FI of the significant comparisons was 5, ranging between 1 and 39. More than half of the comparisons had an FI <5, indicating that the addition of 1-4 events to the treatment arm eliminated the statistical significance. For the comparisons with an FI=0, the RFI ranged between 1 and 29 (Median RFI: 7). When the included RCTs were stratified according to masking, the use of a composite primary endpoint, sample size, outcome category, or dietary adherence assessment method, no differences were exhibited in the FI and RFI between groups, except for the RFI among different compliance assessment methods. CONCLUSIONS In essence, the present study shows that even in the top tiers of evidence hierarchy, research on the MD may lack robustness, setting concerns for the formulation of nutrition recommendations.
Collapse
Affiliation(s)
- Maria G Grammatikopoulou
- Unit of Reproductive Endocrinology, 1st Department of Obstetrics and Gynecology, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Central Macedonia, Greece
- Rheumatology and Clinical Immunology, Faculty of Health Sciences, University of Thessaly, Larissa, Greece
- Nutritional Sciences & Dietetics, Faculty of Health Sciences, International Hellenic University, Thessaloniki, Greece
| | - Meletios P Nigdelis
- Unit of Reproductive Endocrinology, 1st Department of Obstetrics and Gynecology, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Central Macedonia, Greece
| | - Xenophon Theodoridis
- Rheumatology and Clinical Immunology, Faculty of Health Sciences, University of Thessaly, Larissa, Greece
- Medical School, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Central Macedonia, Greece
| | - Konstantinos Gkiouras
- Rheumatology and Clinical Immunology, Faculty of Health Sciences, University of Thessaly, Larissa, Greece
- Medical School, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Central Macedonia, Greece
| | - Antigoni Tranidou
- Medical School, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Central Macedonia, Greece
- Department of Endocrinology, Diabetes and Metabolism, Hippokration General Hospital of Thessaloniki, Thessaloniki, Central Macedonia, Greece
| | - Theodora Papamitsou
- Laboratory of Histology and Embryology, Medical School, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Central Macedonia, Greece
| | - Dimitrios P Bogdanos
- Rheumatology and Clinical Immunology, Faculty of Health Sciences, University of Thessaly, Larissa, Greece
- Division of Transplantation, Immunology and Mucosal Biology, MRC Centre for Transplantation, School of Medical Education, King's College London, London, UK
| | - Dimitrios G Goulis
- Unit of Reproductive Endocrinology, 1st Department of Obstetrics and Gynecology, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Central Macedonia, Greece
| |
Collapse
|
15
|
Degree of Anterolateral Ligament Injury Impacts Outcomes After Double-Bundle Anterior Cruciate Ligament Reconstruction. Arthroscopy 2021; 37:222-230. [PMID: 32949631 DOI: 10.1016/j.arthro.2020.09.003] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/08/2020] [Revised: 08/31/2020] [Accepted: 09/01/2020] [Indexed: 02/02/2023]
Abstract
PURPOSE To evaluate the effect of anterolateral ligament (ALL) injury identified on preoperative magnetic resonance imaging (MRI) on postoperative outcomes after double-bundle (DB) anterior cruciate ligament reconstruction (ACLR). METHODS For this retrospective study, the inclusion criteria were patients who were at least 3 years out of DB ACLR. Exclusion criteria included a delay in MRI over 4 weeks, delay in surgery over 6 months, single-bundle ACLR, and revision surgery. Enrolled patients were divided into 2 groups according to the ALL injury grade in preoperative MRI by a musculoskeletal radiologist who was blinded to the perioperative findings (the high-grade group with complete or nearly complete tear: n = 53 and the low-grade group with intact ALL or partial tear: n = 33). Knee laxity, clinical outcomes using the International Knee Documentation Committee (IKDC) examination form, and revision rates were compared at the last follow-up (8.1 ± 2.2 years). An independent t test was applied to compare continuous variables, and χ2 or Fisher exact test was used to compare the nominal variables. RESULTS The anterior translation was 3.2 ± 1.9 mm in the high-grade group and 1.6 ± 1.0 mm in the low-grade group (P < .001). The high-grade group showed 18 cases with a pivot-shift grade of 2 or 3 (40.0%); however, the low-grade group showed only 1 case with a pivot-shift grade 2 or 3 (3.0%) (P = .002). The high-grade group also showed inferior outcomes in the IKDC objective grade (grade A: 49.0%; grade B: 17.0%; grade C: 30.2%; grade D: 3.8% vs grade A: 90.9%; grade B: 6.1%; grade C: 3.0%; grade D: 0%, P = .001) and IKDC subjective score (87.5 ± 9.9 vs 93.9 ± 5.3, P < .001). In addition, the high-grade group showed a greater revision rate (11.3% vs 0%, P = .045). CONCLUSIONS DB ACLR for patients with high-grade ALL injury resulted in increased knee laxity, worse clinical outcomes, and higher revision rate compared to patients with low-grade ALL injury. LEVEL OF EVIDENCE Level III, retrospective comparative study.
Collapse
|
16
|
Choupoo NS, Das SK, Saikia P, Dey S, Ray S. How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines. Indian J Crit Care Med 2021; 25:773-779. [PMID: 34316171 PMCID: PMC8286372 DOI: 10.5005/jp-journals-10071-23895] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/26/2022] Open
Abstract
Objectives “Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016” provides guidelines in regard to prompt management and resuscitation of sepsis or septic shock. The study is aimed to assess the robustness of randomized controlled trials (RCTs) that formulate these guidelines in terms of fragility index and reverse fragility index. Method RCTs that contributed to these guidelines having parallel two-group design, 1:1 allocation ratio, and at least one dichotomous outcome were included in the study. The median fragility index was calculated for RCTs with significant statistical outcomes, whereas the median reverse fragility index was calculated for RCTs with nonsignificant statistical results. Results Hundred RCTs that met the inclusion criteria were analyzed. The median fragility index was 5.5 [95% confidence interval (CI) 1–30] and median reverse fragility index was 13 (95% CI 12.07–16.8) at a p value of 0.05. The median reverse fragility index was 16 (95% CI 10–26) at a p value of 0.01. Most of the RCTs included in this analysis were of good quality, having a median Jadad score of 6. Conclusion This analysis found that the surviving sepsis guidelines were based on highly robust RCTs with statistically insignificant results and on some moderately robust RCTs with statistically significant results. RCTs with statistically insignificant results were more robust than RCTs with statistically significant results in regard to these guidelines. Highlights The study assessed the robustness of randomized controlled trials (RCTs) that were used to formulate surviving sepsis guidelines. Most RCTs showed statistically nonsignificant results. RCTs with statistically significant results were moderately fragile whereas RCTs with nonsignificant results were more robust. How to cite this article Choupoo NS, Das SK, Saikia P, Dey S, Ray S. How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines. Indian J Crit Care Med 2021;25(7):773–779.
Collapse
Affiliation(s)
- Nang S Choupoo
- Department of Anesthesia, Atal Bihari Vajpayee Medical Institute and Dr RML Hospital, Delhi, India
| | - Saurabh K Das
- Department of Critical Care Medicine, Artemis Hospital, Gurugram, Haryana, India
| | - Priyam Saikia
- Department of Anaesthesiology and Critical Care, Gauhati Medical College and Hospital, Guwahati, Assam, India
| | - Samarjit Dey
- Department of Anaesthesia and Critical Care, AIIMS, Raipur, Chhattisgarh, India
| | - Sumit Ray
- Department of Critical Care, Holy Family Hospital, Delhi, India
| |
Collapse
|
17
|
Potter GE. Dismantling the Fragility Index: A demonstration of statistical reasoning. Stat Med 2020; 39:3720-3731. [PMID: 32781488 DOI: 10.1002/sim.8689] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2019] [Revised: 05/09/2020] [Accepted: 06/11/2020] [Indexed: 12/22/2022]
Abstract
The Fragility Index has been introduced as a complement to the P-value to summarize the statistical strength of evidence for a trial's result. The Fragility Index (FI) is defined in trials with two equal treatment group sizes, with a dichotomous or time-to-event outcome, and is calculated as the minimum number of conversions from nonevent to event in the treatment group needed to shift the P-value from Fisher's exact test over the .05 threshold. As the index lacks a well-defined probability motivation, its interpretation is challenging for consumers. We clarify what the FI may be capturing by separately considering two scenarios: (a) what the FI is capturing mathematically when the probability model is correct and (b) how well the FI captures violations of probability model assumptions. By calculating the posterior probability of a treatment effect, we show that when the probability model is correct, the FI inappropriately penalizes small trials for using fewer events than larger trials to achieve the same significance level. The analysis shows that for experiments conducted without bias, the FI promotes an incorrect intuition of probability, which has not been noted elsewhere and must be dispelled. We illustrate shortcomings of the FI's ability to quantify departures from model assumptions and contextualize the FI concept within current debate around the null hypothesis significance testing paradigm. Altogether, the FI creates more confusion than it resolves and does not promote statistical thinking. We recommend against its use. Instead, sensitivity analyses are recommended to quantify and communicate robustness of trial results.
Collapse
|
18
|
Das S, Xaviar S. Calculation of the fragility index of randomized controlled trials in epilepsy published in twelve major journals. Epilepsy Res 2020; 159:106258. [DOI: 10.1016/j.eplepsyres.2019.106258] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2019] [Revised: 12/08/2019] [Accepted: 12/17/2019] [Indexed: 12/27/2022]
|
19
|
Khan MS, Ochani RK, Shaikh A, Usman MS, Yamani N, Khan SU, Murad MH, Mandrola J, Doukky R, Krasuski RA. Fragility Index in Cardiovascular Randomized Controlled Trials. Circ Cardiovasc Qual Outcomes 2019; 12:e005755. [PMID: 31822121 DOI: 10.1161/circoutcomes.119.005755] [Citation(s) in RCA: 28] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
Abstract
BACKGROUND Efficacy of an intervention is commonly evaluated using P values, in addition to effect size measures such as absolute risk reduction, relative risk reduction, and numbers needed to treat. However, these measures are not always intuitive to clinicians. The fragility index (FI) is a more intuitive number that can facilitate interpretation but can only be used with binary outcomes. FI is the minimum number of patients who must be moved from the nonevent group to the event group to turn a significant result nonsignificant. In this retrospective analysis, we assessed the robustness of cardiovascular randomized controlled trials (RCTs), which report a positive (statistically significant) primary outcome by using the FI. METHODS AND RESULTS We searched Medline from 2007 to 2017 to identify cardiovascular RCTs published in 6 high impact journals (The Lancet, New England Journal of Medicine, Journal of the American Medical Association, Circulation, Journal of the American College of Cardiology and European Heart Journal). Only RCTs with sample sizes >500 and a 2-by-2 factorial design or dichotomous primary outcomes were selected. FI was calculated using a defined approach. Among the cohort of 123 RCTs that met inclusion criteria, median FI was 13 (interquartile range, 5-26). In 28 trials (22.8%), FI ranged between 1 and 4. In 37 trials (30.1%), number of patients lost to follow-up was higher than the FI. Pharmaceutical interventions had higher FI compared with other interventions, FI=19 (7-52; P=0.002). Median FI varied according to subspecialty (electrophysiology=2; heart failure=11; interventional cardiology=8; P=0.020) and multiregional RCTs had higher FI=22 (12-53.25; P=0.023). FI did not differ based on risk of bias indicators, funding, or publication year. CONCLUSIONS Considerable variations in FI were observed among cardiovascular trials, suggesting the need for careful interpretation of results, particularly when number of patients lost to follow-up exceeds FI.
Collapse
Affiliation(s)
- Muhammad Shahzeb Khan
- Department of Internal Medicine, John H. Stroger Jr. Hospital of Cook County, Chicago, IL (M.S.K., N.Y.)
| | - Rohan Kumar Ochani
- Department of Internal Medicine, Dow University of Health Sciences, Karachi, Pakistan (R.K.O., A.S., M.S.U.)
| | - Asim Shaikh
- Department of Internal Medicine, Dow University of Health Sciences, Karachi, Pakistan (R.K.O., A.S., M.S.U.)
| | - Muhammad Shariq Usman
- Department of Internal Medicine, Dow University of Health Sciences, Karachi, Pakistan (R.K.O., A.S., M.S.U.)
| | - Naser Yamani
- Department of Internal Medicine, John H. Stroger Jr. Hospital of Cook County, Chicago, IL (M.S.K., N.Y.)
| | - Safi U Khan
- Department of Internal Medicine, West Virginia University (S.U.K.)
| | - M Hassan Murad
- Evidence-based Practice Center, Mayo Clinic, Rochester, MN (M.H.M.)
| | | | - Rami Doukky
- Division of Cardiology, Rush University Medical Center, Chicago, IL (R.D.)
| | - Richard A Krasuski
- Department of Cardiovascular Medicine, Duke University Health System, Durham, NC (R.A.K.)
| |
Collapse
|