Powering Bias and Clinically Important Treatment Effects in Randomized Trials of Critical Illness

Delta Inflation, Optimism Bias, and Uncertainty in Clinical Trials

The phenomenon of delta inflation, in which design treatment effects tend to exceed observed treatment effects, has been documented in several therapeutic areas. Delta inflation has often been attributed to investigators' optimism bias, or an unwarranted belief in the efficacy of new treatments. In contrast, we argue that delta inflation may be a natural consequence of clinical equipoise, that is, genuine uncertainty about the relative benefits of treatments before a trial is initiated. We review alternative methodologies that can offer more direct evidence about investigators' beliefs, including Bayesian priors and forecasting analysis. The available evidence for optimism bias appears to be mixed, and can be assessed only where uncertainty is expressed explicitly at the trial design stage.

Heterogeneity, Bayesian thinking, and phenotyping in critical care: A primer

PURPOSE

To familiarize clinicians with the emerging concepts in critical care research of Bayesian thinking and personalized medicine through phenotyping and explain their clinical relevance by highlighting how they address the issues of frequent negative trials and heterogeneity of treatment effect.

SUMMARY

The past decades have seen many negative (effect-neutral) critical care trials of promising interventions, culminating in calls to improve the field's research through adopting Bayesian thinking and increasing personalization of critical care medicine through phenotyping. Bayesian analyses add interpretive power for clinicians as they summarize treatment effects based on probabilities of benefit or harm, contrasting with conventional frequentist statistics that either affirm or reject a null hypothesis. Critical care trials are beginning to include prospective Bayesian analyses, and many trials have undergone reanalysis with Bayesian methods. Phenotyping seeks to identify treatable traits to target interventions to patients expected to derive benefit. Phenotyping and subphenotyping have gained prominence in the most syndromic and heterogenous critical care disease states, acute respiratory distress syndrome and sepsis. Grouping of patients has been informative across a spectrum of clinically observable physiological parameters, biomarkers, and genomic data. Bayesian thinking and phenotyping are emerging as elements of adaptive clinical trials and predictive enrichment, paving the way for a new era of high-quality evidence. These concepts share a common goal, sifting through the noise of heterogeneity in critical care to increase the value of existing and future research.

CONCLUSION

The future of critical care medicine will inevitably involve modification of statistical methods through Bayesian analyses and targeted therapeutics via phenotyping. Clinicians must be familiar with these systems that support recommendations to improve decision-making in the gray areas of critical care practice.

Randomized controlled trials in resuscitation

Randomized controlled trials (RCTs) are a gold standard in research and crucial to our understanding of resuscitation science. Many trials in resuscitation have had neutral findings, questioning which treatments are effective in cardiac resuscitation. While it is possible than many interventions do not improve patient outcomes, it is also possible that the large proportion of neutral findings are partially due to design limitations. RCTs can be challenging to implement, and require extensive resources, time, and funding. In addition, conducting RCTs in the out-of-hospital setting provides unique challenges that must be considered for a successful trial. This article will outline many important aspects of conducting trials in resuscitation in the out-of-hospital setting including patient and outcome selection, trial design, and statistical analysis.

Achieving large-scale clinician adoption of AI-enabled decision support

Computerised decision support (CDS) tools enabled by artificial intelligence (AI) seek to enhance accuracy and efficiency of clinician decision-making at the point of care. Statistical models developed using machine learning (ML) underpin most current tools. However, despite thousands of models and hundreds of regulator-approved tools internationally, large-scale uptake into routine clinical practice has proved elusive. While underdeveloped system readiness and investment in AI/ML within Australia and perhaps other countries are impediments, clinician ambivalence towards adopting these tools at scale could be a major inhibitor. We propose a set of principles and several strategic enablers for obtaining broad clinician acceptance of AI/ML-enabled CDS tools.

Composite outcome measures in high-impact critical care randomised controlled trials: a systematic review

BACKGROUND

The use of composite outcome measures (COM) in clinical trials is increasing. Whilst their use is associated with benefits, several limitations have been highlighted and there is limited literature exploring their use within critical care. The primary aim of this study was to evaluate the use of COM in high-impact critical care trials, and compare study parameters (including sample size, statistical significance, and consistency of effect estimates) in trials using composite versus non-composite outcomes.

METHODS

A systematic review of 16 high-impact journals was conducted. Randomised controlled trials published between 2012 and 2022 reporting a patient important outcome and involving critical care patients, were included.

RESULTS

8271 trials were screened, and 194 included. 39.1% of all trials used a COM and this increased over time. Of those using a COM, only 52.6% explicitly described the outcome as composite. The median number of components was 2 (IQR 2-3). Trials using a COM recruited fewer participants (409 (198.8-851.5) vs 584 (300-1566, p = 0.004), and their use was not associated with increased rates of statistical significance (19.7% vs 17.8%, p = 0.380). Predicted effect sizes were overestimated in all but 6 trials. For studies using a COM the effect estimates were consistent across all components in 43.4% of trials. 93% of COM included components that were not patient important.

CONCLUSIONS

COM are increasingly used in critical care trials; however effect estimates are frequently inconsistent across COM components confounding outcome interpretations. The use of COM was associated with smaller sample sizes, and no increased likelihood of statistically significant results. Many of the limitations inherent to the use of COM are relevant to critical care research.

A maturity model for the scientific review of clinical trial designs and their informativeness

BACKGROUND

Informativeness, in the context of clinical trials, defines whether a study's results definitively answer its research questions with meaningful next steps. Many clinical trials end uninformatively. Clinical trial protocols are required to go through reviews in regulatory and ethical domains: areas that focus on specifics outside of trial design, biostatistics, and research methods. Private foundations and government funders rarely require focused scientific design reviews for these areas. There are no documented standards and processes, or even best practices, toward a capability for funders to perform scientific design reviews after their peer review process prior to a funding commitment.

MAIN BODY

Considering the investment in and standardization of ethical and regulatory reviews, and the prevalence of studies never finishing or failing to provide definitive results, it may be that scientific reviews of trial designs with a focus on informativeness offer the best chance for improved outcomes and return-on-investment in clinical trials. A maturity model is a helpful tool for knowledge transfer to help grow capabilities in a new area or for those looking to perform a self-assessment in an existing area. Such a model is offered for scientific design reviews of clinical trial protocols. This maturity model includes 11 process areas and 5 maturity levels. Each of the 55 process area levels is populated with descriptions on a continuum toward an optimal state to improve trial protocols in the areas of risk of failure or uninformativeness.

CONCLUSION

This tool allows for prescriptive guidance on next investments to improve attributes of post-funding reviews of trials, with a focus on informativeness. Traditional pre-funding peer review has limited capacity for trial design review, especially for detailed biostatistical and methodological review. Select non-industry funders have begun to explore or invest in post-funding review programs of grantee protocols, based on exemplars of such programs. Funders with a desire to meet fiduciary responsibilities and mission goals can use the described model to enhance efforts supporting trial participant commitment and faster cures.

Effects of duration of follow-up and lag in data collection on the performance of adaptive clinical trials

Different combined outcome-data lags (follow-up durations plus data-collection lags) may affect the performance of adaptive clinical trial designs. We assessed the influence of different outcome-data lags (0-105 days) on the performance of various multi-stage, adaptive trial designs (2/4 arms, with/without a common control, fixed/response-adaptive randomisation) with undesirable binary outcomes according to different inclusion rates (3.33/6.67/10 patients/day) under scenarios with no, small, and large differences. Simulations were conducted under a Bayesian framework, with constant stopping thresholds for superiority/inferiority calibrated to keep type-1 error rates at approximately 5%. We assessed multiple performance metrics, including mean sample sizes, event counts/probabilities, probabilities of conclusiveness, root mean squared errors (RMSEs) of the estimated effect in the selected arms, and RMSEs between the analyses at the time of stopping and the final analyses including data from all randomised patients. Performance metrics generally deteriorated when the proportions of randomised patients with available data were smaller due to longer outcome-data lags or faster inclusion, that is, mean sample sizes, event counts/probabilities, and RMSEs were larger, while the probabilities of conclusiveness were lower. Performance metric impairments with outcome-data lags ≤45 days were relatively smaller compared to those occurring with ≥60 days of lag. For most metrics, the effects of different outcome-data lags and lower proportions of randomised patients with available data were larger than those of different design choices, for example, the use of fixed versus response-adaptive randomisation. Increased outcome-data lag substantially affected the performance of adaptive trial designs. Trialists should consider the effects of outcome-data lags when planning adaptive trials.

The impact of high versus standard enteral protein provision on functional recovery following intensive care admission: Protocol for a pre-planned secondary Bayesian analysis of the PRECISe trial

BACKGROUND

The PRECISe trial is a pragmatic, multicenter randomized controlled trial that evaluates the effect of high versus standard enteral protein provision on functional recovery in adult, mechanically ventilated critically ill patients. The current protocol presents the rationale and analysis plan for an evaluation of the primary and secondary outcomes under the Bayesian framework, with an emphasis on clinically important effect sizes.

METHODS

This protocol was drafted in agreement with the ROBUST-statement, and is submitted for publication before database lock and primary data analysis. The primary outcome is health-related quality of life as measured by the EQ-5D-5L health utility score and is longitudinally assessed. Secondary outcomes comprise the 6-min walking test and handgrip strength over the entire follow-up period (longitudinal analyses), and 60-day mortality, duration of mechanical ventilation, and EQ-5D-5L health utility scores at 30, 90 and 180 days (cross-sectional). All analyses will primarily be performed under weakly informative priors. When available, informative priors elicited from contemporary literature will also be incorporated under alternative scenarios. In all other cases, objectively formulated skeptical and enthusiastic priors will be defined to assess the robustness of our results. Relevant identified subgroups were: patients with acute kidney injury, severe multi-organ failure and patients with or without sepsis. Results will be presented as absolute risk differences, mean differences, and odds ratios, with accompanying 95% credible intervals. Posterior probabilities will be estimated for clinically important benefit and harm.

DISCUSSION

The proposed secondary, pre-planned Bayesian analysis of the PRECISe trial will provide additional information on the effects of high protein on functional and clinical outcomes in critically ill patients, such as probabilistic interpretation, probabilities of clinically important effect sizes, and the integration of prior evidence. As such, it will complement the interpretation of the primary outcome as well as several secondary and subgroup analyses.

Real-world causal evidence for planned predictive enrichment in critical care trials: A scoping review

BACKGROUND

Randomised clinical trials in critical care are prone to inconclusiveness due, in part, to undue optimism about effect sizes and suboptimal accounting for heterogeneous treatment effects. Although causal evidence from rich real-world critical care can help overcome these challenges by informing predictive enrichment, no overview exists.

METHODS

We conducted a scoping review, systematically searching 10 general and speciality journals for reports published on or after 1 January 2018, of randomised clinical trials enrolling adult critically ill patients. We collected trial metadata on 22 variables including recruitment period, intervention type and early stopping (including reasons) as well as data on the use of causal evidence from secondary data for planned predictive enrichment.

RESULTS

We screened 9020 records and included 316 unique RCTs with a total of 268,563 randomised participants. One hundred seventy-three (55%) trials tested drug interventions, 101 (32%) management strategies and 42 (13%) devices. The median duration of enrolment was 2.2 (IQR: 1.3-3.4) years, and 83% of trials randomised less than 1000 participants. Thirty-six trials (11%) were restricted to COVID-19 patients. Of the 55 (17%) trials that stopped early, 23 (42%) used predefined rules; futility, slow enrolment and safety concerns were the commonest stopping reasons. None of the included RCTs had used causal evidence from secondary data for planned predictive enrichment.

CONCLUSION

Work is needed to harness the rich multiverse of critical care data and establish its utility in critical care RCTs. Such work will likely need to leverage methodology from interventional and analytical epidemiology as well as data science.

Design and Execution of Clinical Trials in the Cardiac Intensive Care Unit

Clinical practice in the contemporary cardiac intensive care unit (CICU) has evolved significantly over the last several decades. With more frequent multisystem organ failure, increasing use of advanced respiratory support, and the advent of new mechanical circulatory support platforms, clinicians in the CICU are increasingly managing patients with complex comorbid disease in addition to their high-acuity cardiovascular illnesses. Here, the authors discuss challenges associated with traditional trial design in the CICU setting and review novel clinical trial designs that may facilitate better evidence generation in the CICU.

How Electronic Medical Record Integration Can Support More Efficient Critical Care Clinical Trials

Large volumes of data are collected on critically ill patients, and using data science to extract information from the electronic medical record (EMR) and to inform the design of clinical trials represents a new opportunity in critical care research. Using improved methods of phenotyping critical illnesses, subject identification and enrollment, and targeted treatment group assignment alongside newer trial designs such as adaptive platform trials can increase efficiency while lowering costs. Some tools such as the EMR to automate data collection are already in use. Refinement of data science approaches in critical illness research will allow for better clinical trials and, ultimately, improved patient outcomes.

Randomized Controlled Trials in ICU in the Four Highest-Impact General Medicine Journals

OBJECTIVE

To study ICU trials published in the four highest-impact general medicine journals by comparing them with concurrently published non-ICU trials in the same journals.

DATA SOURCES

PubMed was searched for randomized controlled trials (RCTs) published between January 2014 and October 2021 in the New England Journal of Medicine , The Lancet , the Journal of the American Medical Association , and the British Medical Journal.

STUDY SELECTION

Original RCT publications investigating any type of intervention in any patient population.

DATA EXTRACTION

ICU RCTs were defined as RCTs exclusively including patients admitted to the ICU. Year and journal of publication, sample size, study design, funding source, study outcome, type of intervention, Fragility Index (FI), and Fragility Quotient were collected.

DATA SYNTHESIS

A total of 2,770 publications were screened. Of 2,431 original RCTs, 132 (5.4%) were ICU RCTs, gradually rising from 4% in 2014 to 7.5% in 2021. ICU RCTs and non-ICU RCTs included a comparable number of patients (634 vs 584, p = 0.528). Notable differences for ICU RCTs were the low occurrence of commercial funding (5% vs 36%, p < 0.001), the low number of RCTs that reached statistical significance (29% vs 65%, p < 0.001), and the low FI when they did reach significance (3 vs 12, p = 0.008).

CONCLUSIONS

In the last 8 years, RCTs in ICU medicine made up a meaningful, and growing, portion of RCTs published in high-impact general medicine journals. In comparison with concurrently published RCTs in non-ICU disciplines, statistical significance was rare and often hinged on the outcome events of just a few patients. Increased attention should be paid to realistic expectations of treatment effects when designing ICU RCTs to detect differences in treatment effects that are reliable and clinically relevant.

Adjunctive Vasopressors in Patients with Septic Shock: Protocol for a Systematic Review and Meta-Analysis

Background

Over one-third of patients with septic shock have adjunctive vasopressors added to first-line vasopressors. However, no randomized trial has detected improved mortality with adjunctive vasopressors. Published systematic reviews and meta-analysis have sought to inform the use of adjunctive vasopressors, yet each published review has limitations that hinder its interpretation. This review aims to overcome the limitations of previous reviews by systematically synthesizing the direct evidence for adjunctive vasopressor therapy use in adult patients with septic shock.

Methods

We will conduct a systematic review and meta-analysis of randomized controlled trials evaluating adjunctive vasopressors (vasopressin analogues, angiotensin II, hydroxocobalamin, methylene blue, and catecholamine analogues) in adult patients with septic shock. Relevant studies will be identified through comprehensive searches of MEDLINE, Embase, CENTRAL, and reference lists of previous systematic reviews. Only randomized trials comparing adjunctive vasopressors (>75% of subjects on vasopressors at enrollment) to standard care vasopressors in adults with septic shock (>75% of subjects having septic shock) will be included. Titles and abstracts will be screened, full-text articles assessed for eligibility, and data extracted from included studies. Outcomes of interest include short-term mortality, intermediate-term mortality, kidney replacement therapy, digital/peripheral ischemia, and venous thromboembolism. Pairwise meta-analysis using a random-effects model will be utilized to estimate the risk ratio for the outcomes. Risk of bias will be adjudicated with the Cochrane Risk of Bias 2 tool, and GRADE will be used to rate the certainty of the body of evidence.

Discussion

Although adjunctive vasopressors are commonly used in patients with septic shock their effect on patient-important outcomes is unclear. This study is planned to use rigorous systematic review methodology, including strict adhere to established guidelines, in order to overcome limitations of previously-published reviews and inform clinical practice and treatment guidelines for the use of adjunctive vasopressors in adults with septic shock.

Systematic review registration

PROSPERO CRD4202327984.

The Single-Syringe Versus the Double-Syringe Techniques of Adenosine Administration for Supraventricular Tachycardia: A Systematic Review and Meta-Analysis

INTRODUCTION

The intravenous double-syringe technique (DST) of adenosine administration is the first-line treatment for stable supraventricular tachycardia (SVT). Alternatively, the single-syringe technique (SST) was recently found to be potentially beneficial in several studies. This study aimed to perform a meta-analysis of the SST versus the DST of adenosine administration for the treatment of SVT.

METHODS

We assessed EMBASE, PubMed, Cochrane, and ClinicalTrials.gov databases for randomized controlled trials (RCTs) and non-randomized studies of intervention (NRSIs) comparing the DST to the SST of adenosine administration in patients with SVT. Outcomes included termination rate, termination rate at first dose, total administered dose, adverse effects, and discharge rate.

RESULTS

We included four studies (three RCTs and one NRSI) with a total of 178 patients, of whom 99 underwent the SST of adenosine administration. No significant difference was found between treatment groups regarding termination rate, termination rate restricted to RCTs, total administered dose, and discharge rate. Termination rate at first dose (odds ratio 2.87; confidence interval 1.11-7.41; p = 0.03; I² = 0%) was significantly increased in patients who received the SST. Major adverse effects were observed in only one study.

CONCLUSIONS

The SST is probably as safe as the DST and at least as effective for SVT termination, SVT termination at first dose, and discharge rate from the emergency department. However, definitive superiority of one technique is not feasible given the limited sample size.

REGISTRATION

PROSPERO identifier nº CRD42022345125.

Comparative Effectiveness of Fludrocortisone and Hydrocortisone vs Hydrocortisone Alone Among Patients With Septic Shock

Importance

Patients with septic shock may benefit from the initiation of corticosteroids. However, the comparative effectiveness of the 2 most studied corticosteroid regimens (hydrocortisone with fludrocortisone vs hydrocortisone alone) is unclear.

Objective

To compare the effectiveness of adding fludrocortisone to hydrocortisone vs hydrocortisone alone among patients with septic shock using target trial emulation.

Design, Setting, and Participants

This retrospective cohort study from 2016 to 2020 used the enhanced claims-based Premier Healthcare Database, which included approximately 25% of US hospitalizations. Participants were adult patients hospitalized with septic shock and receiving norepinephrine who began hydrocortisone treatment. Data analysis was performed from May 2022 to December 2022.

Exposure

Addition of fludrocortisone on the same calendar day that hydrocortisone treatment was initiated vs use of hydrocortisone alone.

Main Outcome and Measures

Composite of hospital death or discharge to hospice. Adjusted risk differences were calculated using doubly robust targeted maximum likelihood estimation.

Results

Analyses included 88 275 patients, 2280 who began treatment with hydrocortisone-fludrocortisone (median [IQR] age, 64 [54-73] years; 1041 female; 1239 male) and 85 995 (median [IQR] age, 67 [57-76] years; 42 136 female; 43 859 male) who began treatment with hydrocortisone alone. The primary composite outcome of death in hospital or discharge to hospice occurred among 1076 (47.2%) patients treated with hydrocortisone-fludrocortisone vs 43 669 (50.8%) treated with hydrocortisone alone (adjusted absolute risk difference, -3.7%; 95% CI, -4.2% to -3.1%; P < .001).

Conclusions and Relevance

In this comparative effectiveness cohort study among adult patients with septic shock who began hydrocortisone treatment, the addition of fludrocortisone was superior to hydrocortisone alone.

Debate: Intermittent Hemodialysis versus Continuous Kidney Replacement Therapy in the Critically Ill Patient: The Argument for CKRT

Continuous kidney replacement therapy (CKRT) is well entrenched as one of the dominant KRT modalities in modern critical care practice. Since its introduction four decades ago, there have been considerable innovations in CKRT machines that have improved precision, safety, and simplicity. CKRT is the preferred KRT modality for critically ill patients with hemodynamic instability. Early physical therapy and rehabilitation can be feasibly and safely provided to patients connected to CKRT, thus obviating concerns about immobility. Although randomized clinical trials have not shown a mortality difference when comparing CKRT and intermittent hemodialysis, CKRT allows precision delivery of solute and fluid removal that can be readily adjusted in the face of dynamic circumstances. Accumulated evidence from observational studies, although susceptible to bias, has shown that CKRT, when compared with intermittent hemodialysis, is associated with better short- and long-term kidney recovery and KRT independence. Critical care medicine encompasses a wide range of sick patients, and no single KRT modality is likely to ideally suit every patient in every context and for every condition. The provision of KRT represents a spectrum of modalities to which patients can flexibly transition in response to their evolving condition. As a vital tool for organ support in the intensive care unit, CKRT enables the personalization of KRT to meet the clinical demands of patients during the most severe phases of their illness.

Causal inference for planning randomised critical care trials: Protocol for a scoping review

BACKGROUND

Randomised clinical trials in critical care are prone to inconclusiveness owing, in part, to undue optimism about effect sizes and suboptimal accounting for heterogeneous treatment effects. Planned predictive enrichment based on secondary critical care data (often very rich with respect to both data types and temporal granularity) and causal inference methods may help overcome these challenges, but no overview exists about their use to this end.

METHODS

We will conduct a scoping review to assess the extent and nature of the use of causal inference from secondary data for planned predictive enrichment of randomised clinical trials in critical care. We will systematically search 10 general and specialty journals for reports published on or after 1 January 2018, of randomised clinical trials enrolling adult critically ill patients. We will collect trial metadata (e.g., recruitment period and phase) and, when available, information pertaining to the focus of the review (predictive enrichment based on causal inference estimates from secondary data): causal inference methods, estimation techniques and software used; types of patient populations; data provenance, types and models; and the availability of the data (public or not). The results will be reported in a descriptive manner.

DISCUSSION

The outlined scoping review aims to assess the use of causal inference methods and secondary data for planned predictive enrichment in randomised critical care trials. This will help guide methodological improvements to increase the utility, and facilitate the use, of causal inference estimates when planning such trials in the future.

A Bayesian reanalysis of the Standard versus Accelerated Initiation of Renal-Replacement Therapy in Acute Kidney Injury (STARRT-AKI) trial

Background

Timing of initiation of kidney-replacement therapy (KRT) in critically ill patients remains controversial. The Standard versus Accelerated Initiation of Renal-Replacement Therapy in Acute Kidney Injury (STARRT-AKI) trial compared two strategies of KRT initiation (accelerated versus standard) in critically ill patients with acute kidney injury and found neutral results for 90-day all-cause mortality. Probabilistic exploration of the trial endpoints may enable greater understanding of the trial findings. We aimed to perform a reanalysis using a Bayesian framework.

Methods

We performed a secondary analysis of all 2927 patients randomized in multi-national STARRT-AKI trial, performed at 168 centers in 15 countries. The primary endpoint, 90-day all-cause mortality, was evaluated using hierarchical Bayesian logistic regression. A spectrum of priors includes optimistic, neutral, and pessimistic priors, along with priors informed from earlier clinical trials. Secondary endpoints (KRT-free days and hospital-free days) were assessed using zero–one inflated beta regression.

Results

The posterior probability of benefit comparing an accelerated versus a standard KRT initiation strategy for the primary endpoint suggested no important difference, regardless of the prior used (absolute difference of 0.13% [95% credible interval [CrI] − 3.30%; 3.40%], − 0.39% [95% CrI − 3.46%; 3.00%], and 0.64% [95% CrI − 2.53%; 3.88%] for neutral, optimistic, and pessimistic priors, respectively). There was a very low probability that the effect size was equal or larger than a consensus-defined minimal clinically important difference. Patients allocated to the accelerated strategy had a lower number of KRT-free days (median absolute difference of − 3.55 days [95% CrI − 6.38; − 0.48]), with a probability that the accelerated strategy was associated with more KRT-free days of 0.008. Hospital-free days were similar between strategies, with the accelerated strategy having a median absolute difference of 0.48 more hospital-free days (95% CrI − 1.87; 2.72) compared with the standard strategy and the probability that the accelerated strategy had more hospital-free days was 0.66.

Conclusions

In a Bayesian reanalysis of the STARRT-AKI trial, we found very low probability that an accelerated strategy has clinically important benefits compared with the standard strategy. Patients receiving the accelerated strategy probably have fewer days alive and KRT-free. These findings do not support the adoption of an accelerated strategy of KRT initiation.

Principal investigators over-optimistically forecast scientific and operational outcomes for clinical trials

Objective

To assess the accuracy of principal investigators’ (PIs) predictions about three events for their own clinical trials: positivity on trial primary outcomes, successful recruitment and timely trial completion.

Study design and setting

A short, electronic survey was used to elicit subjective probabilities within seven months of trial registration. When trial results became available, prediction skill was calculated using Brier scores (BS) and compared against uninformative prediction (i.e. predicting 50% all of the time).

Results

740 PIs returned surveys (16.7% response rate). Predictions on all three events tended to exceed observed event frequency. Averaged PI skill did not surpass uninformative predictions (e.g., BS = 0.25) for primary outcomes (BS = 0.25, 95% CI 0.20, 0.30) and were significantly worse for recruitment and timeline predictions (BS 0.38, 95% CI 0.33, 0.42; BS = 0.52, 95% CI 0.50, 0.55, respectively). PIs showed poor calibration for primary outcome, recruitment, and timelines (calibration index = 0.064, 0.150 and 0.406, respectively), modest discrimination in primary outcome predictions (AUC = 0.76, 95% CI 0.65, 0.85) but minimal discrimination in the other two outcomes (AUC = 0.64, 95% CI 0.57, 0.70; and 0.55, 95% CI 0.47, 0.62, respectively).

Conclusion

PIs showed overconfidence in favorable outcomes and exhibited limited skill in predicting scientific or operational outcomes for their own trials. They nevertheless showed modest ability to discriminate between positive and non-positive trial outcomes. Low survey response rates may limit generalizability.

Critical Care Randomized Trials Demonstrate Power Failure: A Low Positive Predictive Value of Findings in the Critical Care Research Field

BACKGROUND

We aimed to determine the post-hoc power of randomized controlled trials (RCTs) in critical care, and describe the implications for long-term positive (PPV) and negative predictive value (NPV) of statistically significant and non-significant findings respectively in the research field.

METHODS

We reviewed three cohorts of RCTs. "Adult-RCTs" were 216 multicenter RCTs with a mortality outcome from a published systematic review. "Pediatric-RCTs" were 120 RCTs with a mortality outcome, obtained by search of picutrials.net. "Consecutive-RCTs" were 90 recent RCTs obtained by screening publications in 6 journals. Post-hoc power for each study was calculated at α 0.05 and 0.005, for measures of small, medium, and large effect-size, using G*Power software. Long-run expected PPV and NPV of critical care research field findings were then calculated.

RESULTS

With α 0.05, post-hoc power for small effect-size was very low in all RCT-cohorts (eg, median 24% in Adult-RCTs). For medium effect-size, post-hoc power was low, except for Adult-RCTs (eg, median 9% in Pediatric-RCTs). For large effect-size, post-hoc power for non-human-animal Consecutive-RCTs was low (median 32%). With α 0.005, post-hoc power was even lower. The corollary was that both PPV and NPV were poor for small effect-size, unless α 0.005 was used. Even with α 0.005, with realistic (vs. optimistic) prior probability of the alternative hypothesis, the PPV was low (eg, in Adult-RCTs 57.1% vs. 92.3%). Adding mild bias (0.1) reduced the PPV even further. For medium effect-size both PPV and NPV were better; nevertheless, with α 0.05 and realistic prior probability of the alternative hypothesis the PPV was poor, and with α 0.005 and mild bias (0.1) the PPV was very low (eg, Adult-RCTs median 44.1%).

CONCLUSIONS

To improve the predictive value of findings in the critical care research field, RCTs should be designed to have 80% power for realistic effect-size at α 0.005.

Advancing precision medicine for acute respiratory distress syndrome

Acute respiratory distress syndrome (ARDS) is a heterogeneous clinical syndrome. Understanding of the complex pathways involved in lung injury pathogenesis, resolution, and repair has grown considerably in recent decades. Nevertheless, to date, only therapies targeting ventilation-induced lung injury have consistently proven beneficial, and despite these gains, ARDS morbidity and mortality remain high. Many candidate therapies with promise in preclinical studies have been ineffective in human trials, probably at least in part due to clinical and biological heterogeneity that modifies treatment responsiveness in human ARDS. A precision medicine approach to ARDS seeks to better account for this heterogeneity by matching therapies to subgroups of patients that are anticipated to be most likely to benefit, which initially might be identified in part by assessing for heterogeneity of treatment effect in clinical trials. In October 2019, the US National Heart, Lung, and Blood Institute convened a workshop of multidisciplinary experts to explore research opportunities and challenges for accelerating precision medicine in ARDS. Topics of discussion included the rationale and challenges for a precision medicine approach in ARDS, the roles of preclinical ARDS models in precision medicine, essential features of cohort studies to advance precision medicine, and novel approaches to clinical trials to support development and validation of a precision medicine strategy. In this Position Paper, we summarise workshop discussions, recommendations, and unresolved questions for advancing precision medicine in ARDS. Although the workshop took place before the COVID-19 pandemic began, the pandemic has highlighted the urgent need for precision therapies for ARDS as the global scientific community grapples with many of the key concepts, innovations, and challenges discussed at this workshop.

Randomised clinical trials in critical care: past, present and future

Randomised clinical trials (RCTs) are the gold standard for providing unbiased evidence of intervention effects. Here, we provide an overview of the history of RCTs and discuss the major challenges and limitations of current critical care RCTs, including overly optimistic effect sizes; unnuanced conclusions based on dichotomization of results; limited focus on patient-centred outcomes other than mortality; lack of flexibility and ability to adapt, increasing the risk of inconclusive results and limiting knowledge gains before trial completion; and inefficiency due to lack of re-use of trial infrastructure. We discuss recent developments in critical care RCTs and novel methods that may provide solutions to some of these challenges, including a research programme approach (consecutive, complementary studies of multiple types rather than individual, independent studies), and novel design and analysis methods. These include standardization of trial protocols; alternative outcome choices and use of core outcome sets; increased acceptance of uncertainty, probabilistic interpretations and use of Bayesian statistics; novel approaches to assessing heterogeneity of treatment effects; adaptation and platform trials; and increased integration between clinical trials and clinical practice. We outline the advantages and discuss the potential methodological and practical disadvantages with these approaches. With this review, we aim to inform clinicians and researchers about conventional and novel RCTs, including the rationale for choosing one or the other methodological approach based on a thorough discussion of pros and cons. Importantly, the most central feature remains the randomisation, which provides unparalleled restriction of confounding compared to non-randomised designs by reducing confounding to chance.

Reverse Bayesian Implications of p-Values Reported in Critical Care Randomized Trials

Background

Misinterpretations of the p-value in null-hypothesis statistical testing are common. We aimed to determine the implications of observed p-values in critical care randomized controlled trials (RCTs).

Methods

We included three cohorts of published RCTs: Adult-RCTs reporting a mortality outcome, Pediatric-RCTs reporting a mortality outcome, and recent Consecutive-RCTs reporting p-value ≤.10 in six higher-impact journals. We recorded descriptive information from RCTs. Reverse Bayesian implications of obtained p-values were calculated, reported as percentages with inter-quartile ranges.

Results

Obtained p-value was ≤.005 in 11/216 (5.1%) Adult-RCTs, 2/120 (1.7%) Pediatric-RCTs, and 37/90 (41.1%) Consecutive-RCTs. An obtained p-value .05–.0051 had high False Positive Rates; in Adult-RCTs, minimum (assuming prior probability of the alternative hypothesis was 50%) and realistic (assuming prior probability of the alternative hypothesis was 10%) False Positive Rates were 16.7% [11.2, 21.8] and 64.3% [53.2, 71.4]. An obtained p-value ≤.005 had lower False Positive Rates; in Adult-RCTs the realistic False Positive Rate was 7.7% [7.7, 16.0]. The realistic probability of the alternative hypothesis for obtained p-value .05–.0051 (ie, Positive Predictive Value) was 28.0% [24.1, 34.8], 30.6% [27.7, 48.5], 29.3% [24.3, 41.0], and 32.7% [24.1, 43.5] for Adult-RCTs, Pediatric-RCTs, Consecutive-RCTs primary and secondary outcome, respectively. The maximum Positive Predictive Value for p-value category .05–.0051 was median 77.8%, 79.8%, 78.8%, and 81.4% respectively. To have maximum or realistic Positive Predictive Value >90% or >80%, RCTs needed to have obtained p-value ≤.005. The credibility of p-value .05–.0051 findings were easy to challenge, and the credibility to rule-out an effect with p-value >.05 to .10 was low. The probability that a replication study would obtain p-value ≤.05 did not approach 90% unless the obtained p-value was ≤.005.

Conclusions

Unless the obtained p-value was ≤.005, the False Positive Rate was high, and the Positive Predictive Value and probability of replication of “statistically significant” findings were low.

Clinical trials in critical care: can a Bayesian approach enhance clinical and scientific decision making?

Recent Bayesian reanalyses of prominent trials in critical illness have generated controversy by contradicting the initial conclusions based on conventional frequentist analyses. Many clinicians might be sceptical that Bayesian analysis, a philosophical and statistical approach that combines prior beliefs with data to generate probabilities, provides more useful information about clinical trials than the frequentist approach. In this Personal View, we introduce clinicians to the rationale, process, and interpretation of Bayesian analysis through a systematic review and reanalysis of interventional trials in critical illness. In the majority of cases, Bayesian and frequentist analyses agreed. In the remainder, Bayesian analysis identified interventions where benefit was probable despite the absence of statistical significance, where interpretation depended substantially on choice of prior distribution, and where benefit was improbable despite statistical significance. Bayesian analysis in critical care medicine can help to distinguish harm from uncertainty and establish the probability of clinically important benefit for clinicians, policy makers, and patients.