Delta inflation: a bias in the design of randomized controlled trials in critical care medicine

Delta Inflation, Optimism Bias, and Uncertainty in Clinical Trials

The phenomenon of delta inflation, in which design treatment effects tend to exceed observed treatment effects, has been documented in several therapeutic areas. Delta inflation has often been attributed to investigators' optimism bias, or an unwarranted belief in the efficacy of new treatments. In contrast, we argue that delta inflation may be a natural consequence of clinical equipoise, that is, genuine uncertainty about the relative benefits of treatments before a trial is initiated. We review alternative methodologies that can offer more direct evidence about investigators' beliefs, including Bayesian priors and forecasting analysis. The available evidence for optimism bias appears to be mixed, and can be assessed only where uncertainty is expressed explicitly at the trial design stage.

Accuracy of Event Rate and Effect Size Estimation in Major Cardiovascular Trials: A Systematic Review

Importance

For the design of a randomized clinical trial (RCT), estimation of the expected event rate and effect size of an intervention is needed to calculate the sample size. Overestimation may lead to an underpowered trial.

Objective

To evaluate the accuracy of published estimates of event rate and effect size in contemporary cardiovascular RCTs.

Evidence Review

A systematic search was conducted in MEDLINE for multicenter cardiovascular RCTs associated with MeSH (Medical Subject Headings) terms for cardiovascular diseases published in the New England Journal of Medicine, JAMA, or the Lancet between January 1, 2010, and December 31, 2019. Identified trials underwent abstract review; eligible trials then underwent full review, and those with insufficiently reported data were excluded. Data were extracted from the original publication or the study protocol, and a random-effects model was used for data pooling. This review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guideline. The primary outcome was the accuracy of event rate and effect size estimation. Accuracy was determined by comparing the observed event rate in the control group and the effect size with their hypothesized values. Linear regression was used to determine the association between estimation accuracy and trial characteristics.

Findings

Of the 873 RCTs identified, 374 underwent full review and 30 were subsequently excluded, resulting in 344 trials for analysis. The median observed event rate was 9.0% (IQR, 4.3% to 21.4%), which was significantly lower than the estimated event rate of 11.0% (IQR, 6.0% to 25.0%) with a median deviation of -12.3% (95% CI, -16.4% to -5.6%; P < .001). More than half of the trials (196 [61.1%]) overestimated the expected event rate. Accuracy of event rate estimation was associated with a higher likelihood of refuting the null hypothesis (0.13 [95% CI, 0.01 to 0.25]; P = .03). The median observed effect size in superiority trials was 0.91 (IQR, 0.74 to 0.99), which was significantly lower than the estimated effect size of 0.72 (IQR, 0.60 to 0.80), indicating a median overestimation of 23.1% (95% CI, 17.9% to 28.3%). A total of 216 trials (82.1%) overestimated the effect size.

Conclusions and Relevance

In this systematic review of contemporary cardiovascular RCTs, event rates of the primary end point and effect sizes of an intervention were frequently overestimated. This overestimation may have contributed to the inability to adequately test the trial hypothesis.

Real-world causal evidence for planned predictive enrichment in critical care trials: A scoping review

BACKGROUND

Randomised clinical trials in critical care are prone to inconclusiveness due, in part, to undue optimism about effect sizes and suboptimal accounting for heterogeneous treatment effects. Although causal evidence from rich real-world critical care can help overcome these challenges by informing predictive enrichment, no overview exists.

METHODS

We conducted a scoping review, systematically searching 10 general and speciality journals for reports published on or after 1 January 2018, of randomised clinical trials enrolling adult critically ill patients. We collected trial metadata on 22 variables including recruitment period, intervention type and early stopping (including reasons) as well as data on the use of causal evidence from secondary data for planned predictive enrichment.

RESULTS

We screened 9020 records and included 316 unique RCTs with a total of 268,563 randomised participants. One hundred seventy-three (55%) trials tested drug interventions, 101 (32%) management strategies and 42 (13%) devices. The median duration of enrolment was 2.2 (IQR: 1.3-3.4) years, and 83% of trials randomised less than 1000 participants. Thirty-six trials (11%) were restricted to COVID-19 patients. Of the 55 (17%) trials that stopped early, 23 (42%) used predefined rules; futility, slow enrolment and safety concerns were the commonest stopping reasons. None of the included RCTs had used causal evidence from secondary data for planned predictive enrichment.

CONCLUSION

Work is needed to harness the rich multiverse of critical care data and establish its utility in critical care RCTs. Such work will likely need to leverage methodology from interventional and analytical epidemiology as well as data science.

Using the Delphi process to determine the minimum clinically important effect size for the Balanced-2 randomised controlled trial

BACKGROUND

The sample size calculation is an important step in designing randomised controlled trials. For a trial comparing a control and an intervention group, where the outcome is binary, the sample size calculation requires choosing values for the anticipated event rates in both the control and intervention groups (the effect size), and the error rates. The Difference ELicitation in TriAls guidance recommends that the effect size should be both realistic, and clinically important to stakeholder groups. Overestimating the effect size leads to sample sizes that are too small to reliably detect the true population effect size, which in turn results in low achieved power. In this study, we use the Delphi approach to gain consensus on what the minimum clinically important effect size is for Balanced-2, a randomised controlled trial comparing processed electroencephalogram-guided 'light' to 'deep' general anaesthesia on the incidence of postoperative delirium in older adults undergoing major surgery.

METHODS

Delphi rounds were conducted using electronic surveys. Surveys were administered to two stakeholder groups: specialist anaesthetists from a general adult department in Auckland City Hospital, New Zealand (Group 1), and specialist anaesthetists with expertise in clinical research, identified from the Australian and New Zealand College of Anaesthetist's Clinical Trials Network (Group 2). A total of 187 anaesthetists were invited to participate (81 from Group 1 and 106 from Group 2). Results from each Delphi round were summarised and presented in subsequent rounds until consensus was reached (>70% agreement).

RESULTS

The overall response rate for the first Delphi survey was 47% (88/187). The median minimum clinically important effect size was 5.0% (interquartile range: 5.0-10.0) for both stakeholder groups. The overall response rate for the second Delphi survey was 51% (95/187). Consensus was reached after the second round, as 74% of respondents in Group 1 and 82% of respondents in Group 2 agreed with the median effect size. The combined minimum clinically important effect size across both groups was 5.0% (interquartile range: 3.0-6.5).

CONCLUSIONS

This study demonstrates that surveying stakeholder groups using a Delphi process is a simple way of defining a minimum clinically important effect size, which aids the sample size calculation and determines whether a randomised study is feasible.

Interpreting frequentist hypothesis tests: insights from Bayesian inference

Randomized controlled trials are one of the best ways of quantifying the effectiveness of medical interventions. Therefore, when the authors of a randomized superiority trial report that differences in the primary outcome between the intervention group and the control group are "significant" (i.e., P ≤ 0.05), we might assume that the intervention has an effect on the outcome. Similarly, when differences between the groups are "not significant," we might assume that the intervention does not have an effect on the outcome. Nevertheless, both assumptions are frequently incorrect.In this article, we explore the relationship that exists between real treatment effects and declarations of statistical significance based on P values and confidence intervals. We explain why, in some circumstances, the chance an intervention is ineffective when P ≤ 0.05 exceeds 25% and the chance an intervention is effective when P > 0.05 exceeds 50%.Over the last decade, there has been increasing interest in Bayesian methods as an alternative to frequentist hypothesis testing. We provide a robust but nontechnical introduction to Bayesian inference and explain why a Bayesian posterior distribution overcomes many of the problems associated with frequentist hypothesis testing.Notwithstanding the current interest in Bayesian methods, frequentist hypothesis testing remains the default method for statistical inference in medical research. Therefore, we propose an interim solution to the "significance problem" based on simplified Bayesian metrics (e.g., Bayes factor, false positive risk) that can be reported along with traditional P values and confidence intervals. We calculate these metrics for four well-known multicentre trials. We provide links to online calculators so readers can easily estimate these metrics for published trials. In this way, we hope decisions on incorporating the results of randomized trials into clinical practice can be enhanced, minimizing the chance that useful treatments are discarded or that ineffective treatments are adopted.

Randomized Controlled Trials in ICU in the Four Highest-Impact General Medicine Journals

OBJECTIVE

To study ICU trials published in the four highest-impact general medicine journals by comparing them with concurrently published non-ICU trials in the same journals.

DATA SOURCES

PubMed was searched for randomized controlled trials (RCTs) published between January 2014 and October 2021 in the New England Journal of Medicine , The Lancet , the Journal of the American Medical Association , and the British Medical Journal.

STUDY SELECTION

Original RCT publications investigating any type of intervention in any patient population.

DATA EXTRACTION

ICU RCTs were defined as RCTs exclusively including patients admitted to the ICU. Year and journal of publication, sample size, study design, funding source, study outcome, type of intervention, Fragility Index (FI), and Fragility Quotient were collected.

DATA SYNTHESIS

A total of 2,770 publications were screened. Of 2,431 original RCTs, 132 (5.4%) were ICU RCTs, gradually rising from 4% in 2014 to 7.5% in 2021. ICU RCTs and non-ICU RCTs included a comparable number of patients (634 vs 584, p = 0.528). Notable differences for ICU RCTs were the low occurrence of commercial funding (5% vs 36%, p < 0.001), the low number of RCTs that reached statistical significance (29% vs 65%, p < 0.001), and the low FI when they did reach significance (3 vs 12, p = 0.008).

CONCLUSIONS

In the last 8 years, RCTs in ICU medicine made up a meaningful, and growing, portion of RCTs published in high-impact general medicine journals. In comparison with concurrently published RCTs in non-ICU disciplines, statistical significance was rare and often hinged on the outcome events of just a few patients. Increased attention should be paid to realistic expectations of treatment effects when designing ICU RCTs to detect differences in treatment effects that are reliable and clinically relevant.

Association between days alive without life support/out of hospital and health-related quality of life

BACKGROUND

Trials in critically ill patients increasingly focus on days alive without life support (DAWOLS) or days alive out of hospital (DAOOH) and health-related quality of life (HRQoL). DAWOLS and DAOOH convey more information than mortality and are simpler and faster to collect than HRQoL. However, whether these outcomes are associated with HRQoL is uncertain. We thus aimed to assess the associations between DAWOLS and DAOOH and long-term HRQoL.

METHODS

Secondary analysis of the COVID STEROID 2 trial including adults with COVID-19 and severe hypoxaemia and the Handling Oxygenation Targets in the Intensive Care Unit (HOT-ICU) trial including adult intensive care unit patients with acute hypoxaemic respiratory failure. Associations between DAWOLS and DAOOH at day 28 and 90 and long-term HRQoL (after 6 or 12 months) using the EuroQol 5-dimension 5-level survey (EQ VAS and EQ-5D-5L index values) were assessed using flexible models and evaluated using measures of fit and prediction adequacy in both datasets (comprising internal performance and external validation), non-parametric correlation coefficients and graphical presentations.

RESULTS

We found no strong associations between DAWOLS or DAOOH and HRQoL in survivors at HRQoL-follow-up (615 and 1476 patients, respectively). There was substantial variability in outcomes, and predictions from the best fitted models were poor both internally and externally in the other trial dataset, which also showed inadequate calibration. Moderate associations were found when including non-survivors, although predictions remained uncertain and calibration inadequate.

CONCLUSION

DAWOLS and DAOOH were poorly associated with HRQoL in adult survivors of severe or critical illness included in the COVID STEROID 2 and HOT-ICU trials.

Changes in Practice of Controlled Hypothermia after Cardiac Arrest in the Past 20 Years: A Critical Care Perspective

For 20 years, induced hypothermia and targeted temperature management have been recommended to mitigate brain injury and increase survival after cardiac arrest. On the basis of animal research and small clinical trials, the International Liaison Committee on Resuscitation strongly advocated hypothermia at 32-34 °C for 12-24 hours for comatose patients with out-of-hospital cardiac arrest with initial rhythm of ventricular fibrillation or nonperfusing ventricular tachycardia. The intervention was implemented worldwide. In the past decade, hypothermia and targeted temperature management have been investigated in larger clinical randomized trials focusing on target temperature depth, target temperature duration, prehospital versus in-hospital initiation, nonshockable rhythms, and in-hospital cardiac arrest. Systematic reviews suggest little or no effect of delivering the intervention on the basis of the summary of evidence, and the International Liaison Committee on Resuscitation today recommends only to treat fever and keep body temperature below 37.5 °C (weak recommendation, low-certainty evidence). Here we describe the evolution of temperature management for patients with cardiac arrest during the past 20 years and how the accrued evidence has influenced not only the recommendations but also the guideline process. We also discuss possible paths forward in this field, bringing up both whether fever management is at all beneficial for patients with cardiac arrest and which knowledge gaps future clinical trials in temperature management should address.

Observed intervention effects for mortality in randomised clinical trials: a methodological study protocol

INTRODUCTION

It is essential to choose a realistic anticipated intervention effect when calculating a sample size for a randomised clinical trial. Unfortunately, anticipated intervention effects are often inflated, when compared with the 'true' intervention effects. This is documented for mortality in critical care trials. A similar pattern might exist across different medical specialties. This study aims to estimate the range of observed intervention effects for all-cause mortality in trials included in Cochrane Reviews, within each Cochrane Review Group.

METHODS AND ANALYSIS

We will include randomised clinical trials assessing all-cause mortality as an outcome. Trials will be identified from Cochrane Reviews published in the Cochrane Database of Systematic Reviews. Cochrane Reviews will be clustered according to the registered Cochrane Review Group (eg, Anaesthesia, Emergency and Critical Care) and the statistical analyses will be conducted for each Cochrane Review Group and overall. The median relative risk and IQR for all-cause mortality and the proportion of trials with a relative all-cause mortality risk within seven different ranges will be reported (relative risk below 0.70, 0.70-0.79, 0.80-0.89, 0.90-1.09, 1.10-1.19, 1.20-1.30 and above 1.30). Subgroup analyses will explore the effects of original design, sample size, risk of bias, disease, intervention type, follow-up length, participating centres, funding type, information size and outcome hierarchy.

ETHICS AND DISSEMINATION

Since we will use summary data from trials already approved by relevant ethical committees, this study does not require ethical approval. Regardless of our findings, the results will be published in an international peer-reviewed journal.

Use of days alive without life support and similar count outcomes in randomised clinical trials - an overview and comparison of methodological choices and analysis methods

BACKGROUND

Days alive without life support (DAWOLS) and similar outcomes that seek to summarise mortality and non-mortality experiences are increasingly used in critical care research. The use of these outcomes is challenged by different definitions and non-normal outcome distributions that complicate statistical analysis decisions.

METHODS

We scrutinized the central methodological considerations when using DAWOLS and similar outcomes and provide a description and overview of the pros and cons of various statistical methods for analysis supplemented with a comparison of these methods using data from the COVID STEROID 2 randomised clinical trial. We focused on readily available regression models of increasing complexity (linear, hurdle-negative binomial, zero-one-inflated beta, and cumulative logistic regression models) that allow comparison of multiple treatment arms, adjustment for covariates and interaction terms to assess treatment effect heterogeneity.

RESULTS

In general, the simpler models adequately estimated group means despite not fitting the data well enough to mimic the input data. The more complex models better fitted and thus better replicated the input data, although this came with increased complexity and uncertainty of estimates. While the more complex models can model separate components of the outcome distributions (i.e., the probability of having zero DAWOLS), this complexity means that the specification of interpretable priors in a Bayesian setting is difficult. Finally, we present multiple examples of how these outcomes may be visualised to aid assessment and interpretation.

CONCLUSIONS

This summary of central methodological considerations when using, defining, and analysing DAWOLS and similar outcomes may help researchers choose the definition and analysis method that best fits their planned studies.

TRIAL REGISTRATION

COVID STEROID 2 trial, ClinicalTrials.gov: NCT04509973, ctri.nic.in: CTRI/2020/10/028731.

A systematic review and meta-analysis of sample size methodology for traumatic hemorrhage trials

BACKGROUND

Trauma hemorrhage remains the most common cause of preventable mortality in trauma. To guide clinical practice, RCTs provide high-quality evidence to inform clinical decision making. The clinical relevance and inferences made by RCTs are dependent on assumptions made during sample size calculation.

METHODS

To describe the quality of methodology for sample size determination, we conducted a systemic review RCTs evaluating interventions that aim to improve survival in adults with trauma-related hemorrhage. Estimated and actual outcome data are compared, including components of sample size determination.

RESULTS

A total of 13 RCTs were included. We noted a high rate of negative trial results (11 of 13 studies). Most studies were multi-center and conducted in North America, evaluating patients with blunt and penetrating injuries. The criteria for hemorrhagic shock varied across studies. All studies did not accurately estimate the mortality rate during sample size calculation. All but one study overestimated the mortality reduction during sample size calculation; the median absolute mortality reduction was 3%, compared with a target of 10%. Only the CRASH-2 study used a minimal clinically important different for treatment effect target. No RCTs employed prognostic enrichment. Most studies were terminated (8 of 13), mainly for futility.

CONCLUSION

Taken together, this review highlights that current clinical trial methodology is limited by imprecise control group risk estimates, overly optimistic treatment effect estimates, and lack of transparent justification for such targets. These limitations result in studies at high risk for futility and potentially premature abandonment of promising therapies. Given the high morbidity and mortality of trauma-related hemorrhage, we recommend that future conduct of trauma RCTs incorporate (1) prognostic enrichment to inform baseline risk, (2) justify target treatment differences based on clinical importance and realistic estimates of feasibility, and (3) be transparent and provide justification for the assumptions made.

LEVEL OF EVIDENCE

Systematic Review/Meta-Analysis; Level III.

Most Placebo-Controlled Trials in Inflammatory Bowel Disease were Underpowered Because of Overestimated Drug Efficacy Rates: Results from a Systematic Review of Induction Studies

BACKGROUND AND AIMS

Most pharmaceutical clinical trials for inflammatory bowel disease [IBD] are placebo-controlled and require effect size estimation for a drug relative to placebo. We compared expected effect sizes in sample size calculations [SSCs] to actual effect sizes in IBD clinical trials.

METHODS

MEDLINE, EMBASE, CENTRAL and the Cochrane library were searched from inception to March 26, 2021, to identify placebo-controlled induction studies for luminal Crohn's disease [CD] and ulcerative colitis [UC] that reported an SSC and a primary endpoint of clinical remission/response. Expected effects were subtracted from actual effects, and interquartile ranges [IQRs] for each corresponding median difference were calculated. Linear regression was used to assess whether placebo or drug event rate misspecifications were responsible for these differences.

RESULTS

Of eligible studies, 36.9% [55/149] were excluded because of incomplete SSC reporting, yielding 94 studies [46 CD, 48 UC]. Treatment effects were overestimated in CD for remission (-12.6% [IQR: -16.3 to -1.6%]), in UC for remission (-10.2% [IQR: -16.5 to -5.6%]) and in CD for response (-15.3% [IQR: -27.1 to -5.8%]). Differences observed were due to overestimated drug event rates, whereas expected and actual placebo event rates were similar. A meta-regression demonstrated associations between overestimated treatment effect sizes and several trial characteristics: isolated ileal disease, longer CD duration, extensive colitis [UC], single-centre, phase 2 and no endoscopic endpoint component [UC].

CONCLUSION

Overestimation of IBD therapy efficacy rates resulted in smaller-than-expected treatment effects. These results should be used to inform SSCs and trial design for IBD drug development.

Multicentre randomised trials in anaesthesia: an analysis using Bayesian metrics

Are the results of randomised trials reliable and are p values and confidence intervals the best way of quantifying efficacy? Low power is common in medical research, which reduces the probability of obtaining a 'significant result' and declaring the intervention had an effect. Metrics derived from Bayesian methods may provide an insight into trial data unavailable from p values and confidence intervals. We did a structured review of multicentre trials in anaesthesia that were published in the New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Journal of Anaesthesia and Anesthesiology between February 2011 and November 2021. We documented whether trials declared a non-zero effect by an intervention on the primary outcome. We documented the expected and observed effect sizes. We calculated a Bayes factor from the published trial data indicating the probability of the data under the null hypothesis of zero effect relative to the alternative hypothesis of a non-zero effect. We used the Bayes factor to calculate the post-test probability of zero effect for the intervention (having assumed 50% belief in zero effect before the trial). We contacted all authors to estimate the costs of running the trials. The median (IQR [range]) hypothesised and observed absolute effect sizes were 7% (3-13% [0-25%]) vs. 2% (1-7% [0-24%]), respectively. Non-zero effects were declared for 12/56 outcomes (21%). The Bayes factor favouring a zero effect relative to a non-zero effect for these 12 trials was 0.000001-1.9, with post-test zero effect probabilities for the intervention of 0.0001-65%. The other 44 trials did not declare non-zero effects, with Bayes factors favouring zero effect of 1-688, and post-test probabilities of zero effect of 53-99%. The median (IQR [range]) study costs reported by 20 corresponding authors in US$ were $1,425,669 ($514,766-$2,526,807 [$120,758-$24,763,921]). We think that inadequate power and mortality as an outcome are why few trials declared non-zero effects. Bayes factors and post-test probabilities provide a useful insight into trial results, particularly when p values approximate the significance threshold.

Fooled by Significance Testing: An Analysis of the LOVIT Vitamin C Trial

In this article, I discuss the potential pitfalls of interpreting p values, confidence intervals, and declarations of statistical significance. To illustrate the issues, I discuss the LOVIT trial, which compared high-dose vitamin C with placebo in mechanically ventilated patients with sepsis. The primary outcome - the proportion of patients who died or had persisting organ dysfunction at day 28 - was significantly higher in patients who received vitamin C (p = .01). The authors had hypothesized that vitamin C would have a beneficial effect, although the prior evidence for benefit was weak. There was no prior evidence for a harmful effect of high-dose vitamin C. Consequently, the pretest probability for harm was low. The sample size was calculated assuming a 10% absolute risk difference, which was optimistic. Overestimating the effect size when calculating the sample size leads to low power. For these reasons, we should be skeptical that vitamin C causes harm in septic patients, despite the significant result. p-values and confidence intervals are probabilities concerning the chance of obtaining the observed data. However, we are more interested in the chance the intervention has a real effect on the outcome. That is to say, we are more interested in whether the hypothesis is true. A Bayesian approach allows us to estimate the false positive risk, which is the post-test probability there is no effect of the intervention. The false positive risk for the LOVIT trial (calculated from the published summary data using uniform priors for the parameter values) is 70%. Most likely, high-dose vitamin C does not cause harm in septic patients. Most likely it has no effect at all. If there is an effect, it is probably small and most likely beneficial.

Causal inference for planning randomised critical care trials: Protocol for a scoping review

BACKGROUND

Randomised clinical trials in critical care are prone to inconclusiveness owing, in part, to undue optimism about effect sizes and suboptimal accounting for heterogeneous treatment effects. Planned predictive enrichment based on secondary critical care data (often very rich with respect to both data types and temporal granularity) and causal inference methods may help overcome these challenges, but no overview exists about their use to this end.

METHODS

We will conduct a scoping review to assess the extent and nature of the use of causal inference from secondary data for planned predictive enrichment of randomised clinical trials in critical care. We will systematically search 10 general and specialty journals for reports published on or after 1 January 2018, of randomised clinical trials enrolling adult critically ill patients. We will collect trial metadata (e.g., recruitment period and phase) and, when available, information pertaining to the focus of the review (predictive enrichment based on causal inference estimates from secondary data): causal inference methods, estimation techniques and software used; types of patient populations; data provenance, types and models; and the availability of the data (public or not). The results will be reported in a descriptive manner.

DISCUSSION

The outlined scoping review aims to assess the use of causal inference methods and secondary data for planned predictive enrichment in randomised critical care trials. This will help guide methodological improvements to increase the utility, and facilitate the use, of causal inference estimates when planning such trials in the future.

The Effect of Intensive Care Unit Diaries on Posttraumatic Stress Disorder, Anxiety, and Depression: A Systematic Review and Meta-analysis of Randomized Controlled Trials

INTRODUCTION

Patients discharged from the intensive care unit (ICU) suffer from long-term symptoms affecting the physical, psychological, and cognitive well-being and cannot understand memories and dreams. Intensive care unit diaries describe daily events about the patient and may allow them to reconstruct their experience.

OBJECTIVE

We conducted a systematic review and meta-analysis of randomized controlled trials (RCTs) to investigate the effects of ICU diaries on posttraumatic stress disorder (PTSD), depression, and anxiety.

METHODS

Five electronic databases were searched up to May 6, 2022. We included RCTs comparing patients admitted to the ICU who received a diary to those who did not receive a diary. The primary outcome was the rate of PTSD. Secondary outcomes were rates of depression and anxiety.

RESULTS

We included 7 RCTs. Patients who received a diary during the ICU admission had reduced rate of PTSD (78/432 [18%] vs 106/422 [25%]; risk ratio [RR], 0.73; 95% confidence interval [CI], 0.57-0.94; P = .02; I2 = 0%; trial sequential analysis-adjusted CI, 0.55-0.97) when compared with patients who did not receive a diary. We found a non-statistically significant difference toward a reduction in the rate of depression (38/232 [16%] vs 54/224 [24%]; RR, 0.70; 95% CI, 0.49-1.01; P = .06; I2 = 0%) and anxiety (63/232 [27%] vs 70/224 [31%]; RR, 0.64; 95% CI, 0.29-1.40; P = .26; I2 = 67%).

CONCLUSIONS

Providing an ICU diary to patients admitted to the ICU reduced the rate of PTSD symptoms compared with usual care.

Trials with 'non-significant' results are not insignificant trials: a common significance threshold distorts reporting and interpretation of trial results

We discuss a newly published study examining how phrases are used in clinical trials to describe results when the estimated P-value is close to (slightly above or slightly below) 0.05, which has been arbitrarily designated by convention as the boundary for 'statistical significance'. Terms such as 'marginally significant', 'trending towards significant', and 'nominally significant' are well represented in biomedical literature, but are not actually scientifically meaningful. Acknowledging that 'statistical significance' remains a major determinant of publication, we propose that scientific journals de-emphasise the use of P-values for null hypothesis significance testing, a purpose for which they were never intended, and avoid the use of these ambiguous and confusing terms in scientific articles. Instead, investigators could simply report their findings: effect sizes, P-values, and confidence intervals (or their Bayesian equivalents), and leave it to the discerning reader to infer the clinical applicability and importance. Our goal should be to move away from describing studies (or trials) as positive or negative based on an arbitrary P-value threshold, and rather to judge whether the scientific evidence provided is informative or uninformative.

The Impact of Sample Size Misestimations on the Interpretation of ARDS Trials: Systematic Review and Meta-analysis

BACKGROUND

Indeterminate randomized controlled trials (RCTs) in ARDS may arise from sample size misspecification, leading to abandonment of efficacious therapies.

RESEARCH QUESTIONS

If evidence exists for sample size misspecification in ARDS RCTs, has this led to rejection of potentially beneficial therapies? Does evidence exist for prognostic enrichment in RCTs using mortality as a primary outcome?

STUDY DESIGN AND METHODS

We identified 150 ARDS RCTs commencing recruitment after the 1994 American European Consensus Conference ARDS definition and published before October 31, 2020. We examined predicted-observed sample size, predicted-observed control event rate (CER), predicted-observed average treatment effect (ATE), and the relationship between observed CER and observed ATE for RCTs with mortality and nonmortality primary outcome measures. To quantify the strength of evidence, we used Bayesian-averaged meta-analysis, trial sequential analysis, and Bayes factors.

RESULTS

Only 84 of 150 RCTs (56.0%) reported sample size estimations. In RCTs with mortality as the primary outcome, CER was overestimated in 16 of 28 RCTs (57.1%). To achieve predicted ATE, interventions needed to prevent 40.8% of all deaths, compared with the original prediction of 29.3%. Absolute reduction in mortality ≥ 10% was observed in 5 of 28 RCTs (17.9%), but predicted in 21 of 28 RCTs (75%). For RCTs with mortality as the primary outcome, no association was found between observed CER and observed ATE (pooled OR: β = -0.04; 95% credible interval, -0.18 to 0.09). We identified three interventions that are not currently standard of care with a Bayesian-averaged effect size of > 0.20 and moderate strength of existing evidence: corticosteroids, airway pressure release ventilation, and noninvasive ventilation.

INTERPRETATION

Reporting of sample size estimations was inconsistent in ARDS RCTs, and misspecification of CER and ATE was common. Prognostic enrichment strategies in ARDS RCTs based on all-cause mortality are unlikely to be successful. Bayesian methods can be used to prioritize interventions for future effectiveness RCTs.

Critical Care Randomized Trials Demonstrate Power Failure: A Low Positive Predictive Value of Findings in the Critical Care Research Field

BACKGROUND

We aimed to determine the post-hoc power of randomized controlled trials (RCTs) in critical care, and describe the implications for long-term positive (PPV) and negative predictive value (NPV) of statistically significant and non-significant findings respectively in the research field.

METHODS

We reviewed three cohorts of RCTs. "Adult-RCTs" were 216 multicenter RCTs with a mortality outcome from a published systematic review. "Pediatric-RCTs" were 120 RCTs with a mortality outcome, obtained by search of picutrials.net. "Consecutive-RCTs" were 90 recent RCTs obtained by screening publications in 6 journals. Post-hoc power for each study was calculated at α 0.05 and 0.005, for measures of small, medium, and large effect-size, using G*Power software. Long-run expected PPV and NPV of critical care research field findings were then calculated.

RESULTS

With α 0.05, post-hoc power for small effect-size was very low in all RCT-cohorts (eg, median 24% in Adult-RCTs). For medium effect-size, post-hoc power was low, except for Adult-RCTs (eg, median 9% in Pediatric-RCTs). For large effect-size, post-hoc power for non-human-animal Consecutive-RCTs was low (median 32%). With α 0.005, post-hoc power was even lower. The corollary was that both PPV and NPV were poor for small effect-size, unless α 0.005 was used. Even with α 0.005, with realistic (vs. optimistic) prior probability of the alternative hypothesis, the PPV was low (eg, in Adult-RCTs 57.1% vs. 92.3%). Adding mild bias (0.1) reduced the PPV even further. For medium effect-size both PPV and NPV were better; nevertheless, with α 0.05 and realistic prior probability of the alternative hypothesis the PPV was poor, and with α 0.005 and mild bias (0.1) the PPV was very low (eg, Adult-RCTs median 44.1%).

CONCLUSIONS

To improve the predictive value of findings in the critical care research field, RCTs should be designed to have 80% power for realistic effect-size at α 0.005.

Health-related quality of life and days alive without life support or out of hospital: Protocol

BACKGROUND

Mortality is often the primary outcome in randomised clinical trials (RCTs) conducted in critically ill patients. Due to increased awareness on survivors after critical illness and outcomes other than mortality, health-related quality of life (HRQoL) and days alive without life support (DAWOLS) or days alive and out of hospital (DAAOOH) are increasingly being used. DAWOLS and DAAOOH convey more information than mortality, are easier to collect than HRQoL, and are usually assessed at earlier time points, which may be preferable in some situations. However, the associations between DAWOLS-DAAOOH and HRQoL are uncertain.

METHODS

We will assess associations between DAWOLS-DAAOOH at day 28 and 90 (independent variables/predictors) and HRQoL assessed using the EuroQol EQ-5D-5L questionnaire (EQ-VAS and EQ-5D-5L index values) at 6 or 12 months (dependent variables) in two RCTs: the COVID STEROID 2 RCT conducted in adult patients with COVID-19 and severe hypoxaemia and the Handling Oxygenation Targets in the Intensive Care Unit (HOT-ICU) RCT conducted in adult intensive care patients with acute hypoxaemic respiratory failure. We will describe associations using best-fitting fractional polynomial transformations separately in each dataset, with the resulting models presented and assessed in both datasets graphically and using measures of fit and prediction adequacy (i.e., internal performance and external validation). We will use multiple imputation if missingness exceeds 5%.

DISCUSSION

The outlined study will provide important knowledge on the associations between DAWOLS-DAAOOH and HRQoL in adult critically ill patients, which may help researchers and clinical trialists prioritise and select outcomes in future RCTs conducted in this population.

A Bayesian analysis of mortality outcomes in multicentre clinical trials in critical care

BACKGROUND

Multicentre RCTs are widely used by critical care researchers to answer important clinical questions. However, few trials evaluating mortality outcomes report statistically significant results. We hypothesised that the low proportion of trials reporting statistically significant differences for mortality outcomes is plausibly explained by lower-than-expected effect sizes combined with a low proportion of participants who could realistically benefit from studied interventions.

METHODS

We reviewed multicentre trials in critical care published over a 10-yr period in the New England Journal of Medicine, the Journal of the American Medical Association, and the Lancet. To test our hypothesis, we analysed the results using a Bayesian model to investigate the relationship between the proportion of effective interventions and the proportion of statistically significant results for prior distributions of effect size and trial participant susceptibility.

RESULTS

Five of 54 trials (9.3%) reported a significant difference in mortality between the control and the intervention groups. The median expected and observed differences in absolute mortality were 8.0% and 2.0%, respectively. Our modelling shows that, across trials, a lower-than-expected effect size combined with a low proportion of potentially susceptible participants is consistent with the observed proportion of trials reporting significant differences even when most interventions are effective.

CONCLUSIONS

When designing clinical trials, researchers most likely overestimate true population effect sizes for critical care interventions. Bayesian modelling demonstrates that that it is not necessarily the case that most studied interventions lack efficacy. In fact, it is plausible that many studied interventions have clinically important effects that are missed.

Powering Bias and Clinically Important Treatment Effects in Randomized Trials of Critical Illness

OBJECTIVES

Recurring issues in clinical trial design may bias results toward the null, yielding findings inconclusive for treatment effects. This study evaluated for powering bias among high-impact critical care trials and the associated risk of masking clinically important treatment effects.

DESIGN, SETTING, AND PATIENTS

Secondary analysis of multicenter randomized trials of critically ill adults in which mortality was the main endpoint. Trials were eligible for inclusion if published between 2008 and 2018 in leading journals. Analyses evaluated for accuracy of estimated control group mortality, adaptive sample size strategy, plausibility of predicted treatment effect, and results relative to the minimal clinically important difference. The main outcome was the mortality risk difference at the study-specific follow-up interval.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Of 101 included trials, 12 met statistical significance for their main endpoint, five for increased intervention-associated mortality. Most trials (77.3%) overestimated control group mortality in power calculations (observed minus predicted difference, -6.7% ± 9.8%; p < 0.01). Due to this misestimation of control group mortality, in 14 trials, the intervention would have had to prevent at least half of all deaths to achieve the hypothesized treatment effect. Seven trials prespecified adaptive sample size strategies that might have mitigated this issue. The observed risk difference for mortality fell within 5% of predicted in 20 trials, of which 16 did not reach statistical significance. Half of trials (47.0%) were powered for an absolute risk reduction greater than or equal to 10%, but this effect size was observed in only three trials with a statistically significant treatment benefit. Most trials (67.3%) could not exclude clinically important treatment benefit or harm.

CONCLUSIONS

The design of most high-impact critical care trials biased results toward the null by overestimating control group mortality and powering for unrealistic treatment effects. Clinically important treatment effects often cannot be excluded.

Are treatment effect assumptions in orthodontic studies overoptimistic?

Background

At the clinical trial design stage, assumptions regarding the treatment effects to be detected should be appropriate so that the required sample size can be calculated. There is evidence in the medical literature that sample size assumption can be overoptimistic. The aim of this study was to compare the distribution of the assumed effects versus that of the observed effects as a proxy for overoptimistic treatment effect assumptions at the study design stage.

Materials and method

Systematic reviews (SRs) published between 1 January 2010 and 31 December 2019 containing at least one meta-analysis on continuous outcomes were identified electronically. SR and primary study level characteristics were extracted from the SRs and the individual trials. Details on the sample size calculation process and assumptions and the observed treatment effects were extracted.

Results

Eighty-five SRs with meta-analysis containing 347 primary trials were included. The median number of SR authors was 5 (interquartile range: 4–7). At the primary study level, the majority were single centre (78.1%), utilized a parallel design (52%), and rated as an unclear/moderate level of risk of bias (34.3%). A sample size was described in only 31.7% (110/347) of studies. From this cohort of 110 studies, in only 37 studies was the assumed clinical difference that the study was designed to detect reported (37/110). The assumed treatment effect was recalculated for the remaining 73 studies (73/110). The one-sided exact signed rank test showed a significant difference between the assumed and observed treatment effects (P < 0.001) suggesting greater values for the assumed effect sizes.

Conclusions

Careful consideration of the assumptions at the design stage of orthodontic studies are necessary in order to reduce the unreliability of clinical study results and research waste.

Between-trial heterogeneity in ARDS research

Purpose

Most randomized controlled trials (RCTs) in patients with acute respiratory distress syndrome (ARDS) revealed indeterminate or conflicting study results. We aimed to systematically evaluate between-trial heterogeneity in reporting standards and trial outcome.

Methods

A systematic review of RCTs published between 2000 and 2019 was performed including adult ARDS patients receiving lung-protective ventilation. A random-effects meta-regression model was applied to quantify heterogeneity (non-random variability) and to evaluate trial and patient characteristics as sources of heterogeneity.

Results

In total, 67 RCTs were included. The 28-day control-group mortality rate ranged from 10 to 67% with large non-random heterogeneity (I² = 88%, p < 0.0001). Reported baseline patient characteristics explained some of the outcome heterogeneity, but only six trials (9%) reported all four independently predictive variables (mean age, mean lung injury score, mean plateau pressure and mean arterial pH). The 28-day control group mortality adjusted for patient characteristics (i.e. the residual heterogeneity) ranged from 18 to 45%. Trials with significant benefit in the primary outcome reported a higher control group mortality than trials with an indeterminate outcome or harm (mean 28-day control group mortality: 44% vs. 28%; p = 0.001).

Conclusion

Among ARDS RCTs in the lung-protective ventilation era, there was large variability in the description of baseline characteristics and significant unexplainable heterogeneity in 28-day control group mortality. These findings signify problems with the generalizability of ARDS research and underline the urgent need for standardized reporting of trial and baseline characteristics.

Supplementary Information

The online version of this article (10.1007/s00134-021-06370-w) contains supplementary material, which is available to authorized users.

Esmolol in Cardiac Surgery: A Randomized Controlled Trial

OBJECTIVE

To assess whether the administration of the ultra-short-acting β-blocker esmolol in cardiac surgery could have a cardioprotective effect that translates into improved postoperative outcomes.

DESIGN

Single-center, double-blinded, parallel-group randomized controlled trial.

SETTING

A tertiary care referral center.

PARTICIPANTS

Patients undergoing elective cardiac surgery with preoperative evidence of left ventricular end-diastolic diameter >60 mm and/or left ventricular ejection fraction <50%.

INTERVENTIONS

Patients were assigned randomly to receive either esmolol (1 mg/kg as a bolus before aortic cross-clamping and 2 mg/kg mixed in the cardioplegia solution) or placebo in a 1:1 allocation ratio.

MEASUREMENTS AND MAIN RESULTS

The primary composite endpoint of prolonged intensive care unit stay and/or in-hospital mortality occurred in 36/98 patients (36%) in the placebo group versus 27/102 patients (27%) in the esmolol group (p = 0.13). In the esmolol group, a reduction in the maximum inotropic score during the first 24 postoperative hours was observed (10 [interquartile range 5-15] v 7 [interquartile range 5-10.5]; p = 0.04), as well as a trend toward a reduction in postoperative low-cardiac-output syndrome (13/98 v 6/102; p = 0.08) and the rate of hospital admission at one year (26/95 v 16/96; p = 0.08). A trend toward an increase in the number of patients with ejection fraction ≥60% at hospital discharge also was observed (4/95 v 11/92; p = 0.06).

CONCLUSIONS

In the present trial, esmolol as a cardioplegia adjuvant enhanced postoperative cardiac performance but did not reduce a composite endpoint of prolonged intensive care unit stay and/or mortality.

Clinical trials in critical care: can a Bayesian approach enhance clinical and scientific decision making?

Recent Bayesian reanalyses of prominent trials in critical illness have generated controversy by contradicting the initial conclusions based on conventional frequentist analyses. Many clinicians might be sceptical that Bayesian analysis, a philosophical and statistical approach that combines prior beliefs with data to generate probabilities, provides more useful information about clinical trials than the frequentist approach. In this Personal View, we introduce clinicians to the rationale, process, and interpretation of Bayesian analysis through a systematic review and reanalysis of interventional trials in critical illness. In the majority of cases, Bayesian and frequentist analyses agreed. In the remainder, Bayesian analysis identified interventions where benefit was probable despite the absence of statistical significance, where interpretation depended substantially on choice of prior distribution, and where benefit was improbable despite statistical significance. Bayesian analysis in critical care medicine can help to distinguish harm from uncertainty and establish the probability of clinically important benefit for clinicians, policy makers, and patients.

Metabolomic basis for response to high dose vitamin D in critical illness

BACKGROUND & AIMS

It is unclear if intervention can mitigate the dramatic alterations of metabolic homeostasis present in critical illness. Our objective was to determine the associations between increased 25-hydroxyvitamin D levels following high dose vitamin D₃ and more favorable metabolomic profiles in critical illness.

METHODS

We performed a post-hoc metabolomics study of the VITdAL-ICU randomized double-blind, placebo-controlled trial. Trial patients from Medical and Surgical Intensive Care Units at a tertiary university hospital with 25-hydroxyvitamin D level ≤20 ng/mL received either high dose oral vitamin D₃ (540,000 IU) or placebo. We performed an analysis of 578 metabolites from 1215 plasma samples from 428 subjects at randomization (day 0), day 3 and 7. Using mixed-effects modeling, we studied changes in metabolite profiles in subjects receiving intervention or placebo relative to absolute increases in 25-hydroxyvitamin D levels from day 0 to day 3.

RESULTS

55.2% of subjects randomized to high dose vitamin D₃ demonstrated an absolute increase in 25-hydroxyvitamin D ≥ 15 ng/ml from day 0 to day 3. With an absolute increase in 25-hydroxyvitamin D ≥ 15 ng/ml, multiple members of the sphingomyelin, plasmalogen, lysoplasmalogen and lysophospholipid metabolite classes had significantly positive Bonferroni corrected associations over time. Further, multiple representatives of the acylcarnitine and phosphatidylethanolamine metabolite classes had significantly negative Bonferroni corrected associations over time with an absolute increase in 25-hydroxyvitamin D ≥ 15 ng/ml. Changes in these highlighted metabolite classes were associated with decreased 28-day mortality.

CONCLUSIONS

Increases in 25-hydroxyvitamin D following vitamin D₃ intervention are associated with favorable changes in metabolites involved in endothelial protection, enhanced innate immunity and improved mitochondrial function.

Lactated Ringer's Versus 4% Albumin on Lactated Ringer's in Early Sepsis Therapy in Cancer Patients: A Pilot Single-Center Randomized Trial

OBJECTIVE

To investigate the effects of the administration of 4% albumin on lactated Ringer's, when compared with lactated Ringer's alone, in the early phase of sepsis in cancer patients.

DESIGN

Single-center, randomized, double-blind, controlled-parallel trial.

SETTING

A tertiary care university cancer hospital.

PATIENTS

Cancer patients with severe sepsis or septic shock.

INTERVENTIONS

Between October 2014 and December 2016, patients were randomly assigned to receive either bolus of albumin in a lactated Ringer's solution or lactated Ringer's solution alone during the first 6 hours of fluid resuscitation after intensive care medicine (ICU) admission. Primary outcome was defined as death from any cause at 7 days. Secondary outcomes were defined as death from any cause within 28 days, change in Sequence Organ Failure Assessment scores from baseline to day 7, days alive and free of mechanical ventilation, days alive and free of vasopressor, renal replacement therapy during ICU stay, and length of ICU and hospital stay.

MEASUREMENTS AND MAIN RESULTS

A total of 360 patients were enrolled in the trial. At 7 days, 46 of 180 patients (26%) died in the albumin group and 40 of 180 (22%) died in the lactated Ringer's group (p = 0.5). At 28 days, 96 of 180 patients (53%) died in the albumin group and 83 of 180 (46%) died in the lactated Ringer's group (p = 0.2). No significant differences in secondary outcomes were observed.

CONCLUSIONS

Adding albumin to early standard resuscitation with lactated Ringer's in cancer patients with sepsis did not improve 7-day survival.

Typology of Published Randomized Controlled Trials Investigating Initial Ventilation Strategy in Critically Ill Patients With Acute Respiratory Failure: A Methodologic Review

BACKGROUND

Randomized controlled trials (RCTs) in patients with hypoxemic acute respiratory failure (ARF) often failed to show survival benefits and resulted in varying clinical end points.

RESEARCH QUESTION

This methodologic review was conducted of published RCTs on ARF, with a careful attention to whether the study results were positive or negative.

STUDY DESIGN AND METHODS

MEDLINE, Cochrane Central Register of Controlled Trials, and Web of Science were searched for published RCTs in adult patients with ARF between January 1995 and December 2019. The objective was to investigate sources of heterogeneity and factors associated with a positive RCT (ie, with a significant difference on the primary end point). To determine the importance of the primary end point choice, end points were classified as follows: (1) respiratory event end points (corresponding to modification of the ventilation support); (2) physiologic/clinical end points (corresponding to oxygenation or clinical parameters); and (3) mortality. The Cochrane risk-of-bias tool was used to assess study quality.

RESULTS

Seventy-four RCTs were included (57% were single-center RCTs) comparing mainly oxygenation/ventilation strategies (95%) in patients with various ARF causes (62%); studies were stopped prematurely in 20% of the trials. A standardized management of ARF was observed in 24 (32%) trials. Twenty-two distinct primary end points have been used, the first of which were those based on respiratory events (44 RCTs [49%]), namely intubation in 76% of the cases. Physiologic/clinical end points have been used in 21 trials (29%) and mortality in nine (12%). Overall, 42 (57%) RCTs were positive, 52% in studies with respiratory event end points, 76% in studies with physiologic/clinical end points, and 33% in studies with mortality end points. Adjusted for study quality (Cochrane risk-of-bias tool), factors associated with a positive RCT included clinically based primary end points (OR, 8.40; 95% CI, 1.35-65.79), the use of standardized ARF management (OR, 4.55; 95% CI, 1.02-22.88), and single-center trials (OR, 3.85; 95% CI, 1.25-13.11).

INTERPRETATION

The typology of published RCTs in patients with ARF could be used to frame future trial designs in this field and guide clinicians and researchers toward optimal research transfer to the bedside.

Contemporary strategies to improve clinical trial design for critical care research: insights from the First Critical Care Clinical Trialists Workshop

BACKGROUND

Conducting research in critically-ill patient populations is challenging, and most randomized trials of critically-ill patients have not achieved pre-specified statistical thresholds to conclude that the intervention being investigated was beneficial.

METHODS

In 2019, a diverse group of patient representatives, regulators from the USA and European Union, federal grant managers, industry representatives, clinical trialists, epidemiologists, and clinicians convened the First Critical Care Clinical Trialists (3CT) Workshop to discuss challenges and opportunities in conducting and assessing critical care trials. Herein, we present the advantages and disadvantages of available methodologies for clinical trial design, conduct, and analysis, and a series of recommendations to potentially improve future trials in critical care.

CONCLUSION

The 3CT Workshop participants identified opportunities to improve critical care trials using strategies to optimize sample size calculations, account for patient and disease heterogeneity, increase the efficiency of trial conduct, maximize the use of trial data, and to refine and standardize the collection of patient-centered and patient-informed outcome measures beyond mortality.

Measuring and Analyzing Length of Stay in Critical Care Trials

BACKGROUND

In randomized clinical trials among critically ill patients, it is uncertain how choices regarding the measurement and analysis of nonmortal outcomes measured in terms of duration, such as intensive care unit (ICU) length of stay (LOS), affect studies' conclusions.

OBJECTIVES

Assess the definitions and analytic methods used for ICU LOS analyses in published randomized clinical trials.

RESEARCH DESIGN

This is a systematic review and statistical simulation study.

RESULTS

Among the 80 of 150 trials providing sufficient information regarding the chosen definition of ICU LOS, 3 different start times (ICU admission, trial enrollment/randomization, receipt of intervention) and 2 end times (discharge readiness, actual discharge) were used. In roughly three quarters of these studies, ICU LOS was compared using approaches that did not explicitly account for death, either by ignoring it entirely or stratifying the analyses by survival status. The remaining studies used time-to-event (discharge) models censoring at death or applied a fixed LOS value to patients who died. In statistical simulations, we showed that each analytic approach tested a different question regarding ICU LOS, and that approaches that do not explicitly account for death often produce misleading or ambiguous conclusions when treatments produce small effects on mortality, even if those are not detected as significant in the trial.

CONCLUSIONS

There is considerable variability in how ICU LOS is measured and analyzed which impairs the ability to compare results across trials and can produce spurious conclusions. Analyses of duration-based outcomes such as LOS should jointly assess the impact of the intervention on mortality to yield correct interpretations.

Barriers to clinical research in trauma

As with all areas of medicine, high-quality clinical research is essential to improving the care of trauma patients. This research is crucial in developing evidence-based treatments that decrease cost, decrease morbidity, and improve mortality. Trauma continues to extract a significant toll on society and is the single largest cause of years of life lost in the United States. The need to conduct high-quality clinical research in trauma is not disputed. However, significant challenges and barriers unique to the field of trauma make performing this research more difficult. It is critical to be aware of these challenges and barriers to performing clinical research involving trauma patients so these challenges can be accounted for and solutions implemented to minimize their impact on research. This review will focus on the barriers and challenges that are encountered while performing clinical research in trauma.

Sample size re-estimation in a superiority clinical trial using a hybrid classical and Bayesian procedure

When designing studies involving a continuous endpoint, the hypothesized difference in means ( θA ) and the assumed variability of the endpoint ( σ2 ) play an important role in sample size and power calculations. Traditional methods of sample size re-estimation often update one or both of these parameters using statistics observed from an internal pilot study. However, the uncertainty in these estimates is rarely addressed. We propose a hybrid classical and Bayesian method to formally integrate prior beliefs about the study parameters and the results observed from an internal pilot study into the sample size re-estimation of a two-stage study design. The proposed method is based on a measure of power called conditional expected power (CEP), which averages the traditional power curve using the prior distributions of θ and σ2 as the averaging weight, conditional on the presence of a positive treatment effect. The proposed sample size re-estimation procedure finds the second stage per-group sample size necessary to achieve the desired level of conditional expected interim power, an updated CEP calculation that conditions on the observed first-stage results. The CEP re-estimation method retains the assumption that the parameters are not known with certainty at an interim point in the trial. Notional scenarios are evaluated to compare the behavior of the proposed method of sample size re-estimation to three traditional methods.

Reproducibility of clinical research in critical care: a scoping review

BACKGROUND

The ability to reproduce experiments is a defining principle of science. Reproducibility of clinical research has received relatively little scientific attention. However, it is important as it may inform clinical practice, research agendas, and the design of future studies.

METHODS

We used scoping review methods to examine reproducibility within a cohort of randomized trials examining clinical critical care research and published in the top general medical and critical care journals. To identify relevant clinical practices, we searched the New England Journal of Medicine, The Lancet, and JAMA for randomized trials published up to April 2016. To identify a comprehensive set of studies for these practices, included articles informed secondary searches within other high-impact medical and specialty journals. We included late-phase randomized controlled trials examining therapeutic clinical practices in adults admitted to general medical-surgical or specialty intensive care units (ICUs). Included articles were classified using a reproducibility framework. An original study was the first to evaluate a clinical practice. A reproduction attempt re-evaluated that practice in a new set of participants.

RESULTS

Overall, 158 practices were examined in 275 included articles. A reproduction attempt was identified for 66 practices (42%, 95% CI 33-50%). Original studies reported larger effects than reproduction attempts (primary endpoint, risk difference 16.0%, 95% CI 11.6-20.5% vs. 8.4%, 95% CI 6.0-10.8%, P = 0.003). More than half of clinical practices with a reproduction attempt demonstrated effects that were inconsistent with the original study (56%, 95% CI 42-68%), among which a large number were reported to be efficacious in the original study and to lack efficacy in the reproduction attempt (34%, 95% CI 19-52%). Two practices reported to be efficacious in the original study were found to be harmful in the reproduction attempt.

CONCLUSIONS

A minority of critical care practices with research published in high-profile journals were evaluated for reproducibility; less than half had reproducible effects.

Empirical Consequences of Current Recommendations for the Design and Interpretation of Noninferiority Trials

BACKGROUND

Noninferiority trials are increasingly common, though they have less standardized designs and their interpretation is less familiar to clinicians than superiority trials.

OBJECTIVE

To empirically evaluate a cohort of noninferiority trials to determine 1) their interpretation as recommended by CONSORT, 2) choice of alpha threshold and its sidedness, and 3) differences between methods of analysis such as intention-to-treat and per-protocol.

DESIGN

We searched MEDLINE for parallel-group randomized controlled noninferiority trials published in the five highest-impact general medical journals between 2011 and 2016.

MAIN MEASURES

Data abstracted included trial design parameters, results, and interpretation of results based on CONSORT recommendations.

KEY RESULTS

One hundred sixty-three trials and 182 noninferiority comparisons were included in our analysis. Based on CONSORT-recommended interpretation, 79% of experimental therapies met criteria for noninferiority, 13% met criteria for superiority, 20% were declared inconclusive, and 2% met criteria for inferiority. However, for 12% of trials, the experimental therapy was statistically significantly worse than the active control, but CONSORT recommended an interpretation of inconclusive or noninferior. A two-sided alpha equivalent of greater than 0.05 was used in 34% of the trials, and in five of these trials, the use of a standard two-sided alpha of 0.05 led to changes in the interpretation of results that disfavored the experimental therapy. In four of the five comparisons where different methods of analysis (e.g., intention-to-treat and per-protocol) yielded different results, the intention-to-treat analysis was the more conservative. In 11% of trials, a secondary advantage of the new therapy was neither reported nor could it be inferred by reviewers.

CONCLUSIONS

In this cohort, the design and interpretation of noninferiority trials led to significant and systematic bias in favor of the experimental therapy. Clinicians should exercise caution when interpreting these trials. Future trials may be more reliable if design parameters are standardized.

Measurement Error Due to Patient Flow in Estimates of Intensive Care Unit Length of Stay

Clinical endpoints measured in terms of duration, such as intensive care unit (ICU) length of stay (LOS), are widely used in randomized clinical trials (RCTs) and observational research. In analyses of patient-level data from a recent RCT, in which ICU LOS was the primary endpoint, and in administrative data, we showed that additional ICU time is often accrued by patients after they are deemed ready for discharge. This "immutable" time (which cannot plausibly be altered by interventions under study) varies by day, week, and year, adding on average one-third of a day to total LOS. We then used statistical simulations, informed by the administrative data and RCT, to assess the impact of immutable time on the measurement and statistical comparison of patients' ICU LOS. These simulations demonstrated that immutable time combines with clinically necessary ICU time (neither of which is likely to be normally distributed) to produce overall LOS distributions that might either mask true treatment effects or suggest false treatment effects relative to analyses of time to discharge readiness. The extent and direction of bias were complex functions of the statistical method used, mortality rates and distributions, and the magnitude of immutable time relative to intervention-associated reductions in LOS.

Prehospital cooling to improve successful targeted temperature management after cardiac arrest: A randomized controlled trial

RATIONALE

Targeted temperature management (TTM) improves survival with good neurological outcome after out-of-hospital cardiac arrest (OHCA), but is delivered inconsistently and often with delay.

OBJECTIVE

To determine if prehospital cooling by paramedics leads to higher rates of 'successful TTM', defined as achieving a target temperature of 32-34°C within 6h of hospital arrival.

METHODS

Pragmatic RCT comparing prehospital cooling (surface ice packs, cold saline infusion, wristband reminders) initiated 5min after return of spontaneous circulation (ROSC) versus usual resuscitation and transport. The primary outcome was rate of 'successful TTM'; secondary outcomes were rates of applying TTM in hospital, survival with good neurological outcome, pulmonary edema in emergency department, and re-arrest during transport.

RESULTS

585 patients were randomized to receive prehospital cooling (n=279) or control (n=306). Prehospital cooling did not increase rates of 'successful TTM' (30% vs 25%; RR, 1.17; 95% confidence interval [CI] 0.91-1.52; p=0.22), but increased rates of applying TTM in hospital (68% vs 56%; RR, 1.21; 95%CI 1.07-1.37; p=0.003). Survival with good neurological outcome (29% vs 26%; RR, 1.13, 95%CI 0.87-1.47; p=0.37) was similar. Prehospital cooling was not associated with re-arrest during transport (7.5% vs 8.2%; RR, 0.94; 95%CI 0.54-1.63; p=0.83) but was associated with decreased incidence of pulmonary edema in emergency department (12% vs 18%; RR, 0.66; 95%CI 0.44-0.99; p=0.04).

CONCLUSIONS

Prehospital cooling initiated 5min after ROSC did not increase rates of achieving a target temperature of 32-34°C within 6h of hospital arrival but was safe and increased application of TTM in hospital.

Clinical research in critical care. Difficulties and perspectives

In the field of Intensive Care Medicine, improved survival has resulted from better patient care, the early detection of clinical deterioration, and the prevention of iatrogenic complications, while research on new treatments has been followed by an overwhelming number of disappointments. The origins of these fiascos must be sought in the conjunction of methodological problems - common to other disciplines - and the particularities of critically ill patients. The present article discusses both aspects and suggests some options for progress.

Effect sizes in ongoing randomized controlled critical care trials

Background

An important limitation of many critical care trial designs is that they hypothesize large, and potentially implausible, reductions in mortality. Interpretation of trial results could be improved by systematic assessment of the plausibility of trial hypotheses; however, such assessment has not been attempted in the field of critical care medicine. The purpose of this study was to determine clinicians’ views about prior probabilities and plausible effect sizes for ongoing critical care trials where the primary endpoint is landmark mortality.

Methods

We conducted a systematic review of clinical trial registries in September 2015 to identify ongoing critical care medicine trials where landmark mortality was the primary outcome, followed by a clinician survey to obtain opinions about ten large trials. Clinicians were asked to estimate the probability that each trial would demonstrate a mortality effect equal to or larger than that used in its sample size calculations.

Results

Estimates provided by individual clinicians varied from 0% to 100% for most trials, with a median estimate of 15% (IQR 10–20%). The median largest absolute mortality reduction considered plausible was 4.5% (IQR 3.5–5%), compared with a median absolute mortality reduction used in sample size calculations of 5% (IQR 3.6–10%) (P = 0.27).

Conclusions

For some of the largest ongoing critical care trials, many clinicians regard prior probabilities as low and consider that plausible effects on absolute mortality are less than 5%. Further work is needed to determine whether pooled estimates obtained by surveying clinicians are replicable and accurate or whether other methods of estimating prior probability are preferred.

Electronic supplementary material

The online version of this article (doi:10.1186/s13054-017-1726-x) contains supplementary material, which is available to authorized users.

Interventions affecting mortality in critically ill and perioperative patients: A systematic review of contemporary trials

PURPOSE

Confounders in randomized controlled trials (RCTs) reporting significant effects on mortality in critically ill patients using non-surgical techniques have not been systematically explored. We aimed to identify factors unrelated to the reported intervention that might have affected the findings and robustness of such trials.

METHODS

We searched Pubmed/MEDLINE for all RCTs on any non-surgical interventions reporting an effect on unadjusted mortality in critically ill patients between 1/1/2000 and 1/12/2015. We assessed: the number needed to treat/harm (NNT or NNH), sample size, trial design (blinded/unblinded, single or multinational, single or multicenter (sRCT or mRCT)), intention to treat (ITT) analysis, and countries of origin.

RESULTS

Almost half of RCTs were sRCTs. Median sample size was small, and 1/3 were not analyzed according to ITT principle. Lack of ITT analysis was associated with greater effect size (p=0.0028). Harm was more likely in mRCTs (p=0.002) and/or in blinded RCTs (p=0.003). Blinded RCTs had double sample size (p=0.007) and an increased NNT/NNH (p=0.002). Finally, mRCTs had higher NNT (p=0.005) and NNH (p=0.02), and harm was only detected in studies from Western countries (p=0.007).

CONCLUSIONS

These observations imply that major systematic biases exist and affect trial findings irrespective of the intervention being studied.