Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements

Covariate adjustment had similar benefits in small and large randomized controlled trials

OBJECTIVES

Covariate adjustment is a standard statistical approach in the analysis of randomized controlled trials. We aimed to explore whether the benefit of covariate adjustment on statistical significance and power differed between small and large trials, where chance imbalance in prognostic factors necessarily differs.

STUDY DESIGN AND SETTING

We studied two large trial data sets [Global Use of Strategies to Open Occluded Coronary Arteries (GUSTO-I), N = 30,510 and International Stroke Trial (IST), N = 18,372] repeatedly drawing random samples (500,000 times) of sizes 300 and 5,000 per arm and simulated each primary outcome using the control arms. We empirically determined the treatment effects required to fix power at 80% for all unadjusted analyses and calculated the joint probabilities in the discordant cells when cross-classifying adjusted and unadjusted results from logistic regression models (ie, P < 0.05 vs. P ≥ 0.05).

RESULTS

The power gained from an adjusted analysis for small and large samples was between 5% and 6%. Similar proportions of discordance were noted irrespective of the sample size in both the GUSTO-I and the IST data sets.

CONCLUSION

The proportions of change in statistical significance from covariate adjustment of strongly prognostic characteristics were the same for small and large trials with similar gains in statistical power. Covariate adjustment is equally recommendable in small and large trials.

Should we adjust for a confounder if empirical and theoretical criteria yield contradictory results? A simulation study

Confounders can be identified by one of two main strategies: empirical or theoretical. Although confounder identification strategies that combine empirical and theoretical strategies have been proposed, the need for adjustment remains unclear if the empirical and theoretical criteria yield contradictory results due to random error. We simulated several scenarios to mimic either the presence or the absence of a confounding effect and tested the accuracy of the exposure-outcome association estimates with and without adjustment. Various criteria (significance criterion, Change-in-estimate(CIE) criterion with a 10% cutoff and with a simulated cutoff) were imposed, and a range of sample sizes were trialed. In the presence of a true confounding effect, unbiased estimates were obtained only by using the CIE criterion with a simulated cutoff. In the absence of a confounding effect, all criteria performed well regardless of adjustment. When the confounding factor was affected by both exposure and outcome, all criteria yielded accurate estimates without adjustment, but the adjusted estimates were biased. To conclude, theoretical confounders should be adjusted for regardless of the empirical evidence found. The adjustment for factors that do not have a confounding effect minimally effects. Potential confounders affected by both exposure and outcome should not be adjusted for.

Outcomes and statistical power in adult critical care randomized trials

RATIONALE

Intensive care unit (ICU)-based randomized clinical trials (RCTs) among adult critically ill patients commonly fail to detect treatment benefits.

OBJECTIVES

Appraise the rates of success, outcomes used, statistical power, and design characteristics of published trials.

METHODS

One hundred forty-six ICU-based RCTs of diagnostic, therapeutic, or process/systems interventions published from January 2007 to May 2013 in 16 high-impact general or critical care journals were studied.

MEASUREMENT AND MAIN RESULTS

Of 146 RCTs, 54 (37%) were positive (i.e., the a priori hypothesis was found to be statistically significant). The most common primary outcomes were mortality (n = 40 trials), infection-related outcomes (n = 33), and ventilation-related outcomes (n = 30), with positive results found in 10, 58, and 43%, respectively. Statistical power was discussed in 135 RCTs (92%); 92 cited a rationale for their power parameters. Twenty trials failed to achieve at least 95% of their reported target sample size, including 11 that were stopped early due to insufficient accrual/logistical issues. Of 34 superiority RCTs comparing mortality between treatment arms, 13 (38%) accrued a sample size large enough to find an absolute mortality reduction of 10% or less. In 22 of these trials the observed control-arm mortality rate differed from the predicted rate by at least 7.5%.

CONCLUSIONS

ICU-based RCTs are commonly negative and powered to identify what appear to be unrealistic treatment effects, particularly when using mortality as the primary outcome. Additional concerns include a lack of standardized methods for assessing common outcomes, unclear justifications for statistical power calculations, insufficient patient accrual, and incorrect predictions of baseline event rates.

Covariate imbalance and adjustment for logistic regression analysis of clinical trial data

In logistic regression analysis for binary clinical trial data, adjusted treatment effect estimates are often not equivalent to unadjusted estimates in the presence of influential covariates. This article uses simulation to quantify the benefit of covariate adjustment in logistic regression. However, International Conference on Harmonization guidelines suggest that covariate adjustment be prespecified. Unplanned adjusted analyses should be considered secondary. Results suggest that if adjustment is not possible or unplanned in a logistic setting, balance in continuous covariates can alleviate some (but never all) of the shortcomings of unadjusted analyses. The case of log binomial regression is also explored.

The risks and rewards of covariate adjustment in randomized trials: an assessment of 12 outcomes from 8 studies

Background

Adjustment for prognostic covariates can lead to increased power in the analysis of randomized trials. However, adjusted analyses are not often performed in practice.

Methods

We used simulation to examine the impact of covariate adjustment on 12 outcomes from 8 studies across a range of therapeutic areas. We assessed (1) how large an increase in power can be expected in practice; and (2) the impact of adjustment for covariates that are not prognostic.

Results

Adjustment for known prognostic covariates led to large increases in power for most outcomes. When power was set to 80% based on an unadjusted analysis, covariate adjustment led to a median increase in power to 92.6% across the 12 outcomes (range 80.6 to 99.4%). Power was increased to over 85% for 8 of 12 outcomes, and to over 95% for 5 of 12 outcomes. Conversely, the largest decrease in power from adjustment for covariates that were not prognostic was from 80% to 78.5%.

Conclusions

Adjustment for known prognostic covariates can lead to substantial increases in power, and should be routinely incorporated into the analysis of randomized trials. The potential benefits of adjusting for a small number of possibly prognostic covariates in trials with moderate or large sample sizes far outweigh the risks of doing so, and so should also be considered.

Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: a simulation study

Background

Analysis of variance (ANOVA), change-score analysis (CSA) and analysis of covariance (ANCOVA) respond differently to baseline imbalance in randomized controlled trials. However, no empirical studies appear to have quantified the differential bias and precision of estimates derived from these methods of analysis, and their relative statistical power, in relation to combinations of levels of key trial characteristics. This simulation study therefore examined the relative bias, precision and statistical power of these three analyses using simulated trial data.

Methods

126 hypothetical trial scenarios were evaluated (126 000 datasets), each with continuous data simulated by using a combination of levels of: treatment effect; pretest-posttest correlation; direction and magnitude of baseline imbalance. The bias, precision and power of each method of analysis were calculated for each scenario.

Results

Compared to the unbiased estimates produced by ANCOVA, both ANOVA and CSA are subject to bias, in relation to pretest-posttest correlation and the direction of baseline imbalance. Additionally, ANOVA and CSA are less precise than ANCOVA, especially when pretest-posttest correlation ≥ 0.3. When groups are balanced at baseline, ANCOVA is at least as powerful as the other analyses. Apparently greater power of ANOVA and CSA at certain imbalances is achieved in respect of a biased treatment effect.

Conclusions

Across a range of correlations between pre- and post-treatment scores and at varying levels and direction of baseline imbalance, ANCOVA remains the optimum statistical method for the analysis of continuous outcomes in RCTs, in terms of bias, precision and statistical power.

Continuous covariate imbalance and conditional power for clinical trial interim analyses

Oftentimes valid statistical analyses for clinical trials involve adjustment for known influential covariates, regardless of imbalance observed in these covariates at baseline across treatment groups. Thus, it must be the case that valid interim analyses also properly adjust for these covariates. There are situations, however, in which covariate adjustment is not possible, not planned, or simply carries less merit as it makes inferences less generalizable and less intuitive. In this case, covariate imbalance between treatment groups can have a substantial effect on both interim and final primary outcome analyses. This paper illustrates the effect of influential continuous baseline covariate imbalance on unadjusted conditional power (CP), and thus, on trial decisions based on futility stopping bounds. The robustness of the relationship is illustrated for normal, skewed, and bimodal continuous baseline covariates that are related to a normally distributed primary outcome. Results suggest that unadjusted CP calculations in the presence of influential covariate imbalance require careful interpretation and evaluation.

Coping with persistent pain, effectiveness research into self-management (COPERS): statistical analysis plan for a randomised controlled trial

BACKGROUND

The Coping with Persistent Pain, Effectiveness Research into Self-management (COPERS) trial assessed whether a group-based self-management course is effective in reducing pain-related disability in participants with chronic musculoskeletal pain. This article describes the statistical analysis plan for the COPERS trial.

METHODS AND DESIGN

COPERS was a pragmatic, multicentre, unmasked, parallel group, randomised controlled trial. This article describes (a) the overall analysis principles (including which participants will be included in each analysis, how results will be presented, which covariates will be adjusted for, and how we will account for clustering in the intervention group); (b) the primary and secondary outcomes, and how each outcome will be analysed; (c) sensitivity analyses; (d) subgroup analyses; and (e) adherence-adjusted analyses.

TRIAL REGISTRATION

ISRCTN24426731.

Advancing care for traumatic brain injury: findings from the IMPACT studies and perspectives on future research

Research in traumatic brain injury (TBI) is challenging for several reasons; in particular, the heterogeneity between patients regarding causes, pathophysiology, treatment, and outcome. Advances in basic science have failed to translate into successful clinical treatments, and the evidence underpinning guideline recommendations is weak. Because clinical research has been hampered by non-standardised data collection, restricted multidisciplinary collaboration, and the lack of sensitivity of classification and efficacy analyses, multidisciplinary collaborations are now being fostered. Approaches to deal with heterogeneity have been developed by the IMPACT study group. These approaches can increase statistical power in clinical trials by up to 50% and are also relevant to other heterogeneous neurological diseases, such as stroke and subarachnoid haemorrhage. Rather than trying to limit heterogeneity, we might also be able to exploit it by analysing differences in treatment and outcome between countries and centres in comparative effectiveness research. This approach has great potential to advance care in patients with TBI.

Adjusting for multiple prognostic factors in the analysis of randomised trials

Background

When multiple prognostic factors are adjusted for in the analysis of a randomised trial, it is unclear (1) whether it is necessary to account for each of the strata, formed by all combinations of the prognostic factors (stratified analysis), when randomisation has been balanced within each stratum (stratified randomisation), or whether adjusting for the main effects alone will suffice, and (2) the best method of adjustment in terms of type I error rate and power, irrespective of the randomisation method.

Methods

We used simulation to (1) determine if a stratified analysis is necessary after stratified randomisation, and (2) to compare different methods of adjustment in terms of power and type I error rate. We considered the following methods of analysis: adjusting for covariates in a regression model, adjusting for each stratum using either fixed or random effects, and Mantel-Haenszel or a stratified Cox model depending on outcome.

Results

Stratified analysis is required after stratified randomisation to maintain correct type I error rates when (a) there are strong interactions between prognostic factors, and (b) there are approximately equal number of patients in each stratum. However, simulations based on real trial data found that type I error rates were unaffected by the method of analysis (stratified vs unstratified), indicating these conditions were not met in real datasets. Comparison of different analysis methods found that with small sample sizes and a binary or time-to-event outcome, most analysis methods lead to either inflated type I error rates or a reduction in power; the lone exception was a stratified analysis using random effects for strata, which gave nominal type I error rates and adequate power.

Conclusions

It is unlikely that a stratified analysis is necessary after stratified randomisation except in extreme scenarios. Therefore, the method of analysis (accounting for the strata, or adjusting only for the covariates) will not generally need to depend on the method of randomisation used. Most methods of analysis work well with large sample sizes, however treating strata as random effects should be the analysis method of choice with binary or time-to-event outcomes and a small sample size.

Update on the transfusion in gastrointestinal bleeding (TRIGGER) trial: statistical analysis plan for a cluster-randomised feasibility trial

Background

Previous research has suggested an association between more liberal red blood cell (RBC) transfusion and greater risk of further bleeding and mortality following acute upper gastrointestinal bleeding (AUGIB).

Methods and design

The Transfusion in Gastrointestinal Bleeding (TRIGGER) trial is a pragmatic cluster-randomised feasibility trial which aims to evaluate the feasibility of implementing a restrictive vs. liberal RBC transfusion policy for adult patients admitted to hospital with AUGIB in the UK. This trial will help to inform the design and methodology of a phase III trial. The protocol for TRIGGER has been published in Transfusion Medicine Reviews. Recruitment began in September 2012 and was completed in March 2013. This update presents the statistical analysis plan, detailing how analysis of the TRIGGER trial will be performed. It is hoped that prospective publication of the full statistical analysis plan will increase transparency and give readers a clear overview of how TRIGGER will be analysed.

Trial registration

ISRCTN85757829

Paracetamol (Acetaminophen) in stroke 2 (PAIS 2): protocol for a randomized, placebo-controlled, double-blind clinical trial to assess the effect of high-dose paracetamol on functional outcome in patients with acute stroke and a body temperature of 36.5 °C or above

RATIONALE

In the first hours after stroke onset, subfebrile temperatures and fever have been associated with poor functional outcome. In the first Paracetamol (Acetaminophen) in Stroke trial, a randomized clinical trial of 1400 patients with acute stroke, patients who were treated with high-dose paracetamol showed more improvement on the modified Rankin Scale at three-months than patients treated with placebo, but this difference was not statistically significant. In the 661 patients with a baseline body temperature of 37.0 °C or above, treatment with paracetamol increased the odds of functional improvement (odds ratio 1.43; 95% confidence interval: 1.02-1.97). This relation was also found in the patients with a body temperature of 36.5 °C or higher (odds ratio 1.31; 95% confidence interval 1.01-1.68). These findings need confirmation.

AIM

The study aims to assess the effect of high-dose paracetamol in patients with acute stroke and a body temperature of 36.5 °C or above on functional outcome.

DESIGN

The Paracetamol (Acetaminophen) In Stroke 2 trial is a multicenter, randomized, double-blind, placebo-controlled clinical trial. We use a power of 85% to detect a significant difference in the scores on the modified Rankin Scale of the paracetamol group compared with the placebo group at a level of significance of 0.05 and assume a treatment effect of 7%. Fifteen-hundred patients with acute ischemic stroke or intracerebral hemorrhage and a body temperature of 36.5 °C or above will be included within 12 h of symptom onset. Patients will be treated with paracetamol in a daily dose of six-grams or matching placebo for three consecutive days. The Paracetamol (Acetaminophen) In Stroke 2 trial has been registered as NTR2365 in The Netherlands Trial Register.

STUDY OUTCOMES

The primary outcome will be improvement on the modified Rankin Scale at three-months as analyzed by ordinal logistic regression.

DISCUSSION

If high-dose paracetamol will be proven effective, a simple, safe, and extremely cheap therapy will be available for many patients with acute stroke worldwide.

Individual participant data meta-analyses should not ignore clustering

Objectives

Individual participant data (IPD) meta-analyses often analyze their IPD as if coming from a single study. We compare this approach with analyses that rather account for clustering of patients within studies.

Study Design and Setting

Comparison of effect estimates from logistic regression models in real and simulated examples.

Results

The estimated prognostic effect of age in patients with traumatic brain injury is similar, regardless of whether clustering is accounted for. However, a family history of thrombophilia is found to be a diagnostic marker of deep vein thrombosis [odds ratio, 1.30; 95% confidence interval (CI): 1.00, 1.70; P = 0.05] when clustering is accounted for but not when it is ignored (odds ratio, 1.06; 95% CI: 0.83, 1.37; P = 0.64). Similarly, the treatment effect of nicotine gum on smoking cessation is severely attenuated when clustering is ignored (odds ratio, 1.40; 95% CI: 1.02, 1.92) rather than accounted for (odds ratio, 1.80; 95% CI: 1.29, 2.52). Simulations show models accounting for clustering perform consistently well, but downwardly biased effect estimates and low coverage can occur when ignoring clustering.

Conclusion

Researchers must routinely account for clustering in IPD meta-analyses; otherwise, misleading effect estimates and conclusions may arise.

Assessing potential sources of clustering in individually randomised trials

Background

Recent reviews have shown that while clustering is extremely common in individually randomised trials (for example, clustering within centre, therapist, or surgeon), it is rarely accounted for in the trial analysis. Our aim is to develop a general framework for assessing whether potential sources of clustering must be accounted for in the trial analysis to obtain valid type I error rates (non-ignorable clustering), with a particular focus on individually randomised trials.

Methods

A general framework for assessing clustering is developed based on theoretical results and a case study of a recently published trial is used to illustrate the concepts. A simulation study is used to explore the impact of not accounting for non-ignorable clustering in practice.

Results

Clustering is non-ignorable when there is both correlation between patient outcomes within clusters, and correlation between treatment assignments within clusters. This occurs when the intraclass correlation coefficient is non-zero, and when the cluster has been used in the randomisation process (e.g. stratified blocks within centre) or when patients are assigned to clusters after randomisation with different probabilities (e.g. a surgery trial in which surgeons treat patients in only one arm). A case study of an individually randomised trial found multiple sources of clustering, including centre of recruitment, attending surgeon, and site of rehabilitation class. Simulations show that failure to account for non-ignorable clustering in trial analyses can lead to type I error rates over 20% in certain cases; conversely, adjusting for the clustering in the trial analysis gave correct type I error rates.

Conclusions

Clustering is common in individually randomised trials. Trialists should assess potential sources of clustering during the planning stages of a trial, and account for any sources of non-ignorable clustering in the trial analysis.

Optimal Experimental Design With Nesting of Persons in Organizations

This paper introduces optimal design of randomized experiments where individuals are nested within organizations, such as schools, health centers, or companies. The focus is on nested designs with two levels (organization, individual) and two treatment conditions (treated, control), with treatment assignment to organizations, or to individuals within organizations. For each type of assignment, a multilevel model is first presented for the analysis of a quantitative dependent variable or outcome. Simple equations are then given for the optimal sample size per level (number of organizations, number of individuals) as a function of the sampling cost and outcome variance at each level, with realistic examples. Next, it is explained how the equations can be applied if the dependent variable is dichotomous, or if there are covariates in the model, or if the effects of two treatment factors are studied in a factorial nested design, or if the dependent variable is repeatedly measured. Designs with three levels of nesting and the optimal number of repeated measures are briefly discussed, and the paper ends with a short discussion of robust design.

Prognosis Research Strategy (PROGRESS) 3: prognostic model research

Prognostic models are abundant in the medical literature yet their use in practice seems limited. In this article, the third in the PROGRESS series, the authors review how such models are developed and validated, and then address how prognostic models are assessed for their impact on practice and patient outcomes, illustrating these ideas with examples.

Reporting and analysis of trials using stratified randomisation in leading medical journals: review and reanalysis

OBJECTIVES

To assess how often stratified randomisation is used, whether analysis adjusted for all balancing variables, and whether the method of randomisation was adequately reported, and to reanalyse a previously reported trial to assess the impact of ignoring balancing factors in the analysis.

DESIGN

Review of published trials and reanalysis of a previously reported trial.

SETTING

Four leading general medical journals (BMJ, Journal of the American Medical Association, Lancet, and New England Journal of Medicine) and the second Multicenter Intrapleural Sepsis Trial (MIST2).

PARTICIPANTS

258 trials published in 2010 in the four journals. Cluster randomised, crossover, non-randomised, single arm, and phase I or II trials were excluded, as were trials reporting secondary analyses, interim analyses, or results that had been previously published in 2010.

MAIN OUTCOME MEASURES

Whether the method of randomisation was adequately reported, how often balanced randomisation was used, and whether balancing factors were adjusted for in the analysis.

RESULTS

Reanalysis of MIST2 showed that an unadjusted analysis led to larger P values and a loss of power. The review of published trials showed that balanced randomisation was common, with 163 trials (63%) using at least one balancing variable. The most common methods of balancing were stratified permuted blocks (n=85) and minimisation (n=27). The method of randomisation was unclear in 37% of trials. Most trials that balanced on centre or prognostic factors were not adequately analysed; only 26% of trials adjusted for all balancing factors in their primary analysis. Trials that did not adjust for balancing factors in their analysis were less likely to show a statistically significant result (unadjusted 57% v adjusted 78%, P=0.02).

CONCLUSION

Balancing on centre or prognostic factors is common in trials but often poorly described, and the implications of balancing are poorly understood. Trialists should adjust their primary analysis for balancing factors to obtain correct P values and confidence intervals and to avoid an unnecessary loss in power.

When inferiority meets non-inferiority: implications for interim analyses

BACKGROUND

The objective of a non-inferiority trial is to determine whether a new or existing treatment is not less effective than another existing or current treatment by more than a pre-specified margin, Δ, usually with the requirement that the new treatment has some other advantage such as reduced cost or lower toxicity. A possible but unusual result in a non-inferiority trial is for the confidence interval for the treatment effect to lie between zero and Δ, implying that the new treatment is both inferior and non-inferior to the control. Such a result could occur in non-inferiority trials with large sample sizes or large non-inferiority margins. The possibility of this scenario occurring has implications for interim analyses. In standard superiority trials, stopping guidelines are often based on the p value obtained from testing whether treatments are equally effective. In non-inferiority trials, however, even if a new treatment is found to be inferior to the control at an interim analysis, there may still be a reasonable chance of demonstrating non-inferiority in the final analysis.

PURPOSE

To explore the issues arising from trials where a simultaneously inferior and non-inferior result could occur and to describe appropriate methods for deciding whether such trials should be stopped for futility at an interim analysis.

METHODS

Conditional power is used to assess futility or the inability of the trial to show non-inferiority at the final analysis, by calculating the probability of demonstrating non-inferiority in the final analysis conditional on the observed results and upon assumptions on the future results of the trial. The Bullous Pemphigoid Steroids and Tetracyclines Study (BLISTER) trial is an example of a trial where a simultaneous inferior and non-inferior result could occur. A method for calculating conditional power for non-inferiority using simulations is described and applied at a hypothetical interim analysis of this trial.

RESULTS

Stopping guidelines for futility based on conditional power are shown to be better suited to non-inferiority trials than the typical methods used in superiority trials. Simulations are a straightforward and flexible way of calculating conditional power.

LIMITATIONS

Calculating conditional power relies on assumptions about future treatment efficacy, and therefore, a number of different conditional power values can be obtained. Careful consideration should be given to which assumptions are most likely to be true. Additionally, when choosing a stopping guideline for futility, consideration needs to be given to avoid overinflating the type II error rate.

CONCLUSIONS

Conditional power is an appropriate tool for defining stopping guidelines for futility in non-inferiority trials, particularly those with large non-inferiority margins.

Assessment and implication of prognostic imbalance in randomized controlled trials with a binary outcome--a simulation study

BACKGROUND

Chance imbalance in baseline prognosis of a randomized controlled trial can lead to over or underestimation of treatment effects, particularly in trials with small sample sizes. Our study aimed to (1) evaluate the probability of imbalance in a binary prognostic factor (PF) between two treatment arms, (2) investigate the impact of prognostic imbalance on the estimation of a treatment effect, and (3) examine the effect of sample size (n) in relation to the first two objectives.

METHODS

We simulated data from parallel-group trials evaluating a binary outcome by varying the risk of the outcome, effect of the treatment, power and prevalence of the PF, and n. Logistic regression models with and without adjustment for the PF were compared in terms of bias, standard error, coverage of confidence interval and statistical power.

RESULTS

For a PF with a prevalence of 0.5, the probability of a difference in the frequency of the PF≥5% reaches 0.42 with 125/arm. Ignoring a strong PF (relative risk = 5) leads to underestimating the strength of a moderate treatment effect, and the underestimate is independent of n when n is >50/arm. Adjusting for such PF increases statistical power. If the PF is weak (RR = 2), adjustment makes little difference in statistical inference. Conditional on a 5% imbalance of a powerful PF, adjustment reduces the likelihood of large bias. If an absolute measure of imbalance ≥5% is deemed important, including 1000 patients/arm provides sufficient protection against such an imbalance. Two thousand patients/arm may provide an adequate control against large random deviations in treatment effect estimation in the presence of a powerful PF.

CONCLUSIONS

The probability of prognostic imbalance in small trials can be substantial. Covariate adjustment improves estimation accuracy and statistical power, and hence should be performed when strong PFs are observed.

Covariate adjustment increased power in randomized controlled trials: an example in traumatic brain injury

OBJECTIVE

We aimed to determine to what extent covariate adjustment could affect power in a randomized controlled trial (RCT) of a heterogeneous population with traumatic brain injury (TBI).

STUDY DESIGN AND SETTING

We analyzed 14-day mortality in 9,497 participants in the Corticosteroid Randomization After Significant Head Injury (CRASH) RCT of corticosteroid vs. placebo. Adjustment was made using logistic regression for baseline covariates of two validated risk models derived from external data (International Mission on Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury [IMPACT]) and from the CRASH data. The relative sample size (RESS) measure, defined as the ratio of the sample size required by an adjusted analysis to attain the same power as the unadjusted reference analysis, was used to assess the impact of adjustment.

RESULTS

Corticosteroid was associated with higher mortality compared with placebo (odds ratio=1.25, 95% confidence interval=1.13-1.39). RESS of 0.79 and 0.73 were obtained by adjustment using the IMPACT and CRASH models, respectively, which, for example, implies an increase from 80% to 88% and 91% power, respectively.

CONCLUSION

Moderate gains in power may be obtained using covariate adjustment from logistic regression in heterogeneous conditions such as TBI. Although analyses of RCTs might consider covariate adjustment to improve power, we caution against this approach in the planning of RCTs.

Clinical trial design in the neurocritical care unit

Clinical trials provide a robust mechanism to advance science and change clinical practice across the widest possible spectrum. Fundamental in the Neurocritical Care Society's mission is to promote Quality Patient Care by identifying and implementing best medical practices for acute neurological disorders that are consistent with the current scientific knowledge. The next logical step will be to foster rapid growth of our scientific body of evidence, to establish and disseminate these best practices. In this manuscript, five invited experts were impaneled to address questions, identified by the conference organizing committee as fundamental issues for the design of clinical trials in the neurological intensive care unit setting.

The Emergency Department Action in Smoking Cessation (EDASC) trial: impact on delivery of smoking cessation counseling

OBJECTIVES

The focus on acute care, time pressure, and lack of resources hamper the delivery of smoking cessation interventions in the emergency department (ED). The aim of this study was to 1) determine the effect of an emergency nurse-initiated intervention on delivery of smoking cessation counseling based on the 5As framework (ask-advise-assess-assist-arrange) and 2) assess ED nurses' and physicians' perceptions of smoking cessation counseling.

METHODS

The authors conducted a pre-post trial in 789 adult smokers (five or more cigarettes/day) who presented to two EDs. The intervention focused on improving delivery of the 5As by ED nurses and physicians and included face-to-face training and an online tutorial, use of a charting/reminder tool, fax referral of motivated smokers to the state tobacco quitline for proactive telephone counseling, and group feedback to ED staff. To assess ED performance of cessation counseling, a telephone interview of subjects was conducted shortly after the ED visit. Nurses' and physicians' self-efficacy, role satisfaction, and attitudes toward smoking cessation counseling were assessed by survey. Multivariable logistic regression was used to assess the effect of the intervention on performance of the 5As, while adjusting for key covariates.

RESULTS

Of 650 smokers who completed the post-ED interview, a greater proportion had been asked about smoking by an ED nurse (68% vs. 53%, adjusted odds ratio [OR] = 2.0, 95% confidence interval [CI] = 1.3 to 2.9), assessed for willingness to quit (31% vs. 9%, adjusted OR= 4.9, 95% CI = 2.9 to 7.9), and assisted in quitting (23% vs. 6%, adjusted OR = 5.1, 95% CI = 2.7 to 9.5) and had arrangements for follow-up cessation counseling (7% vs. 1%, adjusted OR = 7.1, 95% CI = 2.3 to 21) during the intervention compared to the baseline period. A similar increase was observed for emergency physicians (EPs). ED nurses' self-efficacy and role satisfaction in cessation counseling significantly improved following the intervention; however, there was no change in "pros" and "cons" attitudes toward smoking cessation in either ED nurses or physicians.

CONCLUSIONS

Emergency department nurses and physicians can effectively deliver smoking cessation counseling to smokers in a time-efficient manner. This trial also provides empirical support for expert recommendations that call for nursing staff to play a larger role in delivering public health interventions in the ED.

Robust extraction of covariate information to improve estimation efficiency in randomized trials

In randomized trials, investigators typically rely upon an unadjusted estimate of the mean outcome within each treatment arm to draw causal inferences. Statisticians have underscored the gain in efficiency that can be achieved from covariate adjustment in randomized trials with a focus on problems involving linear models. Despite recent theoretical advances, there has been a reluctance to adjust for covariates based on two primary reasons: (i) covariate-adjusted estimates based on conditional logistic regression models have been shown to be less precise and (ii) concern over the opportunity to manipulate the model selection process for covariate adjustments to obtain favorable results. In this paper, we address these two issues and summarize recent theoretical results on which is based a proposed general methodology for covariate adjustment under the framework of targeted maximum likelihood estimation in trials with two arms where the probability of treatment is 50%. The proposed methodology provides an estimate of the true causal parameter of interest representing the population-level treatment effect. It is compared with the estimates based on conditional logistic modeling, which only provide estimates of subgroup-level treatment effects rather than marginal (unconditional) treatment effects. We provide a clear criterion for determining whether a gain in efficiency can be achieved with covariate adjustment over the unadjusted method. We illustrate our strategy using a resampled clinical trial dataset from a placebo controlled phase 4 study. Results demonstrate that gains in efficiency can be achieved even with binary outcomes through covariate adjustment leading to increased statistical power.

Measuring continuous baseline covariate imbalances in clinical trial data

This paper presents and compares several methods of measuring continuous baseline covariate imbalance in clinical trial data. Simulations illustrate that though the t-test is an inappropriate method of assessing continuous baseline covariate imbalance, the test statistic itself is a robust measure in capturing imbalance in continuous covariate distributions. Guidelines to assess effects of imbalance on bias, type I error rate and power for hypothesis test for treatment effect on continuous outcomes are presented, and the benefit of covariate-adjusted analysis (ANCOVA) is also illustrated.

Optimal end points for acute stroke therapy trials: best ways to measure treatment effects of drugs and devices

BACKGROUND AND PURPOSE

Over the past decade, analysis of completed actual trials, model population studies, and theoretical work have improved approaches to selecting and analyzing end points in acute stroke treatment trials.

METHODS

Narrative review.

RESULTS

Because stroke affects persons in their biological, functional, social, and experiential dimensions, measures of impairment, disability, handicap, and quality of life are all desirable in pivotal trials, with disability being most important. Scales that are valid, reliable, responsive, and easy to administer are preferred; consequently, the modified Rankin Scale has become the most widely used single clinical efficacy measure. Because stroke cripples and kills, most outcome scales array patient outcome in ordered ranks, spread over the entire range from normal to disabled to dead. Generally, shift analysis, analyzing all health state transitions concurrently, is the most efficient analytic technique to detect treatment effects, with sliding dichotomy less efficient and fixed dichotomy least efficient, unless treatment effects strongly cluster at 1 or a few health state transitions that can be prespecified. Test statistics must also take into account interpretability, ie, how well they can be converted into metrics capturing all outcomes the intervention might alter in proportion to the degree they are valued by the patient; full ordinal analysis is most informative, sliding dichotomy is intermediately informative, and fixed dichotomy is least informative regarding this global outcome.

CONCLUSIONS

Stroke trial power and interpretation can be substantially enhanced by adherence to the principles delineated in this review. Full ordinal and sliding dichotomy analysis will most often be advantageous compared with fixed dichotomous approaches.

Early recognition of poor prognosis in Guillain-Barre syndrome

BACKGROUND

Guillain-Barré syndrome (GBS) has a highly diverse clinical course and outcome, yet patients are treated with a standard therapy. Patients with poor prognosis may benefit from additional treatment, provided they can be identified early, when nerve degeneration is potentially reversible and treatment is most effective. We developed a clinical prognostic model for early prediction of outcome in GBS, applicable for clinical practice and future therapeutic trials.

METHODS

Data collected prospectively from a derivation cohort of 397 patients with GBS were used to identify risk factors of being unable to walk at 4 weeks, 3 months, and 6 months. Potential predictors of poor outcome (unable to walk unaided) were considered in univariable and multivariable logistic regression models. The clinical model was based on the multivariable logistic regression coefficients of selected predictors and externally validated in an independent cohort of 158 patients with GBS.

RESULTS

High age, preceding diarrhea, and low Medical Research Council sumscore at hospital admission and at 1 week were independently associated with being unable to walk at 4 weeks, 3 months, and 6 months (all p 0.05-0.001). The model can be used at hospital admission and at day 7 of admission, the latter having a better predictive ability for the 3 endpoints; the area under the receiver operating characteristic curve (AUC) is 0.84-0.87 and at admission the AUC is 0.73-0.77. The model proved to be valid in the validation cohort.

CONCLUSIONS

A clinical prediction model applicable early in the course of disease accurately predicts the first 6 months outcome in GBS.

Two new covariate adjustment methods for non-inferiority assessment of binary clinical trials data

In clinical trials, examining the adjusted treatment difference has become the preferred way to establish non-inferiority (NI) in cases involving a binary endpoint. However, current methods are inadequate in the area of covariate adjustment. In this paper, we introduce two new methods, nonparametric and parametric, of using the probability and probability (P-P) curve to address the issue of unadjusted categorical covariates in the traditional assessment of NI in clinical trials. We also show that the area under the P-P curve is a valid alternative for assessing NI using the adjusted treatment difference, and we compute this area using Mann-Whitney nonparametric statistics. Our simulation studies demonstrate that our proposed methods can not only control type I error at a predefined significance level but also achieve higher statistical power than those of traditional parametric and nonparametric methods that overlook covariate adjustment, especially when covariates are unbalanced in the two treatment groups. We illustrate the effectiveness of our methodology with data from clinical trials of a therapy for coronary heart disease.

Quantifying the cost in power of ignoring continuous covariate imbalances in clinical trial randomization

Motivated by potentially serious imbalances of continuous baseline covariates in clinical trials, we investigated the cost in statistical power of ignoring the balance of these covariates in treatment allocation design for a logistic regression model. Based on data from a clinical trial of acute ischemic stroke treatment, computer simulations were used to create scenarios varying from the best possible baseline covariate balance to the worst possible imbalance, with multiple balance levels between the two extremes. The likelihood of each scenario occurring under simple randomization was evaluated. The power of the main effect test for treatment was examined. Our simulation results show that the worst possible imbalance is highly unlikely, but it can still occur under simple random allocation. Also, power loss could be nontrivial if balancing distributions of important continuous covariates were ignored even if adjustment is made in the analysis for important covariates. This situation, although unlikely, is more serious for trials with a small sample size and for covariates with large influence on primary outcome. These results suggest that attempts should be made to balance known prognostic continuous covariates at the design phase of a clinical trial even when adjustment is planned for these covariates at the analysis.

Reporting on covariate adjustment in randomised controlled trials before and after revision of the 2001 CONSORT statement: a literature review

OBJECTIVES

To evaluate the use and reporting of adjusted analysis in randomised controlled trials (RCTs) and compare the quality of reporting before and after the revision of the CONSORT Statement in 2001.

DESIGN

Comparison of two cross sectional samples of published articles.

DATA SOURCES

Journal articles indexed on PubMed in December 2000 and December 2006.

STUDY SELECTION

Parallel group RCTs with a full publication carried out in humans and published in English

MAIN OUTCOME MEASURES

Proportion of articles reported adjusted analysis; use of adjusted analysis; the reason for adjustment; the method of adjustment and the reporting of adjusted analysis results in the main text and abstract.

RESULTS

In both cohorts, 25% of studies reported adjusted analysis (84/355 in 2000 vs 113/422 in 2006). Compared with articles reporting only unadjusted analyses, articles that reported adjusted analyses were more likely to specify primary outcomes, involve multiple centers, perform stratified randomization, be published in general medical journals, and recruit larger sample sizes. In both years a minority of articles explained why and how covariates were selected for adjustment (20% to 30%). Almost all articles specified the statistical methods used for adjustment (99% in 2000 vs 100% in 2006) but only 5% and 10%, respectively, reported both adjusted and unadjusted results as recommended in the CONSORT guidelines.

CONCLUSION

There was no evidence of change in the reporting of adjusted analysis results five years after the revision of the CONSORT Statement and only a few articles adhered fully to the CONSORT recommendations.

Assessing the performance of prediction models: a framework for traditional and novel measures

The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic [ROC] curve), and goodness-of-fit statistics for calibration.Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision-analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions.We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation).We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

Effectiveness of ribavirin and corticosteroids for severe acute respiratory syndrome

OBJECTIVE

Ribavirin and corticosteroids were used widely as front-line treatments for severe acute respiratory syndrome; however, previous evaluations were inconclusive. We assessed the effectiveness of ribavirin and corticosteroids as the initial treatment for severe acute respiratory syndrome using propensity score analysis.

METHODS

We analyzed data on 1755 patients in Hong Kong and 191 patients in Toronto with severe acute respiratory syndrome using a generalized propensity score approach.

RESULTS

The adjusted excess case fatality ratios of patients with severe acute respiratory syndrome receiving the combined therapy of ribavirin and corticosteroids within 2 days of admission, compared with those receiving neither treatment within 2 days of admission, were 3.8% (95% confidence interval, -1.5 to 9.2) in Hong Kong and 2.1% (95% confidence interval, -44.3 to 48.5) in Toronto.

CONCLUSIONS

Our results add strength to the hypothesis that the combination of ribavirin and corticosteroids has no therapeutic benefit when given early during severe acute respiratory syndrome infection. Further studies may investigate the effects of these treatments later in disease course.

Using regression models to analyze randomized trials: asymptotically valid hypothesis tests despite incorrectly specified models

Regression models are often used to test for cause-effect relationships from data collected in randomized trials or experiments. This practice has deservedly come under heavy scrutiny, because commonly used models such as linear and logistic regression will often not capture the actual relationships between variables, and incorrectly specified models potentially lead to incorrect conclusions. In this article, we focus on hypothesis tests of whether the treatment given in a randomized trial has any effect on the mean of the primary outcome, within strata of baseline variables such as age, sex, and health status. Our primary concern is ensuring that such hypothesis tests have correct type I error for large samples. Our main result is that for a surprisingly large class of commonly used regression models, standard regression-based hypothesis tests (but using robust variance estimators) are guaranteed to have correct type I error for large samples, even when the models are incorrectly specified. To the best of our knowledge, this robustness of such model-based hypothesis tests to incorrectly specified models was previously unknown for Poisson regression models and for other commonly used models we consider. Our results have practical implications for understanding the reliability of commonly used, model-based tests for analyzing randomized trials.

Protocol for the PINCER trial: a cluster randomised trial comparing the effectiveness of a pharmacist-led IT-based intervention with simple feedback in reducing rates of clinically important errors in medicines management in general practices

BACKGROUND

Medication errors are an important cause of morbidity and mortality in primary care. The aims of this study are to determine the effectiveness, cost effectiveness and acceptability of a pharmacist-led information-technology-based complex intervention compared with simple feedback in reducing proportions of patients at risk from potentially hazardous prescribing and medicines management in general (family) practice.

METHODS

RESEARCH SUBJECT GROUP: "At-risk" patients registered with computerised general practices in two geographical regions in England.

DESIGN

Parallel group pragmatic cluster randomised trial.

INTERVENTIONS

Practices will be randomised to either: (i) Computer-generated feedback; or (ii) Pharmacist-led intervention comprising of computer-generated feedback, educational outreach and dedicated support.

PRIMARY OUTCOME MEASURES

The proportion of patients in each practice at six and 12 months post intervention: - with a computer-recorded history of peptic ulcer being prescribed non-selective non-steroidal anti-inflammatory drugs; - with a computer-recorded diagnosis of asthma being prescribed beta-blockers; - aged 75 years and older receiving long-term prescriptions for angiotensin converting enzyme inhibitors or loop diuretics without a recorded assessment of renal function and electrolytes in the preceding 15 months. SECONDARY OUTCOME MEASURES; These relate to a number of other examples of potentially hazardous prescribing and medicines management.

ECONOMIC ANALYSIS

An economic evaluation will be done of the cost per error avoided, from the perspective of the UK National Health Service (NHS), comparing the pharmacist-led intervention with simple feedback. QUALITATIVE ANALYSIS: A qualitative study will be conducted to explore the views and experiences of health care professionals and NHS managers concerning the interventions, and investigate possible reasons why the interventions prove effective, or conversely prove ineffective.

SAMPLE SIZE

34 practices in each of the two treatment arms would provide at least 80% power (two-tailed alpha of 0.05) to demonstrate a 50% reduction in error rates for each of the three primary outcome measures in the pharmacist-led intervention arm compared with a 11% reduction in the simple feedback arm.

DISCUSSION

At the time of submission of this article, 72 general practices have been recruited (36 in each arm of the trial) and the interventions have been delivered. Analysis has not yet been undertaken.

Should stroke trials adjust functional outcome for baseline prognostic factors?

BACKGROUND AND PURPOSE

Many stroke trials have provided neutral results. Suboptimal statistical analyses may be failing to detect effective interventions. Adjusting outcomes for baseline prognostic factors in the analysis may improve the efficiency of analysis of outcomes.

METHODS

Data from 23 stroke trials (25 674 patients) assessing functional outcome were included. The prognostic variables considered were age, sex, and baseline severity. Unadjusted and adjusted ordinal logistic regression models were compared using simulated data from each trial (10 000 simulations per trial). Three levels of treatment effect were assessed with ORs of 0.95, 0.74, and 0.57. The reduction in sample size gained from using the adjusted models, as compared with an unadjusted model, was then calculated as a reflection of the increase in statistical power.

RESULTS

Adjusting outcome for baseline factors led to a reduction in sample size, which was similar across all 3 treatment effects (median percentage reduction, interquartile range): OR=0.95: 35.3% (21.0 to 42.1); OR=0.74: 38.4% (29.4 to 42.7); and OR=0.57: 38.4% (27.4 to 42.2). As the treatment effect increased, the proportion of simulations in which the treatment effect for the adjusted model was greater than for the unadjusted model also increased.

CONCLUSIONS

Adjusting for prognostic factors in stroke trials can reduce sample size by at least 20% to 30% (the lower interquartile range) for a given power. Conversely, trialists may want to power for an unadjusted analysis and then increase statistical power by adjusting for prognostic factors.