Covariate adjustment in randomized controlled trials: General concepts and practical considerations

There has been a growing interest in covariate adjustment in the analysis of randomized controlled trials in past years. For instance, the US Food and Drug Administration recently issued guidance that emphasizes the importance of distinguishing between conditional and marginal treatment effects. Although these effects may sometimes coincide in the context of linear models, this is not typically the case in other settings, and this distinction is often overlooked in clinical trial practice. Considering these developments, this article provides a review of when and how to use covariate adjustment to enhance precision in randomized controlled trials. We describe the differences between conditional and marginal estimands and stress the necessity of aligning statistical analysis methods with the chosen estimand. In addition, we highlight the potential misalignment of commonly used methods in estimating marginal treatment effects. We hereby advocate for the use of the standardization approach, as it can improve efficiency by leveraging the information contained in baseline covariates while remaining robust to model misspecification. Finally, we present practical considerations that have arisen in our respective consultations to further clarify the advantages and limitations of covariate adjustment.

Estimation of separable direct and indirect effects in a continuous-time illness-death model

In this article we study the effect of a baseline exposure on a terminal time-to-event outcome either directly or mediated by the illness state of a continuous-time illness-death process with baseline covariates. We propose a definition of the corresponding direct and indirect effects using the concept of separable (interventionist) effects (Robins and Richardson in Causality and psychopathology: finding the determinants of disorders and their cures, Oxford University Press, 2011; Robins et al. in arXiv:2008.06019 , 2021; Stensrud et al. in J Am Stat Assoc 117:175-183, 2022). Our proposal generalizes Martinussen and Stensrud (Biometrics 79:127-139, 2023) who consider similar causal estimands for disentangling the causal treatment effects on the event of interest and competing events in the standard continuous-time competing risk model. Unlike natural direct and indirect effects (Robins and Greenland in Epidemiology 3:143-155, 1992; Pearl in Proceedings of the seventeenth conference on uncertainty in artificial intelligence, Morgan Kaufmann, 2001) which are usually defined through manipulations of the mediator independently of the exposure (so-called cross-world interventions), separable direct and indirect effects are defined through interventions on different components of the exposure that exert their effects through distinct causal mechanisms. This approach allows us to define meaningful mediation targets even though the mediating event is truncated by the terminal event. We present the conditions for identifiability, which include some arguably restrictive structural assumptions on the treatment mechanism, and discuss when such assumptions are valid. The identifying functionals are used to construct plug-in estimators for the separable direct and indirect effects. We also present multiply robust and asymptotically efficient estimators based on the efficient influence functions. We verify the theoretical properties of the estimators in a simulation study, and we demonstrate the use of the estimators using data from a Danish registry study.

On logistic regression with right censored data, with or without competing risks, and its use for estimating treatment effects

Simple logistic regression can be adapted to deal with right-censoring by inverse probability of censoring weighting (IPCW). We here compare two such IPCW approaches, one based on weighting the outcome, the other based on weighting the estimating equations. We study the large sample properties of the two approaches and show that which of the two weighting methods is the most efficient depends on the censoring distribution. We show by theoretical computations that the methods can be surprisingly different in realistic settings. We further show how to use the two weighting approaches for logistic regression to estimate causal treatment effects, for both observational studies and randomized clinical trials (RCT). Several estimators for observational studies are compared and we present an application to registry data. We also revisit interesting robustness properties of logistic regression in the context of RCTs, with a particular focus on the IPCW weighting. We find that these robustness properties still hold when the censoring weights are correctly specified, but not necessarily otherwise.

Estimation of separable direct and indirect effects in continuous time

Many research questions involve time-to-event outcomes that can be prevented from occurring due to competing events. In these settings, we must be careful about the causal interpretation of classical statistical estimands. In particular, estimands on the hazard scale, such as ratios of cause-specific or subdistribution hazards, are fundamentally hard to interpret causally. Estimands on the risk scale, such as contrasts of cumulative incidence functions, do have a clear causal interpretation, but they only capture the total effect of the treatment on the event of interest; that is, effects both through and outside of the competing event. To disentangle causal treatment effects on the event of interest and competing events, the separable direct and indirect effects were recently introduced. Here we provide new results on the estimation of direct and indirect separable effects in continuous time. In particular, we derive the nonparametric influence function in continuous time and use it to construct an estimator that has certain robustness properties. We also propose a simple estimator based on semiparametric models for the two cause-specific hazard functions. We describe the asymptotic properties of these estimators and present results from simulation studies, suggesting that the estimators behave satisfactorily in finite samples. Finally, we reanalyze the prostate cancer trial from Stensrud et al. (2020).

Selection of the optimal personalized treatment from multiple treatments with right-censored multivariate outcome measures

We propose a novel personalized concept for the optimal treatment selection for a situation where the response is a multivariate vector that could contain right-censored variables such as survival time. The proposed method can be applied with any number of treatments and outcome variables, under a broad set of models. Following a working semiparametric Single Index Model that relates covariates and responses, we first define a patient-specific composite score, constructed from individual covariates. We then estimate conditional means of each response, given the patient score, correspond to each treatment, using a nonparametric smooth estimator. Next, a rank aggregation technique is applied to estimate an ordering of treatments based on ranked lists of treatment performance measures given by conditional means. We handle the right-censored data by incorporating the inverse probability of censoring weighting to the corresponding estimators. An empirical study illustrates the performance of the proposed method in finite sample problems. To show the applicability of the proposed procedure for real data, we also present a data analysis using HIV clinical trial data, that contained a right-censored survival event as one of the endpoints.

Sample size calculation for the augmented logrank test in randomized clinical trials

In randomized clinical trials, incorporating baseline covariates can improve the power in hypothesis testing for treatment effects. For survival endpoints, the Cox proportional hazards model with baseline covariates as explanatory variables can improve the standard logrank test in power. Although this has long been recognized, this adjustment is not commonly used as the primary analysis and instead the logrank test followed by the estimation of the hazard ratio between treatment groups is often used. By projecting the score function for the Cox proportional hazards model onto a space of covariates, the logrank test can be more powerful. We derive a power formula for this augmented logrank test under the same setting as the widely used power formula for the logrank test and propose a simple strategy for sizing randomized clinical trials utilizing historical data of the control treatment. Through numerical studies, the proposed procedure was found to have the potential to reduce the sample size substantially as compared to the standard logrank test. A concern to utilize historical data is that those might not reflect well the data structure of the study to design and then the sample size calculated might not be accurate. Since our power formula is applicable to datasets pooled across the treatment arms, the validity of the power calculation at the design stage can be checked in blind reviews.

Estimation for an accelerated failure time model with intermediate states as auxiliary information

The accelerated failure time (AFT) model is a common method for estimating the effect of a covariate directly on a patient's survival time. In some cases, death is the final (absorbing) state of a progressive multi-state process, however when the survival time for a subject is censored, traditional AFT models ignore the intermediate information from the subject's most recent disease state despite its relevance to the mortality process. We propose a method to estimate an AFT model for survival time to the absorbing state that uses the additional data on intermediate state transition times as auxiliary information when a patient is right censored. The method extends the Gehan AFT estimating equation by conditioning on each patient's censoring time and their disease state at their censoring time. With simulation studies, we demonstrate that the estimator is empirically unbiased, and can improve efficiency over commonly used estimators that ignore the intermediate states.

Robust Alternatives to ANCOVA for Estimating the Treatment Effect via a Randomized Comparative Study

In comparing two treatments via a randomized clinical trial, the analysis of covariance (ANCOVA) technique is often utilized to estimate an overall treatment effect. The ANCOVA is generally perceived as a more efficient procedure than its simple two sample estimation counterpart. Unfortunately, when the ANCOVA model is nonlinear, the resulting estimator is generally not consistent. Recently, various nonparametric alternatives to the ANCOVA, such as the augmentation methods, have been proposed to estimate the treatment effect by adjusting the covariates. However, the properties of these alternatives have not been studied in the presence of treatment allocation imbalance. In this article, we take a different approach to explore how to improve the precision of the naive two-sample estimate even when the observed distributions of baseline covariates between two groups are dissimilar. Specifically, we derive a bias-adjusted estimation procedure constructed from a conditional inference principle via relevant ancillary statistics from the observed covariates. This estimator is shown to be asymptotically equivalent to an augmentation estimator under the unconditional setting. We utilize the data from a clinical trial for evaluating a combination treatment of cardiovascular diseases to illustrate our findings.

Augmented estimation for t-year survival with censored regression models

Reliable and accurate risk prediction is fundamental for successful management of clinical conditions. Estimating comprehensive risk prediction models precisely, however, is a difficult task, especially when the outcome of interest is time to a rare event and the number of candidate predictors, p, is not very small. Another challenge in developing accurate risk models arises from potential model misspecification. Time-specific generalized linear models estimated with inverse censoring probability weighting are robust to model misspecification, but may be inefficient in the rare event setting. To improve the efficiency of such robust estimation procedures, various augmentation methods have been proposed in the literature. These procedures can also leverage auxiliary variables such as intermediate outcomes that are predictive of event risk. However, most existing methods do not perform well in the rare event setting, especially when p is not small. In this article, we propose a two-step, imputation-based augmentation procedure that can improve estimation efficiency and that is robust to model misspecification. We also develop regularized augmentation procedures for settings where p is not small, along with procedures to improve the estimation of individualized treatment effect in risk reduction. Numerical studies suggest that our proposed methods substantially outperform existing methods in efficiency gains. The proposed methods are applied to an AIDS clinical trial for treating HIV-infected patients.

Landmark estimation of survival and treatment effects in observational studies

Clinical studies aimed at identifying effective treatments to reduce the risk of disease or death often require long term follow-up of participants in order to observe a sufficient number of events to precisely estimate the treatment effect. In such studies, observing the outcome of interest during follow-up may be difficult and high rates of censoring may be observed which often leads to reduced power when applying straightforward statistical methods developed for time-to-event data. Alternative methods have been proposed to take advantage of auxiliary information that may potentially improve efficiency when estimating marginal survival and improve power when testing for a treatment effect. Recently, Parast et al. (J Am Stat Assoc 109(505):384-394, 2014) proposed a landmark estimation procedure for the estimation of survival and treatment effects in a randomized clinical trial setting and demonstrated that significant gains in efficiency and power could be obtained by incorporating intermediate event information as well as baseline covariates. However, the procedure requires the assumption that the potential outcomes for each individual under treatment and control are independent of treatment group assignment which is unlikely to hold in an observational study setting. In this paper we develop the landmark estimation procedure for use in an observational setting. In particular, we incorporate inverse probability of treatment weights (IPTW) in the landmark estimation procedure to account for selection bias on observed baseline (pretreatment) covariates. We demonstrate that consistent estimates of survival and treatment effects can be obtained by using IPTW and that there is improved efficiency by using auxiliary intermediate event and baseline information. We compare our proposed estimates to those obtained using the Kaplan-Meier estimator, the original landmark estimation procedure, and the IPTW Kaplan-Meier estimator. We illustrate our resulting reduction in bias and gains in efficiency through a simulation study and apply our procedure to an AIDS dataset to examine the effect of previous antiretroviral therapy on survival.

Mark-specific hazard ratio model with missing multivariate marks

An objective of randomized placebo-controlled preventive HIV vaccine efficacy (VE) trials is to assess the relationship between vaccine effects to prevent HIV acquisition and continuous genetic distances of the exposing HIVs to multiple HIV strains represented in the vaccine. The set of genetic distances, only observed in failures, is collectively termed the 'mark.' The objective has motivated a recent study of a multivariate mark-specific hazard ratio model in the competing risks failure time analysis framework. Marks of interest, however, are commonly subject to substantial missingness, largely due to rapid post-acquisition viral evolution. In this article, we investigate the mark-specific hazard ratio model with missing multivariate marks and develop two inferential procedures based on (i) inverse probability weighting (IPW) of the complete cases, and (ii) augmentation of the IPW estimating functions by leveraging auxiliary data predictive of the mark. Asymptotic properties and finite-sample performance of the inferential procedures are presented. This research also provides general inferential methods for semiparametric density ratio/biased sampling models with missing data. We apply the developed procedures to data from the HVTN 502 'Step' HIV VE trial.

Use of auxiliary covariates in estimating a biomarker-adjusted treatment effect model with clinical trial data

A biomarker-adjusted treatment effect (BATE) model describes the effect of one treatment versus another on a subpopulation of patients defined by a biomarker. Such a model can be estimated from clinical trial data without relying on additional modeling assumptions, and the estimator can be made more efficient by incorporating information on the main effect of the biomarker on the outcome of interest. Motivated by an HIV trial known as THRIVE, we consider the use of auxiliary covariates, which are usually available in clinical trials and have been used in overall treatment comparisons, in estimating a BATE model. Such covariates can be incorporated using an existing augmentation technique. For a specific type of estimating functions for difference-based BATE models, the optimal augmentation depends only on the joint main effects of marker and covariates. For a ratio-based BATE model, this result holds in special cases but not in general; however, simulation results suggest that the augmentation based on the joint main effects of marker and covariates is virtually equivalent to the theoretically optimal augmentation, especially when the augmentation terms are estimated from data. Application of these methods and results to the THRIVE data yields new insights on the utility of baseline CD4 cell count and viral load as predictive or treatment selection markers.

Landmark Estimation of Survival and Treatment Effect in a Randomized Clinical Trial

In many studies with a survival outcome, it is often not feasible to fully observe the primary event of interest. This often leads to heavy censoring and thus, difficulty in efficiently estimating survival or comparing survival rates between two groups. In certain diseases, baseline covariates and the event time of non-fatal intermediate events may be associated with overall survival. In these settings, incorporating such additional information may lead to gains in efficiency in estimation of survival and testing for a difference in survival between two treatment groups. If gains in efficiency can be achieved, it may then be possible to decrease the sample size of patients required for a study to achieve a particular power level or decrease the duration of the study. Most existing methods for incorporating intermediate events and covariates to predict survival focus on estimation of relative risk parameters and/or the joint distribution of events under semiparametric models. However, in practice, these model assumptions may not hold and hence may lead to biased estimates of the marginal survival. In this paper, we propose a semi-nonparametric two-stage procedure to estimate and compare t-year survival rates by incorporating intermediate event information observed before some landmark time, which serves as a useful approach to overcome semi-competing risks issues. In a randomized clinical trial setting, we further improve efficiency through an additional calibration step. Simulation studies demonstrate substantial potential gains in efficiency in terms of estimation and power. We illustrate our proposed procedures using an AIDS Clinical Trial Protocol 175 dataset by estimating survival and examining the difference in survival between two treatment groups: zidovudine and zidovudine plus zalcitabine.

On adjustment for auxiliary covariates in additive hazard models for the analysis of randomized experiments

We consider additive hazard models (Aalen, 1989) for the effect of a randomized treatment on a survival outcome, adjusting for auxiliary baseline covariates. We demonstrate that the Aalen least squares estimator of the treatment effect parameter is asymptotically unbiased, even when the hazard's dependence on time or on the auxiliary covariates is misspecified, and even away from the null hypothesis of no treatment effect. We moreover show that adjustment for auxiliary baseline covariates does not change the asymptotic variance of the Aalen least squares estimator of the effect of a randomized treatment. We conclude that, in view of its robustness against model misspecification, Aalen least squares estimation is attractive for evaluating treatment effects on a survival outcome in randomized experiments, and that the primary reasons to consider baseline covariate adjustment in such settings may be the interest in subgroup effects, or the need to adjust for informative censoring or for baseline imbalances. Our results also shed light on the robustness of Aalen least squares estimators against model misspecification in observational studies.

Mark-specific hazard ratio model with multivariate continuous marks: an application to vaccine efficacy

In randomized placebo-controlled preventive HIV vaccine efficacy trials, an objective is to evaluate the relationship between vaccine efficacy to prevent infection and genetic distances of the exposing HIV strains to the multiple HIV sequences included in the vaccine construct, where the set of genetic distances is considered as the continuous multivariate "mark" observed in infected subjects only. This research develops a multivariate mark-specific hazard ratio model in the competing risks failure time analysis framework for the assessment of mark-specific vaccine efficacy. It allows improved efficiency of estimation by employing the semiparametric method of maximum profile likelihood estimation in the vaccine-to-placebo mark density ratio model. The model also enables the use of a more efficient estimation method for the overall log hazard ratio in the Cox model. In addition, we propose testing procedures to evaluate two relevant hypotheses concerning mark-specific vaccine efficacy. The asymptotic properties and finite-sample performance of the inferential procedures are investigated. Finally, we apply the proposed methods to data collected in the Thai RV144 HIV vaccine efficacy trial.

Estimation of treatment effects based on possibly misspecified Cox regression

In randomized clinical trials, a treatment effect on a time-to-event endpoint is often estimated by the Cox proportional hazards model. The maximum partial likelihood estimator does not make sense if the proportional hazard assumption is violated. Xu and O'Quigley (Biostatistics 1:423-439, 2000) proposed an estimating equation, which provides an interpretable estimator for the treatment effect under model misspecification. Namely it provides a consistent estimator for the log-hazard ratio among the treatment groups if the model is correctly specified, and it is interpreted as an average log-hazard ratio over time even if misspecified. However, the method requires the assumption that censoring is independent of treatment group, which is more restricted than that for the maximum partial likelihood estimator and is often violated in practice. In this paper, we propose an alternative estimating equation. Our method provides an estimator of the same property as that of Xu and O'Quigley under the usual assumption for the maximum partial likelihood estimation. We show that our estimator is consistent and asymptotically normal, and derive a consistent estimator of the asymptotic variance. If the proportional hazards assumption holds, the efficiency of the estimator can be improved by applying the covariate adjustment method based on the semiparametric theory proposed by Lu and Tsiatis (Biometrika 95:679-694, 2008).

Variable selection for covariate-adjusted semiparametric inference in randomized clinical trials

Extensive baseline covariate information is routinely collected on participants in randomized clinical trials, and it is well recognized that a proper covariate-adjusted analysis can improve the efficiency of inference on the treatment effect. However, such covariate adjustment has engendered considerable controversy, as post hoc selection of covariates may involve subjectivity and may lead to biased inference, whereas prior specification of the adjustment may exclude important variables from consideration. Accordingly, how to select covariates objectively to gain maximal efficiency is of broad interest. We propose and study the use of modern variable selection methods for this purpose in the context of a semiparametric framework, under which variable selection in modeling the relationship between outcome and covariates is separated from estimation of the treatment effect, circumventing the potential for selection bias associated with standard analysis of covariance methods. We demonstrate that such objective variable selection techniques combined with this framework can identify key variables and lead to unbiased and efficient inference on the treatment effect. A critical issue in finite samples is validity of estimators of uncertainty, such as standard errors and confidence intervals for the treatment effect. We propose an approach to estimation of sampling variation of estimated treatment effect and show its superior performance relative to that of existing methods.

Extended follow-up confirms early vaccine-enhanced risk of HIV acquisition and demonstrates waning effect over time among participants in a randomized trial of recombinant adenovirus HIV vaccine (Step Study)

BACKGROUND

The Step Study tested whether an adenovirus serotype 5 (Ad5)-vectored human immunodeficiency virus (HIV) vaccine could prevent HIV acquisition and/or reduce viral load set-point after infection. At the first interim analysis, nonefficacy criteria were met. Vaccinations were halted; participants were unblinded. In post hoc analyses, more HIV infections occurred in vaccinees vs placebo recipients in men who had Ad5-neutralizing antibodies and/or were uncircumcised. Follow-up was extended to assess relative risk of HIV acquisition in vaccinees vs placebo recipients over time.

METHODS

We used Cox proportional hazard models for analyses of vaccine effect on HIV acquisition and vaccine effect modifiers, and nonparametric and semiparametric methods for analysis of constancy of relative risk over time.

RESULTS

One hundred seventy-two of 1836 men were infected. The adjusted vaccinees vs placebo recipients hazard ratio (HR) for all follow-up time was 1.40 (95% confidence interval [CI], 1.03-1.92; P= .03). Vaccine effect differed by baseline Ad5 or circumcision status during first 18 months, but neither was significant for all follow-up time. The HR among uncircumcised and/or Ad5-seropositive men waned with time since vaccination. No significant vaccine-associated risk was seen among circumcised, Ad5-negative men (HR, 0.97; P=1.0) over all follow-up time.

CONCLUSIONS

The vaccine-associated risk seen in interim analysis was confirmed but waned with time from vaccination.

Semiparametric estimation of treatment effect with time-lagged response in the presence of informative censoring

In many randomized clinical trials, the primary response variable, for example, the survival time, is not observed directly after the patients enroll in the study but rather observed after some period of time (lag time). It is often the case that such a response variable is missing for some patients due to censoring that occurs when the study ends before the patient's response is observed or when the patients drop out of the study. It is often assumed that censoring occurs at random which is referred to as noninformative censoring; however, in many cases such an assumption may not be reasonable. If the missing data are not analyzed properly, the estimator or test for the treatment effect may be biased. In this paper, we use semiparametric theory to derive a class of consistent and asymptotically normal estimators for the treatment effect parameter which are applicable when the response variable is right censored. The baseline auxiliary covariates and post-treatment auxiliary covariates, which may be time-dependent, are also considered in our semiparametric model. These auxiliary covariates are used to derive estimators that both account for informative censoring and are more efficient then the estimators which do not consider the auxiliary covariates.

An audit strategy for progression-free survival

In randomized clinical trials, the use of potentially subjective endpoints has led to frequent use of blinded independent central review (BICR) and event adjudication committees to reduce possible bias in treatment effect estimators based on local evaluations (LE). In oncology trials, progression-free survival (PFS) is one such endpoint. PFS requires image interpretation to determine whether a patient's cancer has progressed, and BICR has been advocated to reduce the potential for endpoints to be biased by knowledge of treatment assignment. There is current debate, however, about the value of such reviews with time-to-event outcomes such as PFS. We propose a BICR audit strategy as an alternative to a complete-case BICR to provide assurance of the presence of a treatment effect. We develop an auxiliary-variable estimator of the log-hazard ratio that is more efficient than simply using the audited (i.e., sampled) BICR data for estimation. Our estimator incorporates information from the LE on all the cases and the audited BICR cases, and is an asymptotically unbiased estimator of the log-hazard ratio from BICR. The estimator offers considerable efficiency gains that improve as the correlation between LE and BICR increases. A two-stage auditing strategy is also proposed and evaluated through simulation studies. The method is applied retrospectively to a large oncology trial that had a complete-case BICR, showing the potential for efficiency improvements.

Statistical interpretation of the RV144 HIV vaccine efficacy trial in Thailand: a case study for statistical issues in efficacy trials

Recently, the RV144 randomized, double-blind, efficacy trial in Thailand reported that a prime-boost human immunodeficiency virus (HIV) vaccine regimen conferred ∼30% protection against HIV acquisition. However, different analyses seemed to give conflicting results, and a heated debate ensued as scientists and the broader public struggled with their interpretation. The lack of accounting for statistical principles helped flame the debate, and we leverage these principles to provide a more scientific interpretation. We first address interpretation of frequentist results, including interpretation of P values, synthesis of results from multiple analyses (ie, intention-to-treat versus per-protocol/fully immunized), and accounting for external efficacy trials. Second, we address how Bayesian statistics, which provide clearly interpretable statements about probabilities that the vaccine efficacy takes certain values, provide more information for weighing the evidence about efficacy than do frequentist statistics alone. Third, we evaluate RV144 for completeness of end point ascertainment and integrity of blinding, necessary tasks for establishing robustly interpretable results.

Semiparametric inference for a 2-stage outcome-auxiliary-dependent sampling design with continuous outcome

Two-stage design has long been recognized to be a cost-effective way for conducting biomedical studies. In many trials, auxiliary covariate information may also be available, and it is of interest to exploit these auxiliary data to improve the efficiency of inferences. In this paper, we propose a 2-stage design with continuous outcome where the second-stage data is sampled with an "outcome-auxiliary-dependent sampling" (OADS) scheme. We propose an estimator which is the maximizer for an estimated likelihood function. We show that the proposed estimator is consistent and asymptotically normally distributed. The simulation study indicates that greater study efficiency gains can be achieved under the proposed 2-stage OADS design by utilizing the auxiliary covariate information when compared with other alternative sampling schemes. We illustrate the proposed method by analyzing a data set from an environmental epidemiologic study.

Increasing power in randomized trials with right censored outcomes through covariate adjustment

Targeted maximum likelihood methodology is applied to provide a test that makes use of the covariate data that are commonly collected in randomized trials, and does not require assumptions beyond those of the logrank test when censoring is uninformative. Under informative censoring, the logrank test is biased, whereas the test provided in this article is consistent under consistent estimation of the censoring mechanism or the conditional hazard for survival. Two approaches based on this methodology are provided: (1) a substitution-based approach that targets treatment and time-specific survival from which the logrank parameter is estimated, and (2) directly targeting the logrank parameter.

Fast FSR variable selection with applications to clinical trials

A new version of the false selection rate variable selection method of Wu, Boos, and Stefanski (2007, Journal of the American Statistical Association 102, 235-243) is developed that requires no simulation. This version allows the tuning parameter in forward selection to be estimated simply by hand calculation from a summary table of output even for situations where the number of explanatory variables is larger than the sample size. Because of the computational simplicity, the method can be used in permutation tests and inside bagging loops for improved prediction. Illustration is provided in clinical trials for linear regression, logistic regression, and Cox proportional hazards regression.