Planning a method for covariate adjustment in individually randomised trials: a practical guide

Remote monitoring of rheumatoid arthritis (REMORA): study protocol for a stepped wedge cluster randomized trial and process evaluation of an integrated symptom tracking intervention

BACKGROUND

Management of rheumatoid arthritis (RA) relies on symptoms reported by patients during infrequent outpatient clinic visits. These reports are often incomplete and inaccurate due to poor recall, leading to suboptimal treatment decisions and outcomes. Asking people to track symptoms in-between visits and integrating the data into clinical pathways may improve this. However, knowledge on how to implement this into practice and its impact on services and outcomes remains scarce in RA. Therefore, we evaluate the comparative effectiveness and cost-effectiveness of integrated symptom tracking in people with RA over and above usual care, while generating insights on factors for successful implementation.

METHODS

In this superiority stepped wedge cluster-randomized controlled trial with continuous recruitment short exposure design, 16 rheumatology outpatient departments (clusters) recruit a total of 732 people with active RA. They initially offer clinic visits according to standard of care before switching in pairs to visits with integrated symptom tracking. Clusters switch in randomized order every 3 weeks. Integrated symptom tracking consists of (1) a mobile app for patients to track their symptoms daily and other RA aspects weekly/monthly, and (2) an interactive dashboard visualizing the app data, which healthcare professionals access from their electronic health record system. Clinic visits happen according to usual practice, with tracked symptom data only reviewed during visits. Our primary outcome is a difference in marginal mean disease activity score at 12 ± 3 months between standard of care and integrated symptom tracking, after accounting for baseline values, cluster, and other covariates. Secondary outcomes include patient-reported disease activity, quality of life and quality-adjusted life-years, medication/resource use, consultation and decision-making experience, self-management, and illness perception. We also conduct interviews and observations as part of a parallel process evaluation to gather information on implementation.

DISCUSSION

Our trial will generate high-quality evidence of comparative and cost-effectiveness of integrated symptom tracking compared to standard of care in people with RA, with our process evaluation delivering knowledge on successful implementation. This optimizes the chances of integrated symptom tracking being adopted more widely if we find it is (cost-) effective.

TRIAL REGISTRATION

Registered 4-Jun-2024 on https://www.isrctn.com/ , ISRCTN51539448. TRIAL OPEN SCIENCE FRAMEWORK REPOSITORY: https://osf.io/sj9ha/ .

Understanding, fast and shallow: Individual differences in memory performance associated with cognitive load predict the illusion of explanatory depth

People are often overconfident about their ability to explain how everyday phenomena and artifacts work (devices, natural processes, historical events, etc.). However, the metacognitive mechanisms involved in this bias have not been fully elucidated. The aim of this study was to establish whether the ability to perform deliberate and analytic processes moderates the effect of informational cues such as the social desirability of knowledge on the Illusion of Explanatory Depth (IOED). To this purpose, the participants' cognitive load was manipulated as they provided initial estimates of causal understanding of national historical events in the standard IOED paradigm. The results showed that neither the social desirability of specific causal knowledge nor the cognitive load manipulations had direct effects on the IOED. However, subsequent exploratory analyses indicated that high cognitive load was related to lower performance on concurrent memory tasks, which in turn was associated with a higher IOED magnitude. Higher analytical processing was also related to lower IOED. Implications for both dual-process models of metacognition and the design of task environments that help to reduce this bias are discussed.

Increasing Power in Phase III Oncology Trials With Multivariable Regression: An Empirical Assessment of 535 Primary End Point Analyses

PURPOSE

A previous study demonstrated that power against the (unobserved) true effect for the primary end point (PEP) of most phase III oncology trials is low, suggesting an increased risk of false-negative findings in the field of late-phase oncology. Fitting models with prognostic covariates is a potential solution to improve power; however, the extent to which trials leverage this approach, and its impact on trial interpretation at scale, is unknown. To that end, we hypothesized that phase III trials using multivariable PEP analyses are more likely to demonstrate superiority versus trials with univariable analyses.

METHODS

PEP analyses were reviewed from trials registered on ClinicalTrials.gov. Adjusted odds ratios (aORs) were calculated by logistic regressions.

RESULTS

Of the 535 trials enrolling 454,824 patients, 69% (n = 368) used a multivariable PEP analysis. Trials with multivariable PEP analyses were more likely to demonstrate PEP superiority (57% [209 of 368] v 42% [70 of 167]; aOR, 1.78 [95% CI, 1.18 to 2.72]; P = .007). Among trials with a multivariable PEP model, 16 conditioned on covariates and 352 stratified by covariates. However, 108 (35%) of 312 trials with stratified analyses lost power by categorizing a continuous variable, which was especially common among immunotherapy trials (aOR, 2.45 [95% CI, 1.23 to 4.92]; P = .01).

CONCLUSION

Trials increasing power by fitting multivariable models were more likely to demonstrate PEP superiority than trials with unadjusted analysis. Underutilization of conditioning models and empirical power loss associated with covariate categorization required by stratification were identified as barriers to power gains. These findings underscore the opportunity to increase power in phase III trials with conventional methodology and improve patient access to effective novel therapies.

Impact of the diagnostic label for a low-risk prostate lesion: protocol for two online factorial randomised experiments

INTRODUCTION

Many types of prostate cancer present minimal risk to a man's lifespan or well-being, but existing terminology makes it difficult for men to distinguish these from high-risk prostate cancers. This study aims to explore whether using an alternative label for low-risk prostate cancer influences management choice and anxiety levels among Australian men and their partners.

METHODS AND ANALYSIS

We will run two separate studies for Australian men and Australian women with a male partner. Both studies are between-subjects factorial (3×2) randomised online hypothetical experiments. Following consent, eligible participants will be randomised 1:1:1 to three labels: 'low-risk prostate cancer, Gleason Group 1', 'low-risk prostate neoplasm' or 'low-risk prostate lesion'. Participants will then undergo a second randomisation step with 1:1 allocation to the provision of detailed information on the benefits and harms of different management choices versus the provision of less detailed information about management choices. The required sample sizes are 1290 men and 1410 women. The primary outcome is the participant choice of their preferred management strategy: no immediate treatment (prostate-specific antigen (PSA)-based monitoring or active surveillance using PSA, MRI, biopsy with delayed treatment for disease progression) versus immediate treatment (prostatectomy or radiation therapy). Secondary outcomes include preferred management choice (from the four options listed above), diagnosis anxiety, management choice anxiety and management choice at a later time point (for participants who initially choose a monitoring strategy).

ETHICS AND DISSEMINATION

Ethics approval has been received from The University of Sydney Human Research Ethics Committee (2023/572). The results of the study will be published in a peer-reviewed medical journal and a plain language summary of the findings will be shared on the Wiser Healthcare publications page http://www.wiserhealthcare.org.au/category/publications/ TRIAL REGISTRATION NUMBERS: Australian New Zealand Clinical Trials Registry (ID 386701 and 386889).

Covariate adjustment in randomized controlled trials: General concepts and practical considerations

There has been a growing interest in covariate adjustment in the analysis of randomized controlled trials in past years. For instance, the US Food and Drug Administration recently issued guidance that emphasizes the importance of distinguishing between conditional and marginal treatment effects. Although these effects may sometimes coincide in the context of linear models, this is not typically the case in other settings, and this distinction is often overlooked in clinical trial practice. Considering these developments, this article provides a review of when and how to use covariate adjustment to enhance precision in randomized controlled trials. We describe the differences between conditional and marginal estimands and stress the necessity of aligning statistical analysis methods with the chosen estimand. In addition, we highlight the potential misalignment of commonly used methods in estimating marginal treatment effects. We hereby advocate for the use of the standardization approach, as it can improve efficiency by leveraging the information contained in baseline covariates while remaining robust to model misspecification. Finally, we present practical considerations that have arisen in our respective consultations to further clarify the advantages and limitations of covariate adjustment.

D3/Penta 21 clinical trial design: A randomised non-inferiority trial with nested drug licensing substudy to assess dolutegravir and lamivudine fixed dose formulations for the maintenance of virological suppression in children with HIV-1 infection, aged 2 to 15 years

BACKGROUND

There is increasing interest in utilising two-drug regimens for HIV treatment with the goal of reducing toxicity and improve acceptability. The D3 trial evaluates the efficacy and safety of DTG/3TC in children and adolescents and includes a nested pharmacokinetics(PK) substudy for paediatric drug licensing.

METHODS

D3 is an ongoing open-label, phase III, 96-week non-inferiority randomised controlled trial(RCT) conducted in South Africa, Spain, Thailand, Uganda and the United Kingdom. D3 has enrolled 386 children aged 2- < 15 years, virologically suppressed for ≥6 months, with no prior treatment failure. Participants were randomised 1:1 to receive DTG/3TC or DTG plus two nucleoside reverse transcriptase inhibitors(NRTIs), stratified by region, age (2- < 6, 6- < 12, 12- < 15 years) and DTG use at enrolment (participants permitted to start DTG at enrolment). The primary outcome is confirmed HIV-1 RNA viral rebound ≥50 copies/mL by 96-weeks. The trial employs the Smooth Away From Expected(SAFE) non-inferiority frontier, which specifies the non-inferiority margin and significance level based on the observed event risk in the control arm. The nested PK substudy evaluates WHO weight-band-aligned dosing in the DTG/3TC arm.

DISCUSSION

D3 is the first comparative trial evaluating DTG/3TC in children and adolescents. Implications of integrating a PK substudy and supplying data for prompt regulatory submission, were carefully considered to ensure the integrity of the ongoing trial. The trial uses an innovative non-inferiority frontier for the primary analysis to allow for a lower-than-expected confirmed viral rebound risk in the control arm, while ensuring interpretability of results and maintaining the planned sample size in an already funded trial.

TRIAL REGISTRATION

International Standard Randomised Clinical Trial Number Register: ISRCTN17157458. European Clinical Trials Database: 2020-001426-57.

CLINICALTRIALS

gov: NCT04337450.

Categorisation of continuous covariates for stratified randomisation: How should we adjust?

To obtain valid inference following stratified randomisation, treatment effects should be estimated with adjustment for stratification variables. Stratification sometimes requires categorisation of a continuous prognostic variable (eg, age), which raises the question: should adjustment be based on randomisation categories or underlying continuous values? In practice, adjustment for randomisation categories is more common. We reviewed trials published in general medical journals and found none of the 32 trials that stratified randomisation based on a continuous variable adjusted for continuous values in the primary analysis. Using data simulation, this article evaluates the performance of different adjustment strategies for continuous and binary outcomes where the covariate-outcome relationship (via the link function) was either linear or non-linear. Given the utility of covariate adjustment for addressing missing data, we also considered settings with complete or missing outcome data. Analysis methods included linear or logistic regression with no adjustment for the stratification variable, adjustment for randomisation categories, or adjustment for continuous values assuming a linear covariate-outcome relationship or allowing for non-linearity using fractional polynomials or restricted cubic splines. Unadjusted analysis performed poorly throughout. Adjustment approaches that misspecified the underlying covariate-outcome relationship were less powerful and, alarmingly, biased in settings where the stratification variable predicted missing outcome data. Adjustment for randomisation categories tends to involve the highest degree of misspecification, and so should be avoided in practice. To guard against misspecification, we recommend use of flexible approaches such as fractional polynomials and restricted cubic splines when adjusting for continuous stratification variables in randomised trials.

Covariate adjustment and estimation of difference in proportions in randomized clinical trials

Difference in proportions is frequently used to measure treatment effect for binary outcomes in randomized clinical trials. The estimation of difference in proportions can be assisted by adjusting for prognostic baseline covariates to enhance precision and bolster statistical power. Standardization or g-computation is a widely used method for covariate adjustment in estimating unconditional difference in proportions, because of its robustness to model misspecification. Various inference methods have been proposed to quantify the uncertainty and confidence intervals based on large-sample theories. However, their performances under small sample sizes and model misspecification have not been comprehensively evaluated. We propose an alternative approach to estimate the unconditional variance of the standardization estimator based on the robust sandwich estimator to further enhance the finite sample performance. Extensive simulations are provided to demonstrate the performances of the proposed method, spanning a wide range of sample sizes, randomization ratios, and model specification. We apply the proposed method in a real data example to illustrate the practical utility.

Perils of Randomized Controlled Trial Survival Extrapolation Assuming Treatment Effect Waning: Why the Distinction Between Marginal and Conditional Estimates Matters

OBJECTIVES

A long-term, constant, protective treatment effect is a strong assumption when extrapolating survival beyond clinical trial follow-up; hence, sensitivity to treatment effect waning is commonly assessed for economic evaluations. Forcing a hazard ratio (HR) to 1 does not necessarily estimate loss of individual-level treatment effect accurately because of HR selection bias. A simulation study was designed to explore the behavior of marginal HRs under a waning conditional (individual-level) treatment effect and demonstrate bias in forcing a marginal HR to 1 when the estimand is "survival difference with individual-level waning".

METHODS

Data were simulated under 4 parameter combinations (varying prognostic strength of heterogeneity and treatment effect). Time-varying marginal HRs were estimated in scenarios where the true conditional HR attenuated to 1. Restricted mean survival time differences, estimated having constrained the marginal HR to 1, were compared with true values to assess bias induced by marginal constraints.

RESULTS

Under loss of conditional treatment effect, the marginal HR took a value >1 because of covariate imbalances. Constraining this value to 1 lead to restricted mean survival time difference bias of up to 0.8 years (57% increase). Inflation of effect size estimates also increased with the magnitude of initial protective treatment effect.

CONCLUSIONS

Important differences exist between survival extrapolations assuming marginal versus conditional treatment effect waning. When a marginal HR is constrained to 1 to assess efficacy under individual-level treatment effect waning, the survival benefits associated with the new treatment will be overestimated, and incremental cost-effectiveness ratios will be underestimated.

Model-based standardization using multiple imputation

BACKGROUND

When studying the association between treatment and a clinical outcome, a parametric multivariable model of the conditional outcome expectation is often used to adjust for covariates. The treatment coefficient of the outcome model targets a conditional treatment effect. Model-based standardization is typically applied to average the model predictions over the target covariate distribution, and generate a covariate-adjusted estimate of the marginal treatment effect.

METHODS

The standard approach to model-based standardization involves maximum-likelihood estimation and use of the non-parametric bootstrap. We introduce a novel, general-purpose, model-based standardization method based on multiple imputation that is easily applicable when the outcome model is a generalized linear model. We term our proposed approach multiple imputation marginalization (MIM). MIM consists of two main stages: the generation of synthetic datasets and their analysis. MIM accommodates a Bayesian statistical framework, which naturally allows for the principled propagation of uncertainty, integrates the analysis into a probabilistic framework, and allows for the incorporation of prior evidence.

RESULTS

We conduct a simulation study to benchmark the finite-sample performance of MIM in conjunction with a parametric outcome model. The simulations provide proof-of-principle in scenarios with binary outcomes, continuous-valued covariates, a logistic outcome model and the marginal log odds ratio as the target effect measure. When parametric modeling assumptions hold, MIM yields unbiased estimation in the target covariate distribution, valid coverage rates, and similar precision and efficiency than the standard approach to model-based standardization.

CONCLUSION

We demonstrate that multiple imputation can be used to marginalize over a target covariate distribution, providing appropriate inference with a correctly specified parametric outcome model and offering statistical performance comparable to that of the standard approach to model-based standardization.

Practical considerations for sample size calculation for cluster randomized trials

Cluster randomized trials are an essential design in public health and medical research, when individual randomization is infeasible or undesirable for scientific or logistical reasons. However, the correlation among observations within clusters leads to a decrease in statistical power compared to an individually randomised trial with the same total sample size. This correlation - often quantified using the intra-cluster correlation coefficient - must be accounted for in the sample size calculation to ensure that the trial is adequately powered. In this paper, we first describe the principles of sample size calculation for parallel-arm CRTs, and explain how these calculations can be extended to CRTs with cross-over designs, with a baseline measurement and stepped-wedge designs. We introduce tools to guide researchers with their sample size calculation and discuss methods to inform the choice of the a priori estimate of the intra-cluster correlation coefficient for the calculation. We also include additional considerations with respect to anticipated attrition, a small number of clusters, and use of covariates in the randomisation process and in the analysis.

Intravenous MELAtonin for prevention of Postoperative Agitation and Emergence Delirium in children (MELA-PAED): A protocol and statistical analysis plan for a randomized clinical trial

BACKGROUND

Emergence agitation and delirium in children remain a common clinical challenge in the post-anesthetic care unit. Preoperative oral melatonin has been suggested as an effective preventive drug with a favorable safety profile. The oral bioavailability of melatonin, however, is low. Therefore, the MELA-PAED trial aims to investigate the efficacy and safety of intraoperative intravenous melatonin for the prevention of emergence agitation in pediatric surgical patients.

METHODS

MELA-PAED is a randomized, double-blind, parallel two-arm, multi-center, superiority trial comparing intravenous melatonin with placebo. Four hundred participants aged 1-6 years will be randomized 1:1 to either the intervention or placebo. The intervention consists of intravenous melatonin 0.15 mg/kg administered approximately 30 min before the end of surgery. Participants will be monitored in the post-anesthetic care unit (PACU), and the Post Hospitalization Behavior Questionnaire for Ambulatory Surgery (PHBQ-AS) will be performed on days 1, 7, and 14 after the intervention. Serious Adverse Events (SAE) will be assessed up to 30 days after the intervention.

RESULTS

The primary outcome is the incidence of emergence agitation, assessed dichotomously as any Watcha score >2 during the participant's stay in the post-anesthetic care unit. Secondary outcomes are opioid consumption in the post-anesthetic care unit and adverse events. Exploratory outcomes include SAEs, postoperative pain, postoperative nausea and vomiting, and time to awakening, to first oral intake, and to discharge readiness.

CONCLUSION

The MELA-PAED trial investigates the efficacy of intravenous intraoperative melatonin for the prevention of emergence agitation in pediatric surgical patients. Results may provide further knowledge concerning the use of melatonin in pediatric perioperative care.

Use of conditional estimates of effect in cancer epidemiology: An application to lung cancer treatment

BACKGROUND

In oncology clinical trials, there is the assumption that randomization sufficiently balances confounding covariates and therefore average treatment effects are usually reported. This paper explores the wider benefits provided by conditioning on covariates for reasons other than mitigation of confounding.

METHODS

We reanalyzed the data from primary randomized controlled trials listed in two meta-analyses to explore the significance of conditioning on smoking status in terms of the effect magnitude of treatment on progression free survival in non-small cell lung cancer.

RESULTS

The reanalysis revealed that conditioning on smoking status using sub-group analyses provided the closest empiric estimate of individual treatment effect based on smoking status and significantly reduced the heterogeneity of treatment effect observed across studies. In addition, smoking status was determined to be a modifier of the effect of treatment.

CONCLUSION

Conditioning on prognostic covariates in randomized trials in oncology helps generate the closest empiric estimates of individual treatment benefit, addresses heterogeneity due to varying covariate distributions across trials and facilitates future decision making as well as evidence synthesis. Conditioning using sub-group analyses also allows examination for effect modification in meta-analysis.

A randomized double-blind clinical trial on safety and efficacy of tauroursodeoxycholic acid (TUDCA) as add-on treatment in patients affected by amyotrophic lateral sclerosis (ALS): the statistical analysis plan of TUDCA-ALS trial

BACKGROUND

Amyotrophic lateral sclerosis (ALS) is a highly debilitating neurodegenerative condition. Despite recent advancements in understanding the molecular mechanisms underlying ALS, there have been no significant improvements in therapeutic options for ALS patients in recent years. Currently, there is no cure for ALS, and the only approved treatment in Europe is riluzole, which has been shown to slow the disease progression and prolong survival by approximately 3 months. Recently, tauroursodeoxycholic acid (TUDCA) has emerged as a promising and effective treatment for neurodegenerative diseases due to its neuroprotective activities.

METHODS

The ongoing TUDCA-ALS study is a double-blinded, parallel arms, placebo-controlled, randomized multicenter phase III trial with the aim to assess the efficacy and safety of TUDCA as add-on therapy to riluzole in patients with ALS. The primary outcome measure is the treatment response defined as a minimum of 20% improvement in the ALS Functional Rating Scale-Revised (ALSFRS-R) slope during the randomized treatment period (18 months) compared to the lead-in period (3 months). Randomization will be stratified by country. Primary analysis will be conducted based on the intention-to-treat principle through an unadjusted logistic regression model. Patient recruitment commenced on February 22, 2019, and was closed on December 23, 2021. The database will be locked in September 2023.

DISCUSSION

This paper provides a comprehensive description of the statistical analysis plan in order to ensure the reproducibility of the analysis and avoid selective reporting of outcomes and data-driven analysis. Sensitivity analyses have been included in the protocol to assess the impact of intercurrent events related to the coronavirus disease 2019. By focusing on clinically meaningful and robust outcomes, this trial aims to determine whether TUDCA can be effective in slowing the disease progression in patients with ALS.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03800524 . Registered on January 11, 2019.

Key considerations for designing, conducting and analysing a cluster randomized trial

Not only do cluster randomized trials require a larger sample size than individually randomized trials, they also face many additional complexities. The potential for contamination is the most commonly used justification for using cluster randomization, but the risk of contamination should be carefully weighed against the more serious problem of questionable scientific validity in settings with post-randomization identification or recruitment of participants unblinded to the treatment allocation. In this paper we provide some simple guidelines to help researchers conduct cluster trials in a way that minimizes potential biases and maximizes statistical efficiency. The overarching theme of this guidance is that methods that apply to individually randomized trials rarely apply to cluster randomized trials. We recommend that cluster randomization be only used when necessary-balancing the benefits of cluster randomization with its increased risks of bias and increased sample size. Researchers should also randomize at the lowest possible level-balancing the risks of contamination with ensuring an adequate number of randomization units-as well as exploring other options for statistically efficient designs. Clustering should always be allowed for in the sample size calculation; and the use of restricted randomization (and adjustment in the analysis for covariates used in the randomization) should be considered. Where possible, participants should be recruited before randomizing clusters and, when recruiting (or identifying) participants post-randomization, recruiters should be masked to the allocation. In the analysis, the target of inference should align with the research question, and adjustment for clustering and small sample corrections should be used when the trial includes less than about 40 clusters.

An observer-blinded, cluster randomised trial of a typhoid conjugate vaccine in an urban South Indian cohort

BACKGROUND

Typhoid fever causes nearly 110,000 deaths among 9.24 million cases globally and disproportionately affects developing countries. As a control measure in such regions, typhoid conjugate vaccines (TCVs) are recommended by the World Health Organization (WHO). We present here the protocol of a cluster randomised vaccine trial to assess the impact of introducing TyphiBEV® vaccine to those between 1 and 30 years of age in a high-burden setting.

METHODS

The primary objective is to determine the relative and absolute rate reduction of symptomatic, blood-culture-confirmed S. Typhi infection among participants vaccinated with TyphiBEV® in vaccine clusters compared with the unvaccinated participants in non-vaccine clusters. The study population is residents of 30 wards of Vellore (a South Indian city) with participants between the ages of 1 and 30 years who provide informed consent. The wards will be divided into 60 contiguous clusters and 30 will be randomly selected for its participants to receive TyphiBEV® at the start of the study. No placebo/control is planned for the non-intervention clusters, which will receive the vaccine at the end of the trial. Participants will not be blinded to their intervention. Episodes of typhoid fever among participants will be captured via stimulated, passive fever surveillance in the area for 2 years after vaccination, which will include the most utilised healthcare facilities. Observers blinded to the participants' intervention statuses will record illness details. Relative and absolute rate reductions will be calculated at the end of this surveillance and used to estimate vaccine effectiveness.

DISCUSSION

The results from our trial will allow countries to make better-informed decisions regarding the TCV that they will roll-out and may improve the global supplies and affordability of the vaccines.

TRIAL REGISTRATION

Clinical Trials Registry of India (CTRI) CTRI/2022/03/041314. Prospectively registered on 23 March 2022 ( https://ctri.nic.in/Clinicaltrials/pmaindet2.php?trialid=62548&EncHid=&userName=vellore%20typhoid ). CTRI collects the full WHO Trial Registration Data Set.

Measuring the performance of prediction models to personalize treatment choice

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Adjustment for baseline characteristics in randomized trials using logistic regression: sample-based model versus true model

BACKGROUND

Adjustment for baseline prognostic factors in randomized clinical trials is usually performed by means of sample-based regression models. Sample-based models may be incorrect due to overfitting. To assess whether overfitting is a problem in practice, we used simulated data to examine the performance of the sample-based model in comparison to a "true" adjustment model, in terms of estimation of the treatment effect.

METHODS

We conducted a simulation study using samples drawn from a "population" in which both the treatment effect and the effect of the potential confounder were specified. The outcome variable was binary. Using logistic regression, we compared three estimates of the treatment effect in each situation: unadjusted, adjusted for the confounder using the sample, adjusted for the confounder using the true effect. Experimental factors were sample size (from 2 × 50 to 2 × 1000), treatment effect (logit of 0, 0.5, or 1.0), confounder type (continuous or binary), and confounder effect (logit of 0, - 0.5, or - 1.0). The assessment criteria for the estimated treatment effect were bias, variance, precision (proportion of estimates within 0.1 logit units), type 1 error, and power.

RESULTS

Sample-based adjustment models yielded more biased estimates of the treatment effect than adjustment models that used the true confounder effect but had similar variance, accuracy, power, and type 1 error rates. The simulation also confirmed the conservative bias of unadjusted analyses due to the non-collapsibility of the odds ratio, the smaller variance of unadjusted estimates, and the bias of the odds ratio away from the null hypothesis in small datasets.

CONCLUSIONS

Sample-based adjustment yields similar results to exact adjustment in estimating the treatment effect. Sample-based adjustment is preferable to no adjustment.

A comparison of covariate adjustment approaches under model misspecification in individually randomized trials

Adjustment for baseline covariates in randomized trials has been shown to lead to gains in power and can protect against chance imbalances in covariates. For continuous covariates, there is a risk that the the form of the relationship between the covariate and outcome is misspecified when taking an adjusted approach. Using a simulation study focusing on individually randomized trials with small sample sizes, we explore whether a range of adjustment methods are robust to misspecification, either in the covariate-outcome relationship or through an omitted covariate-treatment interaction. Specifically, we aim to identify potential settings where G-computation, inverse probability of treatment weighting (IPTW), augmented inverse probability of treatment weighting (AIPTW) and targeted maximum likelihood estimation (TMLE) offer improvement over the commonly used analysis of covariance (ANCOVA). Our simulations show that all adjustment methods are generally robust to model misspecification if adjusting for a few covariates, sample size is 100 or larger, and there are no covariate-treatment interactions. When there is a non-linear interaction of treatment with a skewed covariate and sample size is small, all adjustment methods can suffer from bias; however, methods that allow for interactions (such as G-computation with interaction and IPTW) show improved results compared to ANCOVA. When there are a high number of covariates to adjust for, ANCOVA retains good properties while other methods suffer from under- or over-coverage. An outstanding issue for G-computation, IPTW and AIPTW in small samples is that standard errors are underestimated; they should be used with caution without the availability of small-sample corrections, development of which is needed. These findings are relevant for covariate adjustment in interim analyses of larger trials.

Less is More? First Impressions From COSMIC-313

The COSMIC-313 phase 3 randomized controlled trial tested the triplet combination of cabozantinib with nivolumab and ipilimumab in comparison with nivolumab plus ipilimumab control as fist-line systemic therapy in metastatic clear cell renal cell carcinoma. The first results presented at the 2022 European Society of Medical Oncology Congress are a milestone for the renal cell carcinoma field because they signal the advent of triplet combinations as potential treatment options for our patients. The present commentary highlights some considerations and potential next steps based on these first impressions.

Adjustment for Baseline Characteristics in Randomized Clinical Trials

This Guide to Statistics and Methods provides an overview of the use of adjustment for baseline characteristics in the analysis of randomized clinical trials and emphasizes several important considerations.

Two-stage matching-adjusted indirect comparison

BACKGROUND

Anchored covariate-adjusted indirect comparisons inform reimbursement decisions where there are no head-to-head trials between the treatments of interest, there is a common comparator arm shared by the studies, and there are patient-level data limitations. Matching-adjusted indirect comparison (MAIC), based on propensity score weighting, is the most widely used covariate-adjusted indirect comparison method in health technology assessment. MAIC has poor precision and is inefficient when the effective sample size after weighting is small.

METHODS

A modular extension to MAIC, termed two-stage matching-adjusted indirect comparison (2SMAIC), is proposed. This uses two parametric models. One estimates the treatment assignment mechanism in the study with individual patient data (IPD), the other estimates the trial assignment mechanism. The first model produces inverse probability weights that are combined with the odds weights produced by the second model. The resulting weights seek to balance covariates between treatment arms and across studies. A simulation study provides proof-of-principle in an indirect comparison performed across two randomized trials. Nevertheless, 2SMAIC can be applied in situations where the IPD trial is observational, by including potential confounders in the treatment assignment model. The simulation study also explores the use of weight truncation in combination with MAIC for the first time.

RESULTS

Despite enforcing randomization and knowing the true treatment assignment mechanism in the IPD trial, 2SMAIC yields improved precision and efficiency with respect to MAIC in all scenarios, while maintaining similarly low levels of bias. The two-stage approach is effective when sample sizes in the IPD trial are low, as it controls for chance imbalances in prognostic baseline covariates between study arms. It is not as effective when overlap between the trials' target populations is poor and the extremity of the weights is high. In these scenarios, truncation leads to substantial precision and efficiency gains but induces considerable bias. The combination of a two-stage approach with truncation produces the highest precision and efficiency improvements.

CONCLUSIONS

Two-stage approaches to MAIC can increase precision and efficiency with respect to the standard approach by adjusting for empirical imbalances in prognostic covariates in the IPD trial. Further modules could be incorporated for additional variance reduction or to account for missingness and non-compliance in the IPD trial.