Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals

Cluster Randomized Trials Designed to Support Generalizable Inferences

When planning a cluster randomized trial, evaluators often have access to an enumerated cohort representing the target population of clusters. Practicalities of conducting the trial, such as the need to oversample clusters with certain characteristics in order to improve trial economy or support inferences about subgroups of clusters, may preclude simple random sampling from the cohort into the trial, and thus interfere with the goal of producing generalizable inferences about the target population. We describe a nested trial design where the randomized clusters are embedded within a cohort of trial-eligible clusters from the target population and where clusters are selected for inclusion in the trial with known sampling probabilities that may depend on cluster characteristics (e.g., allowing clusters to be chosen to facilitate trial conduct or to examine hypotheses related to their characteristics). We develop and evaluate methods for analyzing data from this design to generalize causal inferences to the target population underlying the cohort. We present identification and estimation results for the expectation of the average potential outcome and for the average treatment effect, in the entire target population of clusters and in its non-randomized subset. In simulation studies, we show that all the estimators have low bias but markedly different precision. Cluster randomized trials where clusters are selected for inclusion with known sampling probabilities that depend on cluster characteristics, combined with efficient estimation methods, can precisely quantify treatment effects in the target population, while addressing objectives of trial conduct that require oversampling clusters on the basis of their characteristics.

Methods for Extending Inferences From Observational Studies: Considering Causal Structures, Identification Assumptions, and Estimators

Most prior work in quantitative approaches to generalizability and transportability emphasizes extending causal effect estimates from randomized trials to target populations. Extending findings from observational studies is also of scientific interest, and identifiability assumptions and estimation methods differ from randomized settings when there is selection on both the exposure and exposure-outcome mediators in combination with exposure-outcome confounders (and both confounders and mediators can modify exposure-outcome effects). We argue that this causal structure is common in observational studies, particularly in the field of life course epidemiology, for example, when extending estimates of the effect of an early-life exposure on a later-life outcome from a cohort enrolled in midlife or late life. We describe identifiability assumptions and identification using observed data in such settings, highlighting differences from work extending findings from randomized trials. We describe statistical methods, including weighting, outcome modeling, and doubly robust approaches, to estimate potential outcome means and average treatment effects in the target population and illustrate performance of the methods in a simulation study. We show that in the presence of selection into the study sample on both exposure and confounders, estimators must be able to address confounding in the target population. When there is also selection on mediators of the exposure-outcome relationship, estimators need to be able to use different sets of variables to account for selection (including the mediator), and confounding. We discuss conceptual implications of our results as well as highlight unresolved practical questions for applied work to extend findings from observational studies to target populations.

Learning about treatment effects in a new target population under transportability assumptions for relative effect measures

Investigators often believe that relative effect measures conditional on covariates, such as risk ratios and mean ratios, are "transportable" across populations. Here, we examine the identification of causal effects in a target population using an assumption that conditional relative effect measures are transportable from a trial to the target population. We show that transportability for relative effect measures is largely incompatible with transportability for difference effect measures, unless the treatment has no effect on average or one is willing to make even stronger transportability assumptions that imply the transportability of both relative and difference effect measures. We then describe how marginal (population-averaged) causal estimands in a target population can be identified under the assumption of transportability of relative effect measures, when we are interested in the effectiveness of a new experimental treatment in a target population where the only treatment in use is the control treatment evaluated in the trial. We extend these results to consider cases where the control treatment evaluated in the trial is only one of the treatments in use in the target population, under an additional partial exchangeability assumption in the target population (i.e., an assumption of no unmeasured confounding in the target population with respect to potential outcomes under the control treatment in the trial). We also develop identification results that allow for the covariates needed for transportability of relative effect measures to be only a small subset of the covariates needed to control confounding in the target population. Last, we propose estimators that can be easily implemented in standard statistical software and illustrate their use using data from a comprehensive cohort study of stable ischemic heart disease.

Generalizability in real-world trials

Real-world evidence (RWE) trials have a key advantage over conventional randomized controlled trials (RCTs) due to their potentially better generalizability. High generalizability of study results facilitates new biological insights and enables targeted therapeutic strategies. Random sampling of RWE trial participants is regarded as the gold standard for generalizability. Additionally, the use of sample correction procedures can increase the generalizability of trial results, even when using nonrandomly sampled real-world data (RWD). This study presents descriptive evidence on the extent to which the design of currently planned or already conducted RWE trials takes sampling into account. It also examines whether random sampling or procedures for correcting nonrandom samples are considered. Based on text mining of publicly available metadata provided during registrations of RWE trials on clinicaltrials.gov, EU-PAS, and the OSF-RWE registry, it is shown that the share of RWE trial registrations with information on sampling increased from 65.27% in 2002 to 97.43% in 2022, with a corresponding increase from 14.79% to 28.30% for trials with random samples. For RWE trials with nonrandom samples, there is an increase from 0.00% to 0.95% of trials in which sample correction procedures are used. We conclude that the potential benefits of RWD in terms of generalizing trial results are not yet being fully realized.

Transporting randomized trial results to estimate counterfactual survival functions in target populations

When the distributions of treatment effect modifiers differ between a randomized trial and an external target population, the sample average treatment effect in the trial may be substantially different from the target population average treatment, and accurate estimation of the latter requires adjusting for the differential distribution of effect modifiers. Despite the increasingly rich literature on transportability, little attention has been devoted to methods for transporting trial results to estimate counterfactual survival functions in target populations, when the primary outcome is time to event and subject to right censoring. In this article, we study inverse probability weighting and doubly robust estimators to estimate counterfactual survival functions and the target average survival treatment effect in the target population, and provide their respective approximate variance estimators. We focus on a common scenario where the target population information is observed only through a complex survey, and elucidate how the survey weights can be incorporated into each estimator we considered. Simulation studies are conducted to examine the finite-sample performances of the proposed estimators in terms of bias, efficiency and coverage, under both correct and incorrect model specifications. Finally, we apply the proposed method to assess transportability of the results in the Action to Control Cardiovascular Risk in Diabetes-Blood Pressure (ACCORD-BP) trial to all adults with Diabetes in the United States.

Invited Commentary: Combining Information to Answer Epidemiologic Questions About a Target Population

Epidemiologists are attempting to address research questions of increasing complexity by developing novel methods for combining information from diverse sources. Cole et al. (Am J Epidemiol. 2023;192(3)467-474) provide 2 examples of the process of combining information to draw inferences about a population proportion. In this commentary, we consider combining information to learn about a target population as an epidemiologic activity and distinguish it from more conventional meta-analyses. We examine possible rationales for combining information and discuss broad methodological considerations, with an emphasis on study design, assumptions, and sources of uncertainty.

genRCT: a statistical analysis framework for generalizing RCT findings to real-world population

When evaluating the real-world treatment effect, the analysis based on randomized clinical trials (RCTs) often introduces generalizability bias due to the difference in risk factors between the trial participants and the real-world patient population. This problem of lack of generalizability associated with the RCT-only analysis can be addressed by leveraging observational studies with large sample sizes that are representative of the real-world population. A set of novel statistical methods, termed "genRCT", for improving the generalizability of the trial has been developed using calibration weighting, which enforces the covariates balance between the RCT and observational study. This paper aims to review statistical methods for generalizing the RCT findings by harnessing information from large observational studies that represent real-world patients. Specifically, we discuss the choices of data sources and variables to meet key theoretical assumptions and principles. We introduce and compare estimation methods for continuous, binary, and survival endpoints. We showcase the use of the R package genRCT through a case study that estimates the average treatment effect of adjuvant chemotherapy for the stage 1B non-small cell lung patients represented by a large cancer registry.

Transporting survival of an HIV clinical trial to the external target populations

Due to the heterogeneity of the randomized controlled trial (RCT) and external target populations, the estimated treatment effect from the RCT is not directly applicable to the target population. For example, the patient characteristics of the ACTG 175 HIV trial are significantly different from that of the three external target populations of interest: US early-stage HIV patients, Thailand HIV patients, and southern Ethiopia HIV patients. This paper considers several methods to transport the treatment effect from the ACTG 175 HIV trial to the target populations beyond the trial population. Most transport methods focus on continuous and binary outcomes; on the contrary, we derive and discuss several transport methods for survival outcomes: an outcome regression method based on a Cox proportional hazard (PH) model, an inverse probability weighting method based on the models for treatment assignment, sampling score, and censoring, and a doubly robust method that combines both methods, called the augmented calibration weighting (ACW) method. However, as the PH assumption was found to be incorrect for the ACTG 175 trial, the methods that depend on the PH assumption may lead to the biased quantification of the treatment effect. To account for the violation of the PH assumption, we extend the ACW method with the linear spline-based hazard regression model that does not require the PH assumption. Applying the aforementioned methods for transportability, we explore the effect of PH assumption, or the violation thereof, on transporting the survival results from the ACTG 175 trial to various external populations.

Consensus on the definition and assessment of external validity of randomized controlled trials: A Delphi study

External validity is an important parameter that needs to be considered for decision making in health research, but no widely accepted measurement tool for the assessment of external validity of randomized controlled trials (RCTs) exists. One of the most limiting factors for creating such a tool is probably the substantial heterogeneity and lack of consensus in this field. The objective of this study was to reach consensus on a definition of external validity and on criteria to assess the external validity of RCTs included in systematic reviews. A three-round online Delphi study was conducted. The development of the Delphi survey was based on findings from a previous systematic review. Potential panelists were identified through a comprehensive web search. Consensus was reached when at least 67% of the panelists agreed to a proposal. Eighty-four panelists from different countries and various disciplines participated in at least one round of this study. Consensus was reached on the definition of external validity ("External validity is the extent to which results of trials provide an acceptable basis for generalization to other circumstances such as variations in populations, settings, interventions, outcomes, or other relevant contextual factors"), and on 14 criteria to assess the external validity of RCTs in systematic reviews. The results of this Delphi study provide a consensus-based reference standard for future tool development. Future research should focus on adapting, pilot testing, and validating these criteria to develop measurement tools for the assessment of external validity.

Visualizing External Validity: Graphical Displays to Inform the Extension of Treatment Effects from Trials to Clinical Practice

BACKGROUND

In the presence of effect measure modification, estimates of treatment effects from randomized controlled trials may not be valid in clinical practice settings. The development and application of quantitative approaches for extending treatment effects from trials to clinical practice settings is an active area of research.

METHODS

In this article, we provide researchers with a practical roadmap and four visualizations to assist in variable selection for models to extend treatment effects observed in trials to clinical practice settings and to assess model specification and performance. We apply this roadmap and visualizations to an example extending the effects of adjuvant chemotherapy (5-fluorouracil vs. plus oxaliplatin) for colon cancer from a trial population to a population of individuals treated in community oncology practices in the United States.

RESULTS

The first visualization screens for potential effect measure modifiers to include in models extending trial treatment effects to clinical practice populations. The second visualization displays a measure of covariate overlap between the clinical practice populations and the trial population. The third and fourth visualizations highlight considerations for model specification and influential observations. The conceptual roadmap describes how the output from the visualizations helps interrogate the assumptions required to extend treatment effects from trials to target populations.

CONCLUSIONS

The roadmap and visualizations can inform practical decisions required for quantitatively extending treatment effects from trials to clinical practice settings.

Transporting results in an observational epidemiology setting: purposes, methods, and applied example

In the medical domain, substantial effort has been invested in generating internally valid estimates in experimental as well as observational studies, but limited effort has been made in testing generalizability, or external validity. Testing the external validity of scientific findings is nevertheless crucial for the application of knowledge across populations. In particular, transporting estimates obtained from observational studies requires the combination of methods for causal inference and methods to transport the effect estimates in order to minimize biases inherent to observational studies and to account for differences between the study and target populations. In this paper, the conceptual framework and assumptions behind transporting results from a population-based study population to a target population is described in an observational setting. An applied example to life-course epidemiology, where internal validity was constructed for illustrative purposes, is shown by using the targeted maximum likelihood estimator.

Estimating Subgroup Effects in Generalizability and Transportability Analyses

Methods for extending-generalizing or transporting-inferences from a randomized trial to a target population involve conditioning on a large set of covariates that is sufficient for rendering the randomized and nonrandomized groups exchangeable. Yet, decision makers are often interested in examining treatment effects in subgroups of the target population defined in terms of only a few discrete covariates. Here, we propose methods for estimating subgroup-specific potential outcome means and average treatment effects in generalizability and transportability analyses, using outcome model--based (g-formula), weighting, and augmented weighting estimators. We consider estimating subgroup-specific average treatment effects in the target population and its nonrandomized subset, and we provide methods that are appropriate both for nested and non-nested trial designs. As an illustration, we apply the methods to data from the Coronary Artery Surgery Study (North America, 1975-1996) to compare the effect of surgery plus medical therapy versus medical therapy alone for chronic coronary artery disease in subgroups defined by history of myocardial infarction.

Transportability Without Positivity: A Synthesis of Statistical and Simulation Modeling

Studies designed to estimate the effect of an action in a randomized or observational setting often do not represent a random sample of the desired target population. Instead, estimates from that study can be transported to the target population. However, transportability methods generally rely on a positivity assumption, such that all relevant covariate patterns in the target population are also observed in the study sample. Strict eligibility criteria, particularly in the context of randomized trials, may lead to violations of this assumption. Two common approaches to address positivity violations are restricting the target population and restricting the relevant covariate set. As neither of these restrictions is ideal, we instead propose a synthesis of statistical and simulation models to address positivity violations. We propose corresponding g-computation and inverse probability weighting estimators. The restriction and synthesis approaches to addressing positivity violations are contrasted with a simulation experiment and an illustrative example in the context of sexually transmitted infection testing uptake. In both cases, the proposed synthesis approach accurately addressed the original research question when paired with a thoughtfully selected simulation model. Neither of the restriction approaches was able to accurately address the motivating question. As public health decisions must often be made with imperfect target population information, model synthesis is a viable approach given a combination of empirical data and external information based on the best available knowledge.

The role of randomization inference in unraveling individual treatment effects in early phase vaccine trials

Randomization inference is a powerful tool in early phase vaccine trials when estimating the causal effect of a regimen against a placebo or another regimen. Randomization-based inference often focuses on testing either Fisher's sharp null hypothesis of no treatment effect for any participant or Neyman's weak null hypothesis of no sample average treatment effect. Many recent efforts have explored conducting exact randomization-based inference for other summaries of the treatment effect profile, for instance, quantiles of the treatment effect distribution function. In this article, we systematically review methods that conduct exact, randomization-based inference for quantiles of individual treatment effects (ITEs) and extend some results to a special case where naïve participants are expected not to exhibit responses to highly specific endpoints. These methods are suitable for completely randomized trials, stratified completely randomized trials, and a matched study comparing two non-randomized arms from possibly different trials. We evaluate the usefulness of these methods using synthetic data in simulation studies. Finally, we apply these methods to HIV Vaccine Trials Network Study 086 (HVTN 086) and HVTN 205 and showcase a wide range of application scenarios of the methods. R code that replicates all analyses in this article can be found in first author's GitHub page at https://github.com/Zhe-Chen-1999/ITE-Inference.

The VANCO Trial Findings Are Generalizable to a North American Trauma Registry

OBJECTIVES

To estimate the generalizability of treatment effects observed in the VANCO trial to a broader population of patients with tibial plateau or pilon fractures.

METHODS

Design and Setting: Clinical trial data from 36 United States trauma centers and Trauma Quality Programs registry data from more than 875 Level I-III trauma centers in the United States and Canada.Patient Selection Criteria: Patients enrolled in the VANCO trial treated with intrawound vancomycin powder from January 2015 to June 2017 and 31,924 VANCO-eligible TQP patients admitted in 2019 with tibial plateau and pilon fractures.Outcome Measure and Comparisons: Deep surgical site infection and gram-positive deep surgical site infection estimated in the TQP sample weighed by the inverse probability of trial participation.

RESULTS

The 980 patients in the VANCO trial were highly representative of 31,924 TQP VANCO-eligible patients (Tipton generalizability index 0.96). It was estimated that intrawound vancomycin powder reduced the odds of deep surgical infection by odds ratio (OR) = 0.46 (95% confidence interval [CI] 0.25-0.86) and gram-positive deep surgical infection by OR = 0.39 (95% CI, 0.18-0.84) within the TQP sample of VANCO-eligible patients. For reference, the trial average treatment effects for deep surgical infection and gram-positive deep surgical infection were OR = 0.60 (95% CI, 0.37-0.98) and OR = 0.44 (95% CI, 0.23-0.80), respectively.

CONCLUSIONS

This generalizability analysis found that the inferences of the VANCO trial generalize and might even underestimate the effects of intrawound vancomycin powder when observed in a wider population of patients with tibial plateau and pilon fractures.

LEVEL OF EVIDENCE

Therapeutic Level III. See Instructions for Authors for a complete description of levels of evidence.

Quantifying and reducing inequity in average treatment effect estimation

BACKGROUND

Across studies of average treatment effects, some population subgroups consistently have lower representation than others which can lead to discrepancies in how well results generalize.

METHODS

We develop a framework for quantifying inequity due to systemic disparities in sample representation and a method for mitigation during data analysis. Assuming subgroup treatment effects are exchangeable, an unbiased sample average treatment effect estimator will have lower mean-squared error, on average across studies, for subgroups with less representation when treatment effects vary. We present a method for estimating average treatment effects in representation-adjusted samples which enables subgroups to optimally leverage information from the full sample rather than only their own subgroup's data. Two approaches for specifying representation adjustment are offered-one minimizes average mean-squared error for each subgroup separately and the other balances minimization of mean-squared error and equal representation. We conduct simulation studies to compare the performance of the proposed estimators to several subgroup-specific estimators.

RESULTS

We find that the proposed estimators generally provide lower mean squared error, particularly for smaller subgroups, relative to the other estimators. As a case study, we apply this method to a subgroup analysis from a published study.

CONCLUSIONS

We recommend the use of the proposed estimators to mitigate the impact of disparities in representation, though structural change is ultimately needed.

Real-World Cost-Effectiveness of Pulmonary Vein Isolation for Atrial Fibrillation: A Target Trial Approach

OBJECTIVES

Randomized controlled trials of pulmonary vein isolation (PVI) for treating atrial fibrillation (AF) have proven the procedure's efficacy. Studies assessing its empirical cost-effectiveness outside randomized trial settings are lacking. We aimed to evaluate the effectiveness and cost-effectiveness of PVI versus medical therapy for AF.

METHODS

We followed a target trial approach using the Swiss-AF cohort, a prospective observational cohort study that enrolled patients with AF between 2014 and 2017. Resource utilization and cost information were collected through claims data. Quality of life was measured with EQ-5D-3L utilities. We estimated incremental cost-effectiveness ratios (ICERs) from the perspective of the Swiss statutory health insurance system.

RESULTS

Patients undergoing PVI compared with medical therapy had a 5-year overall survival advantage with a hazard ratio of 0.75 (95% CI 0.46-1.21; P = .69) and a 19.8% SD improvement in quality of life (95% CI 15.5-22.9; P < .001), at an incremental cost of 29 604 Swiss francs (CHF) (95% CI 16 354-42 855; P < .001). The estimated ICER was CHF 158 612 per quality-adjusted life-year (QALY) gained within a 5-year time horizon. Assuming similar health effects and costs over 5 additional years changed the ICER to CHF 82 195 per QALY gained. Results were robust to the sensitivity analyses performed.

CONCLUSIONS

Our results show that PVI might be a cost-effective intervention within the Swiss healthcare context in a 10-year time horizon, but unlikely to be so at 5 years, if a willingness-to-pay threshold of CHF 100 000 per QALY gained is assumed. Given data availability, we find target trial designs are a valuable tool for assessing the cost-effectiveness of healthcare interventions outside of randomized controlled trial settings.

Efficient algorithms for building representative matched pairs with enhanced generalizability

Many recent efforts center on assessing the ability of real-world evidence (RWE) generated from non-randomized, observational data to produce results compatible with those from randomized controlled trials (RCTs). One noticeable endeavor is the RCT DUPLICATE initiative. To better reconcile findings from an observational study and an RCT, or two observational studies based on different databases, it is desirable to eliminate differences between study populations. We outline an efficient, network-flow-based statistical matching algorithm that designs well-matched pairs from observational data that resemble the covariate distributions of a target population, for instance, the target-RCT-eligible population in the RCT DUPLICATE initiative studies or a generic population of scientific interest. We demonstrate the usefulness of the method by revisiting the inconsistency regarding a cardioprotective effect of the hormone replacement therapy (HRT) in the Women's Health Initiative (WHI) clinical trial and corresponding observational study. We found that the discrepancy between the trial and observational study persisted in a design that adjusted for the difference in study populations' cardiovascular risk profile, but seemed to disappear in a study design that further adjusted for the difference in HRT initiation age and previous estrogen-plus-progestin use. The proposed method is integrated into the R package match2C.

Addressing Systematic Missing Data in the Context of Causally Interpretable Meta-analysis

Evidence synthesis involves drawing conclusions from trial samples that may differ from the target population of interest, and there is often heterogeneity among trials in sample characteristics, treatment implementation, study design, and assessment of covariates. Stitching together this patchwork of evidence requires subject-matter knowledge, a clearly defined target population, and guidance on how to weigh evidence from different trials. Transportability analysis has provided formal identifiability conditions required to make unbiased causal inference in the target population. In this manuscript, we review these conditions along with an additional assumption required to address systematic missing data. The identifiability conditions highlight the importance of accounting for differences in treatment effect modifiers between the populations underlying the trials and the target population. We perform simulations to evaluate the bias of conventional random effect models and multiply imputed estimates using the pooled trials sample and describe causal estimators that explicitly address trial-to-target differences in key covariates in the context of systematic missing data. Results indicate that the causal transportability estimators are unbiased when treatment effect modifiers are accounted for in the analyses. Results also highlight the importance of carefully evaluating identifiability conditions for each trial to reduce bias due to differences in participant characteristics between trials and the target population. Bias can be limited by adjusting for covariates that are strongly correlated with missing treatment effect modifiers, including data from trials that do not differ from the target on treatment modifiers, and removing trials that do differ from the target and did not assess a modifier.

Generalizability of the Necrotizing Enterocolitis Surgery Trial to the Target Population of Eligible Infants

OBJECTIVE

The objective of this study was to evaluate whether infants randomized in the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network Necrotizing Enterocolitis Surgery Trial differed from eligible infants and whether differences affected the generalizability of trial results.

STUDY DESIGN

Secondary analysis of infants enrolled in Necrotizing Enterocolitis Surgery Trial (born 2010-2017, with follow-up through 2019) at 20 US academic medical centers and an observational data set of eligible infants through 2013. Infants born ≤1000 g and diagnosed with necrotizing enterocolitis or spontaneous intestinal perforation requiring surgical intervention at ≤8 weeks were eligible. The target population included trial-eligible infants (randomized and nonrandomized) born during the first half of the study with available detailed preoperative data. Using model-based weighting methods, we estimated the effect of initial laparotomy vs peritoneal drain had the target population been randomized.

RESULTS

The trial included 308 randomized infants. The target population included 382 (156 randomized and 226 eligible, non-randomized) infants. Compared with the target population, fewer randomized infants had necrotizing enterocolitis (31% vs 47%) or died before discharge (27% vs 41%). However, incidence of the primary composite outcome, death or neurodevelopmental impairment, was similar (69% vs 72%). Effect estimates for initial laparotomy vs drain weighted to the target population were largely unchanged from the original trial after accounting for preoperative diagnosis of necrotizing enterocolitis (adjusted relative risk [95% CI]: 0.85 [0.71-1.03] in target population vs 0.81 [0.64-1.04] in trial) or spontaneous intestinal perforation (1.02 [0.79-1.30] vs 1.11 [0.95-1.31]).

CONCLUSION

Despite differences between randomized and eligible infants, estimated treatment effects in the trial and target population were similar, supporting the generalizability of trial results.

TRIAL REGISTRATION

ClinicalTrials.gov ID: NCT01029353.

A causal roadmap for generating high-quality real-world evidence

Increasing emphasis on the use of real-world evidence (RWE) to support clinical policy and regulatory decision-making has led to a proliferation of guidance, advice, and frameworks from regulatory agencies, academia, professional societies, and industry. A broad spectrum of studies use real-world data (RWD) to produce RWE, ranging from randomized trials with outcomes assessed using RWD to fully observational studies. Yet, many proposals for generating RWE lack sufficient detail, and many analyses of RWD suffer from implausible assumptions, other methodological flaws, or inappropriate interpretations. The Causal Roadmap is an explicit, itemized, iterative process that guides investigators to prespecify study design and analysis plans; it addresses a wide range of guidance within a single framework. By supporting the transparent evaluation of causal assumptions and facilitating objective comparisons of design and analysis choices based on prespecified criteria, the Roadmap can help investigators to evaluate the quality of evidence that a given study is likely to produce, specify a study to generate high-quality RWE, and communicate effectively with regulatory agencies and other stakeholders. This paper aims to disseminate and extend the Causal Roadmap framework for use by clinical and translational researchers; three companion papers demonstrate applications of the Causal Roadmap for specific use cases.

Reweighting estimators to extend the external validity of clinical trials: methodological considerations

Methods to extend the strong internal validity of randomized controlled trials to reliably estimate treatment effects in target populations are gaining attention. This paper enumerates steps recommended for undertaking such extended inference, discusses currently viable choices for each one, and provides recommendations. We demonstrate a complete extended inference from a clinical trial studying a pharmaceutical treatment for Alzheimer's disease (AD) to a realistic target population of European residents diagnosed with AD. This case study highlights approaches to overcoming practical difficulties and demonstrates limitations of reliably extending inference from a trial to a real-world population.

Adjusting for Underrepresentation Reveals Widespread Underestimation of Parkinson's Disease Symptom Burden

BACKGROUND

Clinical research is limited by underrepresentation, but the impact of underrepresentation on patient-reported outcomes in Parkinson's disease (PD) is unknown.

OBJECTIVES

To produce nationwide estimates of non-motor symptom (NMS) prevalence and PD-related quality of life (QOL) limitations while accounting for underrepresentation.

METHODS

We performed a cross-sectional analysis of data from the Fox Insight (FI) study, an ongoing prospective longitudinal study of persons with self-reported PD. Using epidemiologic literature and United States (US) Census Bureau, Medicare, and National Health and Aging Trends Study data, we simulated a "virtual census" of the PD population. To compare the PD census to the FI cohort, we used logistic regression to model the odds of study participation and calculate predicted probabilities of participation for inverse probability weighting.

RESULTS

There are an estimated 849,488 persons living with PD in the US. Compared to 22,465 eligible FI participants, non-participants are more likely to be older, female, and non-White; live in rural regions; have more severe PD; and have lower levels of education. When these predictors were incorporated into a multivariable regression model, predicted probability of participation was much higher for FI participants than non-participants, indicating a significant difference in the underlying populations (propensity score distance 2.62). Estimates of NMS prevalence and QOL limitation were greater when analyzed using inverse probability of participation weighting compared to unweighted means and frequencies.

CONCLUSIONS

PD-related morbidity may be underestimated because of underrepresentation, and inverse probability of participation weighting can be used to give greater weight to underrepresented groups and produce more generalizable estimates. © 2023 International Parkinson and Movement Disorder Society.

Estimating the area under the ROC curve when transporting a prediction model to a target population

We propose methods for estimating the area under the receiver operating characteristic (ROC) curve (AUC) of a prediction model in a target population that differs from the source population that provided the data used for original model development. If covariates that are associated with model performance, as measured by the AUC, have a different distribution in the source and target populations, then AUC estimators that only use data from the source population will not reflect model performance in the target population. Here, we provide identification results for the AUC in the target population when outcome and covariate data are available from the sample of the source population, but only covariate data are available from the sample of the target population. In this setting, we propose three estimators for the AUC in the target population and show that they are consistent and asymptotically normal. We evaluate the finite-sample performance of the estimators using simulations and use them to estimate the AUC in a nationally representative target population from the National Health and Nutrition Examination Survey for a lung cancer risk prediction model developed using source population data from the National Lung Screening Trial.

Evaluation of randomized controlled trials: a primer and tutorial for mental health researchers

BACKGROUND

Considered one of the highest levels of evidence, results of randomized controlled trials (RCTs) remain an essential building block in mental health research. They are frequently used to confirm that an intervention "works" and to guide treatment decisions. Given their importance in the field, it is concerning that the quality of many RCT evaluations in mental health research remains poor. Common errors range from inadequate missing data handling and inappropriate analyses (e.g., baseline randomization tests or analyses of within-group changes) to unduly interpretations of trial results and insufficient reporting. These deficiencies pose a threat to the robustness of mental health research and its impact on patient care. Many of these issues may be avoided in the future if mental health researchers are provided with a better understanding of what constitutes a high-quality RCT evaluation.

METHODS

In this primer article, we give an introduction to core concepts and caveats of clinical trial evaluations in mental health research. We also show how to implement current best practices using open-source statistical software.

RESULTS

Drawing on Rubin's potential outcome framework, we describe that RCTs put us in a privileged position to study causality by ensuring that the potential outcomes of the randomized groups become exchangeable. We discuss how missing data can threaten the validity of our results if dropouts systematically differ from non-dropouts, introduce trial estimands as a way to co-align analyses with the goals of the evaluation, and explain how to set up an appropriate analysis model to test the treatment effect at one or several assessment points. A novice-friendly tutorial is provided alongside this primer. It lays out concepts in greater detail and showcases how to implement techniques using the statistical software R, based on a real-world RCT dataset.

DISCUSSION

Many problems of RCTs already arise at the design stage, and we examine some avoidable and unavoidable "weak spots" of this design in mental health research. For instance, we discuss how lack of prospective registration can give way to issues like outcome switching and selective reporting, how allegiance biases can inflate effect estimates, review recommendations and challenges in blinding patients in mental health RCTs, and describe problems arising from underpowered trials. Lastly, we discuss why not all randomized trials necessarily have a limited external validity and examine how RCTs relate to ongoing efforts to personalize mental health care.

Elastic integrative analysis of randomised trial and real-world data for treatment heterogeneity estimation

We propose a test-based elastic integrative analysis of the randomised trial and real-world data to estimate treatment effect heterogeneity with a vector of known effect modifiers. When the real-world data are not subject to bias, our approach combines the trial and real-world data for efficient estimation. Utilising the trial design, we construct a test to decide whether or not to use real-world data. We characterise the asymptotic distribution of the test-based estimator under local alternatives. We provide a data-adaptive procedure to select the test threshold that promises the smallest mean square error and an elastic confidence interval with a good finite-sample coverage property.

Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population

Extending (i.e., generalizing or transporting) causal inferences from a randomized trial to a target population requires assumptions that randomized and nonrandomized individuals are exchangeable conditional on baseline covariates. These assumptions are made on the basis of background knowledge, which is often uncertain or controversial, and need to be subjected to sensitivity analysis. We present simple methods for sensitivity analyses that directly parameterize violations of the assumptions using bias functions and do not require detailed background knowledge about specific unknown or unmeasured determinants of the outcome or modifiers of the treatment effect. We show how the methods can be applied to non-nested trial designs, where the trial data are combined with a separately obtained sample of nonrandomized individuals, as well as to nested trial designs, where the trial is embedded within a cohort sampled from the target population.

Efficient and robust methods for causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a target population

We present methods for causally interpretable meta-analyses that combine information from multiple randomized trials to draw causal inferences for a target population of substantive interest. We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results for transporting causal inferences from a collection of independent randomized trials to a new target population in which experimental data may not be available. We propose an estimator for the potential outcome mean in the target population under each treatment studied in the trials. The estimator uses covariate, treatment, and outcome data from the collection of trials, but only covariate data from the target population sample. We show that it is doubly robust in the sense that it is consistent and asymptotically normal when at least one of the models it relies on is correctly specified. We study the finite sample properties of the estimator in simulation studies and demonstrate its implementation using data from a multicenter randomized trial.

Improving trial generalizability using observational studies

Complementary features of randomized controlled trials (RCTs) and observational studies (OSs) can be used jointly to estimate the average treatment effect of a target population. We propose a calibration weighting estimator that enforces the covariate balance between the RCT and OS, therefore improving the trial-based estimator's generalizability. Exploiting semiparametric efficiency theory, we propose a doubly robust augmented calibration weighting estimator that achieves the efficiency bound derived under the identification assumptions. A nonparametric sieve method is provided as an alternative to the parametric approach, which enables the robust approximation of the nuisance functions and data-adaptive selection of outcome predictors for calibration. We establish asymptotic results and confirm the finite sample performances of the proposed estimators by simulation experiments and an application on the estimation of the treatment effect of adjuvant chemotherapy for early-stage non-small-cell lung patients after surgery.

John J, Kono N, Henderson VW, Mack WJ. Generalizability of cognitive results from clinical trial participants to an older adult population: Addressing external validity

Introduction

Study inclusion criteria and recruitment practices limit the generalizability of randomized-controlled trial (RCT) results. Statistical modeling could enhance generalizability of outcomes. To illustrate this, the cognition-depression relationship was assessed with and without adjustment relative to the target population of older women.

Methods

Randomized participants from four RCTs and non-randomized participants from two cohorts were included in this study. Prediction models estimated probability of being randomized into trials from target populations. These probabilities were used for inverse odds weighting relative to target populations. Weighted linear regression was used to assess the depression-cognition relationship.

Results

There was no depression-cognition relationship in the combined randomized sample. After applying weights relative to a representative cohort, negative relationships were observed. After applying weights relative to a non-representative cohort, bias of estimates increased.

Discussion

Quantitative approaches to transportability using representative samples may explain the absence of a-priori established relationships in RCTs.

Improving efficiency of inference in clinical trials with external control data

Suppose we are interested in the effect of a treatment in a clinical trial. The efficiency of inference may be limited due to small sample size. However, external control data are often available from historical studies. Motivated by an application to Helicobacter pylori infection, we show how to borrow strength from such data to improve efficiency of inference in the clinical trial. Under an exchangeability assumption about the potential outcome mean, we show that the semiparametric efficiency bound for estimating the average treatment effect can be reduced by incorporating both the clinical trial data and external controls. We then derive a doubly robust and locally efficient estimator. The improvement in efficiency is prominent especially when the external control data set has a large sample size and small variability. Our method allows for a relaxed overlap assumption, and we illustrate with the case where the clinical trial only contains a treated group. We also develop doubly robust and locally efficient approaches that extrapolate the causal effect in the clinical trial to the external population and the overall population. Our results also offer a meaningful implication for trial design and data collection. We evaluate the finite-sample performance of the proposed estimators via simulation. In the Helicobacter pylori infection application, our approach shows that the combination treatment has potential efficacy advantages over the triple therapy.

Transporting a Prediction Model for Use in a New Target Population

We considered methods for transporting a prediction model for use in a new target population, both when outcome and covariate data for model development are available from a source population that has a different covariate distribution compared with the target population and when covariate data (but not outcome data) are available from the target population. We discuss how to tailor the prediction model to account for differences in the data distribution between the source population and the target population. We also discuss how to assess the model's performance (e.g., by estimating the mean squared prediction error) in the target population. We provide identifiability results for measures of model performance in the target population for a potentially misspecified prediction model under a sampling design where the source and the target population samples are obtained separately. We introduce the concept of prediction error modifiers that can be used to reason about tailoring measures of model performance to the target population. We illustrate the methods in simulated data and apply them to transport a prediction model for lung cancer diagnosis from the National Lung Screening Trial to the nationally representative target population of trial-eligible individuals in the National Health and Nutrition Examination Survey.

Regression-based estimation of heterogeneous treatment effects when extending inferences from a randomized trial to a target population

Most work on extending (generalizing or transporting) inferences from a randomized trial to a target population has focused on estimating average treatment effects (i.e., averaged over the target population's covariate distribution). Yet, in the presence of strong effect modification by baseline covariates, the average treatment effect in the target population may be less relevant for guiding treatment decisions. Instead, the conditional average treatment effect (CATE) as a function of key effect modifiers may be a more useful estimand. Recent work on estimating target population CATEs using baseline covariate, treatment, and outcome data from the trial and covariate data from the target population only allows for the examination of heterogeneity over distinct subgroups. We describe flexible pseudo-outcome regression modeling methods for estimating target population CATEs conditional on discrete or continuous baseline covariates when the trial is embedded in a sample from the target population (i.e., in nested trial designs). We construct pointwise confidence intervals for the CATE at a specific value of the effect modifiers and uniform confidence bands for the CATE function. Last, we illustrate the methods using data from the Coronary Artery Surgery Study (CASS) to estimate CATEs given history of myocardial infarction and baseline ejection fraction value in the target population of all trial-eligible patients with stable ischemic heart disease.

Data Integration in Causal Inference

Integrating data from multiple heterogeneous sources has become increasingly popular to achieve a large sample size and diverse study population. This paper reviews development in causal inference methods that combines multiple datasets collected by potentially different designs from potentially heterogeneous populations. We summarize recent advances on combining randomized clinical trial with external information from observational studies or historical controls, combining samples when no single sample has all relevant variables with application to two-sample Mendelian randomization, distributed data setting under privacy concerns for comparative effectiveness and safety research using real-world data, Bayesian causal inference, and causal discovery methods.

Eligibility and Implementation of Rivaroxaban for Secondary Prevention of Atherothrombosis in Clinical Practice-Insights From the CANHEART Study

Background The COMPASS (Cardiovascular Outcomes for People Using Anticoagulation Strategies) trial decreased major adverse cardiovascular events with very low-dose rivaroxaban and aspirin in patients with coronary artery disease and peripheral artery disease. We examined the eligibility and potential real-world impact of this strategy on the COMPASS-eligible population. Methods and Results COMPASS eligibility criteria were applied to the CANHEART (Cardiovascular Health in Ambulatory Care Research Team) registry, a population-based cohort of Ontario adults. We compared 5-year major adverse cardiovascular events and major bleeding rates stratified by COMPASS eligibility and by clinical risk factors. We applied COMPASS trial rivaroxaban/aspirin arm hazard ratios to estimate the potential impact on the COMPASS-eligible cohort. Among 362 797 patients with coronary artery disease or peripheral artery disease, 38% were deemed eligible, 47% ineligible, and 15% indeterminate. Among eligible patients, a greater number of risk factors was associated with higher rates of cardiovascular outcomes, whereas bleeding rates increased minimally. Over 5 years, applying COMPASS treatment effects to eligible patients resulted in a 2.4% absolute risk reduction of major adverse cardiovascular events and a number needed to treat of 42, and a 1.3% absolute risk increase of major bleeding and number needed to harm (NNH) of 77. Those with at least 2 risk factors had a 3.0% absolute risk reduction of major adverse cardiovascular events (number needed to treat =34) and a 1.6% absolute risk increase of major bleeding (number needed to harm =61). Conclusions Implementation of very-low-dose rivaroxaban therapy would potentially impact ≈$$ \approx $$2 in 5 patients with atherosclerotic disease in Ontario. Eligible individuals with ≥$$ \ge $$2 comorbidities represent a high-risk subgroup that may derive the greatest benefit-to-risk ratio. Selection of patients with high-risk predisposing factors appears appropriate in routine practice.

Doubly robust estimators for generalizing treatment effects on survival outcomes from randomized controlled trials to a target population

In the presence of heterogeneity between the randomized controlled trial (RCT) participants and the target population, evaluating the treatment effect solely based on the RCT often leads to biased quantification of the real-world treatment effect. To address the problem of lack of generalizability for the treatment effect estimated by the RCT sample, we leverage observational studies with large samples that are representative of the target population. This article concerns evaluating treatment effects on survival outcomes for a target population and considers a broad class of estimands that are functionals of treatment-specific survival functions, including differences in survival probability and restricted mean survival times. Motivated by two intuitive but distinct approaches, i.e., imputation based on survival outcome regression and weighting based on inverse probability of sampling, censoring, and treatment assignment, we propose a semiparametric estimator through the guidance of the efficient influence function. The proposed estimator is doubly robust in the sense that it is consistent for the target population estimands if either the survival model or the weighting model is correctly specified and is locally efficient when both are correct. In addition, as an alternative to parametric estimation, we employ the nonparametric method of sieves for flexible and robust estimation of the nuisance functions and show that the resulting estimator retains the root-n consistency and efficiency, the so-called rate-double robustness. Simulation studies confirm the theoretical properties of the proposed estimator and show that it outperforms competitors. We apply the proposed method to estimate the effect of adjuvant chemotherapy on survival in patients with early-stage resected non-small cell lung cancer.

Transfer learning of individualized treatment rules from experimental to real-world data

Individualized treatment effect lies at the heart of precision medicine. Interpretable individualized treatment rules (ITRs) are desirable for clinicians or policymakers due to their intuitive appeal and transparency. The gold-standard approach to estimating the ITRs is randomized experiments, where subjects are randomized to different treatment groups and the confounding bias is minimized to the extent possible. However, experimental studies are limited in external validity because of their selection restrictions, and therefore the underlying study population is not representative of the target real-world population. Conventional learning methods of optimal interpretable ITRs for a target population based only on experimental data are biased. On the other hand, real-world data (RWD) are becoming popular and provide a representative sample of the target population. To learn the generalizable optimal interpretable ITRs, we propose an integrative transfer learning method based on weighting schemes to calibrate the covariate distribution of the experiment to that of the RWD. Theoretically, we establish the risk consistency for the proposed ITR estimator. Empirically, we evaluate the finite-sample performance of the transfer learner through simulations and apply it to a real data application of a job training program.

A Framework for Descriptive Epidemiology

In this paper, we propose a framework for thinking through the design and conduct of descriptive epidemiologic studies. A well-defined descriptive question aims to quantify and characterize some feature of the health of a population and must clearly state: 1) the target population, characterized by person and place, and anchored in time; 2) the outcome, event, or health state or characteristic; and 3) the measure of occurrence that will be used to summarize the outcome (e.g., incidence, prevalence, average time to event, etc.). Additionally, 4) any auxiliary variables will be prespecified and their roles as stratification factors (to characterize the outcome distribution) or nuisance variables (to be standardized over) will be stated. We illustrate application of this framework to describe the prevalence of viral suppression on December 31, 2019, among people living with human immunodeficiency virus (HIV) who had been linked to HIV care in the United States. Application of this framework highlights biases that may arise from missing data, especially 1) differences between the target population and the analytical sample; 2) measurement error; 3) competing events, late entries, loss to follow-up, and inappropriate interpretation of the chosen measure of outcome occurrence; and 4) inappropriate adjustment.

A Causal Framework for Making Individualized Treatment Decisions in Oncology

Simple Summary

Physicians routinely make individualized treatment decisions by accounting for the joint effects of patient prognostic covariates and treatments on clinical outcomes. Ideally, this is performed using historical randomized clinical trial (RCT) data. Randomization ensures that unbiased estimates of causal treatment effect parameters can be obtained from the historical RCT data and used to predict each new patient’s outcome based on the joint effect of their baseline covariates and each treatment being considered. However, this process becomes problematic if a patient seen in the clinic is very different from the patients who were enrolled in the RCT. That is, if a new patient does not satisfy the entry criteria of the RCT, then the patient does not belong to the population represented by the patients who were studied in the RCT. In such settings, it still may be possible to utilize the RCT data to help choose a new patient’s treatment. This may be achieved by combining the RCT data with data from other clinical trials, or possibly preclinical experiments, and using the combined dataset to predict the patient’s expected outcome for each treatment being considered. In such settings, combining data from multiple sources in a way that is statistically reliable is not entirely straightforward, and correctly identifying and estimating the effects of treatments and patient covariates on clinical outcomes can be complex. Causal diagrams provide a rational basis to guide this process. The first step is to construct a causal diagram that reflects the plausible relationships between treatment variables, patient covariates, and clinical outcomes. If the diagram is correct, it can be used to determine what additional data may be needed, how to combine data from multiple sources, how to formulate a statistical model for clinical outcomes as a function of treatment and covariates, and how to compute an unbiased treatment effect estimate for each new patient. We use adjuvant therapy of renal cell carcinoma to illustrate how causal diagrams may be used to guide these steps.

Abstract

We discuss how causal diagrams can be used by clinicians to make better individualized treatment decisions. Causal diagrams can distinguish between settings where clinical decisions can rely on a conventional additive regression model fit to data from a historical randomized clinical trial (RCT) to estimate treatment effects and settings where a different approach is needed. This may be because a new patient does not meet the RCT’s entry criteria, or a treatment’s effect is modified by biomarkers or other variables that act as mediators between treatment and outcome. In some settings, the problem can be addressed simply by including treatment–covariate interaction terms in the statistical regression model used to analyze the RCT dataset. However, if the RCT entry criteria exclude a new patient seen in the clinic, it may be necessary to combine the RCT data with external data from other RCTs, single-arm trials, or preclinical experiments evaluating biological treatment effects. For example, external data may show that treatment effects differ between histological subgroups not recorded in an RCT. A causal diagram may be used to decide whether external observational or experimental data should be obtained and combined with RCT data to compute statistical estimates for making individualized treatment decisions. We use adjuvant treatment of renal cell carcinoma as our motivating example to illustrate how to construct causal diagrams and apply them to guide clinical decisions.

Profile Matching for the Generalization and Personalization of Causal Inferences

We introduce profile matching, a multivariate matching method for randomized experiments and observational studies that finds the largest possible unweighted samples across multiple treatment groups that are balanced relative to a covariate profile. This covariate profile can represent a specific population or a target individual, facilitating the generalization and personalization of causal inferences. For generalization, because the profile often amounts to summary statistics for a target population, profile matching does not always require accessing individual-level data, which may be unavailable for confidentiality reasons. For personalization, the profile comprises the characteristics of a single individual. Profile matching achieves covariate balance by construction, but unlike existing approaches to matching, it does not require specifying a matching ratio, as this is implicitly optimized for the data. The method can also be used for the selection of units for study follow-up, and it readily applies to multivalued treatments with many treatment categories. We evaluate the performance of profile matching in a simulation study of the generalization of a randomized trial to a target population. We further illustrate this method in an exploratory observational study of the relationship between opioid use and mental health outcomes. We analyze these relationships for three covariate profiles representing: (i) sexual minorities, (ii) the Appalachian United States, and (iii) the characteristics of a hypothetical vulnerable patient. The method can be implemented via the new function profmatch in the designmatch package for R, for which we provide a step-by-step tutorial.

Using Numerical Methods to Design Simulations: Revisiting the Balancing Intercept

In this paper, we consider methods for generating draws of a binary random variable whose expectation conditional on covariates follows a logistic regression model with known covariate coefficients. We examine approximations for finding a "balancing intercept," that is, a value for the intercept of the logistic model that leads to a desired marginal expectation for the binary random variable. We show that a recently proposed analytical approximation can produce inaccurate results, especially when targeting more extreme marginal expectations or when the linear predictor of the regression model has high variance. We then formulate the balancing intercept as a solution to an integral equation, implement a numerical approximation for solving the equation based on Monte Carlo methods, and show that the approximation works well in practice. Our approach to the basic problem of the balancing intercept provides an example of a broadly applicable strategy for formulating and solving problems that arise in the design of simulation studies used to evaluate or teach epidemiologic methods.

Causally Interpretable Meta-analysis: Application in Adolescent HIV Prevention

Endowing meta-analytic results with a causal interpretation is challenging when there are differences in the distribution of effect modifiers among the populations underlying the included trials and the target population where the results of the meta-analysis will be applied. Recent work on transportability methods has described identifiability conditions under which the collection of randomized trials in a meta-analysis can be used to draw causal inferences about the target population. When the conditions hold, the methods enable estimation of causal quantities such as the average treatment effect and conditional average treatment effect in target populations that differ from the populations underlying the trial samples. The methods also facilitate comparison of treatments not directly compared in a head-to-head trial and assessment of comparative effectiveness within subgroups of the target population. We briefly describe these methods and present a worked example using individual participant data from three HIV prevention trials among adolescents in mental health care. We describe practical challenges in defining the target population, obtaining individual participant data from included trials and a sample of the target population, and addressing systematic missing data across datasets. When fully realized, methods for causally interpretable meta-analysis can provide decision-makers valid estimates of how treatments will work in target populations of substantive interest as well as in subgroups of these populations.

Use of Linked Databases for Improved Confounding Control: Considerations for Potential Selection Bias

Pharmacoepidemiologic studies are increasingly conducted within linked databases, often to obtain richer confounder data. However, the potential for selection bias is frequently overlooked when linked data is available only for a subset of patients. We highlight the importance of accounting for potential selection bias by evaluating the association between antipsychotics and type 2 diabetes in youths within a claims database linked to a smaller laboratory database. We used inverse probability of treatment weights (IPTW) to control for confounding. In analyses restricted to the linked cohorts, we applied inverse probability of selection weights (IPSW) to create a population representative of the full cohort. We used pooled logistic regression weighted by IPTW only or IPTW and IPSW to estimate treatment effects. Metabolic conditions were more prevalent in linked cohorts compared with the full cohort. Within the full cohort, the confounding-adjusted hazard ratio was 2.26 (95% CI: 2.07, 2.49) comparing initiation of antipsychotics with initiation of control medications. Within the linked cohorts, a different magnitude of association was obtained without adjustment for selection, whereas applying IPSW resulted in point estimates similar to the full cohort's (e.g., an adjusted hazard ratio of 1.63 became 2.12). Linked database studies may generate biased estimates without proper adjustment for potential selection bias.

Invited Commentary: Estimation and Bounds Under Data Fusion

In their recent article, Ogburn et al. (Am J Epidemiol. 2021;190(6):1142-1147) raised a cautionary tale for epidemiologic data fusion: Bias may occur if a variable that is completely missing in the primary data set is imputed according to a regression model estimated from an auxiliary data set. However, in some specific settings, a solution may exist. Focusing on a linear outcome regression model with a missing covariate, we show that the bias can be eliminated if the underlying imputation model for the missing covariate is nonlinear in the common variables measured in both data sets. Otherwise, we describe 2 alternative approaches existing in the data fusion literature that could partially resolve this issue: One fits the outcome model by leveraging an additional validation data set containing joint observations of the outcome and the missing covariate, and the other offers informative bounds for the outcome regression coefficients without using validation data. We justify these 3 methods in a linear outcome model and briefly discuss their extension to general settings.

Estimation of DAPT Study Treatment Effects in Contemporary Clinical Practice: Findings From the EXTEND-DAPT Study

BACKGROUND

Differences in patient characteristics, changes in treatment algorithms, and advances in medical technology could each influence the applicability of older randomized trial results to contemporary clinical practice. The DAPT Study (Dual Antiplatelet Therapy) found that longer-duration DAPT decreased ischemic events at the expense of greater bleeding, but subsequent evolution in stent technology and clinical practice may attenuate the benefit of prolonged DAPT in a contemporary population. We evaluated whether the DAPT Study population is different from a contemporary population of US patients receiving percutaneous coronary intervention and estimated the treatment effect of extended-duration antiplatelet therapy after percutaneous coronary intervention in this more contemporary cohort.

METHODS

We compared the characteristics of drug-eluting stent-treated patients randomly assigned in the DAPT Study to a sample of more contemporary drug-eluting stent-treated patients in the National Cardiovascular Data Registry CathPCI Registry from July 2016 to June 2017. After linking trial and registry data, we used inverse-odds of trial participation weighting to account for patient and procedural characteristics and estimated a contemporary real-world treatment effect of 30 versus 12 months of DAPT after coronary stent procedures.

RESULTS

The US drug-eluting stent-treated trial cohort included 8864 DAPT Study patients, and the registry cohort included 568 540 patients. Compared with the trial population, registry patients had more comorbidities and were more likely to present with myocardial infarction and receive 2nd-generation drug-eluting stents. After reweighting trial results to represent the registry population, there was no longer a significant effect of prolonged DAPT on reducing stent thrombosis (reweighted treatment effect: -0.40 [95% CI, -0.99% to 0.15%]), major adverse cardiac and cerebrovascular events (reweighted treatment effect, -0.52 [95% CI, -2.62% to 1.03%]), or myocardial infarction (reweighted treatment effect, -0.97% [95% CI, -2.75% to 0.18%]), but the increase in bleeding with prolonged DAPT persisted (reweighted treatment effect, 2.42% [95% CI, 0.79% to 3.91%]).

CONCLUSIONS

The differences between the patients and devices used in contemporary clinical practice compared with the DAPT Study were associated with the attenuation of benefits and greater harms attributable to prolonged DAPT duration. These findings limit the applicability of the average treatment effects from the DAPT Study in modern clinical practice.

Generalizability of heterogeneous treatment effects based on causal forests applied to two randomized clinical trials of intensive glycemic control

Purpose Machine learning is an attractive tool for identifying heterogeneous treatment effects (HTE) of interventions but generalizability of machine learning derived HTE remains unclear. We examined generalizability of HTE detected using causal forests in two similarly designed randomized trials in type II diabetes patients. Methods We evaluated published HTE of intensive versus standard glycemic control on all-cause mortality from the Action to Control Cardiovascular Risk in Diabetes study (ACCORD) in a second trial, the Veterans Affairs Diabetes Trial (VADT). We then applied causal forests to VADT, ACCORD, and pooled data from both studies and compared variable importance and subgroup effects across samples. Results HTE in ACCORD did not replicate in similar subgroups in VADT, but variable importance was correlated between VADT and ACCORD (Kendall's tau-b 0.75). Applying causal forests to pooled individual-level data yielded seven subgroups with similar HTE across both studies, ranging from risk difference of all-cause mortality of -3.9% (95% CI -7.0, -0.8) to 4.7% (95% CI 1.8, 7.5). Conclusions Machine learning detection of HTE subgroups from randomized trials may not generalize across study samples even when variable importance is correlated. Pooling individual-level data may overcome differences in study populations and/or differences in interventions that limit HTE generalizability.

A note on semiparametric efficient generalization of causal effects from randomized trials to target populations

When effect modifiers influence the decision to participate in randomized trials, generalizing causal effect estimates to an external target population requires the knowledge of two scores - the propensity score for receiving treatment and the sampling score for trial participation. While the former score is known due to randomization, the latter score is usually unknown and estimated from data. Under unconfounded trial participation, we characterize the asymptotic efficiency bounds for estimating two causal estimands - the population average treatment effect and the average treatment effect among the non-participants - and examine the role of the scores. We also study semiparametric efficient estimators that directly balance the weighted trial sample toward the target population, and illustrate their operating characteristics via simulations.

Applicability of Transcatheter Aortic Valve Replacement Trials to Real-World Clinical Practice: Findings From EXTEND-CoreValve

OBJECTIVES

The aim of this study was to examine the applicability of pivotal transcatheter aortic valve replacement (TAVR) trials to the real-world population of Medicare patients undergoing TAVR.

BACKGROUND

It is unclear whether randomized controlled trial results of novel cardiovascular devices apply to patients encountered in clinical practice.

METHODS

Characteristics of patients enrolled in the U.S. CoreValve pivotal trials were compared with those of the population of Medicare beneficiaries who underwent TAVR in U.S. clinical practice between November 2, 2011, and December 31, 2017. Inverse probability weighting was used to reweight the trial cohort on the basis of Medicare patient characteristics, and a "real-world" treatment effect was estimated.

RESULTS

A total of 2,026 patients underwent TAVR in the U.S. CoreValve pivotal trials, and 135,112 patients underwent TAVR in the Medicare cohort. Trial patients were mostly similar to real-world patients at baseline, though trial patients were more likely to have hypertension (50% vs 39%) and coagulopathy (25% vs 17%), whereas real-world patients were more likely to have congestive heart failure (75% vs 68%) and frailty. The estimated real-world treatment effect of TAVR was an 11.4% absolute reduction in death or stroke (95% CI: 7.50%-14.92%) and an 8.7% absolute reduction in death (95% CI: 5.20%-12.32%) at 1 year with TAVR compared with conventional therapy (surgical aortic valve replacement for intermediate- and high-risk patients and medical therapy for extreme-risk patients).

CONCLUSIONS

The trial and real-world populations were mostly similar, with some notable differences. Nevertheless, the extrapolated real-world treatment effect was at least as high as the observed trial treatment effect, suggesting that the absolute benefit of TAVR in clinical trials is similar to the benefit of TAVR in the U.S. real-world setting.