Study Designs for Extending Causal Inferences From a Randomized Trial to a Target Population

Cluster Randomized Trials Designed to Support Generalizable Inferences

When planning a cluster randomized trial, evaluators often have access to an enumerated cohort representing the target population of clusters. Practicalities of conducting the trial, such as the need to oversample clusters with certain characteristics in order to improve trial economy or support inferences about subgroups of clusters, may preclude simple random sampling from the cohort into the trial, and thus interfere with the goal of producing generalizable inferences about the target population. We describe a nested trial design where the randomized clusters are embedded within a cohort of trial-eligible clusters from the target population and where clusters are selected for inclusion in the trial with known sampling probabilities that may depend on cluster characteristics (e.g., allowing clusters to be chosen to facilitate trial conduct or to examine hypotheses related to their characteristics). We develop and evaluate methods for analyzing data from this design to generalize causal inferences to the target population underlying the cohort. We present identification and estimation results for the expectation of the average potential outcome and for the average treatment effect, in the entire target population of clusters and in its non-randomized subset. In simulation studies, we show that all the estimators have low bias but markedly different precision. Cluster randomized trials where clusters are selected for inclusion with known sampling probabilities that depend on cluster characteristics, combined with efficient estimation methods, can precisely quantify treatment effects in the target population, while addressing objectives of trial conduct that require oversampling clusters on the basis of their characteristics.

Identifying target populations to align with decision-makers' needs

Randomized trials estimate the average treatment effect within individuals who are eligible, invited, and agree to enroll. However, decision-makers often require evidence that extends beyond the trial's enrolled population to inform policy or actions for their specific target population. Each decision-maker has distinct target populations, the composition of which may not often align with that of the trial population. As researchers, we should identify a decision-maker for whom we aim to generate evidence early in the research process. We can then specify a target population of their interest and determine if a policy or action can be informed using results from a trial alone, or if additional complementary real-world data and analysis are required. In this commentary, we outline 5 key groupings of decision-makers: policymakers, payers, purchasers, providers, and patients. We then specify relevant target populations for decision-makers interested in the effectiveness of beta-blockers after a myocardial infarction with preserved ejection fraction. Finally, we summarize the scenarios in which results from a randomized trial may or may not apply to these target populations and suggest relevant analytic approaches that can generate evidence to better align with a decision-maker's needs. This article is part of a Special Collection on Pharmacoepidemiology.

Use of transportability methods for real-world evidence generation: a review of current applications

Aim: To evaluate how transportability methods are currently used for real-world evidence (RWE) generation to inform good practices and support adoption and acceptance of these methods in the RWE context. Methods: We conducted a targeted literature review to identify studies that transported an effect estimate of the clinical effectiveness or safety of a biomedical exposure to a target real-world population. Records were identified from PubMed-indexed articles published any time before 25 July 2023 (inclusive). Two reviewers screened abstracts/titles and reviewed the full text of candidate studies to identify the final set of articles. Data on the therapeutic area, exposure(s), outcome(s), original and target populations and details of the transportability analysis (e.g., analytic method used, estimate transported, stated assumptions) were abstracted from each article. Results: Of 458 unique records identified, six were retained in the final review. Articles were published during 2021-2023, focused on the US/Canada context, and covered a range of therapeutic areas. Four studies transported an RCT effect estimate, while two transported effect estimates derived from real-world data. Almost all articles used weighting methods to transport estimates. Two studies discussed all transportability assumptions, and one evaluated the likelihood of meeting all assumptions and the impact of potential violations. Conclusion: The use of transportability methods for RWE generation is an emerging and promising area of research to address evidence gaps in settings with limited data and infrastructure. More transparent and rigorous reporting of methods, assumptions and limitations may increase the use and acceptability of transportability for producing robust evidence on treatment effectiveness and safety.

Methods for Extending Inferences From Observational Studies: Considering Causal Structures, Identification Assumptions, and Estimators

Most prior work in quantitative approaches to generalizability and transportability emphasizes extending causal effect estimates from randomized trials to target populations. Extending findings from observational studies is also of scientific interest, and identifiability assumptions and estimation methods differ from randomized settings when there is selection on both the exposure and exposure-outcome mediators in combination with exposure-outcome confounders (and both confounders and mediators can modify exposure-outcome effects). We argue that this causal structure is common in observational studies, particularly in the field of life course epidemiology, for example, when extending estimates of the effect of an early-life exposure on a later-life outcome from a cohort enrolled in midlife or late life. We describe identifiability assumptions and identification using observed data in such settings, highlighting differences from work extending findings from randomized trials. We describe statistical methods, including weighting, outcome modeling, and doubly robust approaches, to estimate potential outcome means and average treatment effects in the target population and illustrate performance of the methods in a simulation study. We show that in the presence of selection into the study sample on both exposure and confounders, estimators must be able to address confounding in the target population. When there is also selection on mediators of the exposure-outcome relationship, estimators need to be able to use different sets of variables to account for selection (including the mediator), and confounding. We discuss conceptual implications of our results as well as highlight unresolved practical questions for applied work to extend findings from observational studies to target populations.

Learning about treatment effects in a new target population under transportability assumptions for relative effect measures

Investigators often believe that relative effect measures conditional on covariates, such as risk ratios and mean ratios, are "transportable" across populations. Here, we examine the identification of causal effects in a target population using an assumption that conditional relative effect measures are transportable from a trial to the target population. We show that transportability for relative effect measures is largely incompatible with transportability for difference effect measures, unless the treatment has no effect on average or one is willing to make even stronger transportability assumptions that imply the transportability of both relative and difference effect measures. We then describe how marginal (population-averaged) causal estimands in a target population can be identified under the assumption of transportability of relative effect measures, when we are interested in the effectiveness of a new experimental treatment in a target population where the only treatment in use is the control treatment evaluated in the trial. We extend these results to consider cases where the control treatment evaluated in the trial is only one of the treatments in use in the target population, under an additional partial exchangeability assumption in the target population (i.e., an assumption of no unmeasured confounding in the target population with respect to potential outcomes under the control treatment in the trial). We also develop identification results that allow for the covariates needed for transportability of relative effect measures to be only a small subset of the covariates needed to control confounding in the target population. Last, we propose estimators that can be easily implemented in standard statistical software and illustrate their use using data from a comprehensive cohort study of stable ischemic heart disease.

Variable selection when estimating effects in external target populations

External validity is an important part of epidemiologic research. To validly estimate effects in specific external target populations using a chosen effect measure (ie, "transport"), some methods require that one account for all effect measure modifiers (EMMs). However, little is known about how including other variables that are not EMMs (ie, non-EMMs) in adjustment sets affects estimates. Using simulations, we evaluated how inclusion of non-EMMs affected estimation of the transported risk difference (RD) by assessing the impacts of covariates that (1) differ (or not) between the trial and the target, (2) are associated with the outcome (or not), and (3) modify the RD (or not). We assessed variation and bias when covariates with each possible combination of these factors were used to transport RDs using outcome modeling or inverse odds weighting. Inclusion of variables that differed in distribution between the populations but were non-EMMs reduced precision, regardless of whether they were associated with the outcome. However, non-EMMs associated with selection did not amplify bias resulting from omission of necessary EMMs. Including all variables associated with the outcome may result in unnecessarily imprecise estimates when estimating treatment effects in external target populations.

Standardizing to specific target populations in distributed networks and multisite pharmacoepidemiologic studies

Distributed network studies and multisite studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multisite studies may increase interpretability and improve precision. We simulated a 4-site study, standardized each site using inverse odds weights (IOWs) to resemble the 3 smallest sites or the smallest site, estimated IOW-weighted risk differences (RDs), and combined estimates with inverse variance weights (IVWs). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of 1 site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOWs reduced differences between estimates and increased precision when targeting the 3 smallest sites or the smallest site. In the CPRD Aurum study, the IOW + IVW estimate was also more precise (smallest region: RD = 5.41% [95% CI, 1.03-9.79]; IOW + IVW estimate: RD = 3.25% [95% CI, 3.07-3.43]). When performing pharmacoepidemiologic research in distributed networks or multisite studies in the presence of EMMs, designation of target populations has the potential to improve estimate precision and interpretability. This article is part of a Special Collection on Pharmacoepidemiology.

Generalizability of Randomized Clinical Trial Outcomes for Diabetes Control Resulting From Bariatric Surgery

Objective

To assess the external validity of randomized controlled trials (RCTs) of bariatric surgical treatment on diabetes control.

Background

Multisite RCTs provide the strongest evidence supporting clinical treatments and have the greatest internal validity. However, characteristics of trial participants may not be representative of patients receiving treatment in the real world. There is a need to assess how the results of RCTs generalize to all contemporary patient populations undergoing treatments.

Methods

All patients undergoing sleeve gastrectomy at University of California Los Angeles (UCLA) between January 8, 2018 and May 19, 2023 had their baseline characteristics, weight change, and diabetes control compared with those enrolled in the surgical treatment and medications potentially eradicate diabetes efficiently (STAMPEDE) and diabetes surgery study (DSS) RCTs of bariatric surgery's effect on diabetes control. Weight loss and diabetes control were compared between UCLA patients who did and did not fit the entry criteria for these RCTs.

Results

Only 65 (17%) of 387 patients with diabetes fulfilled the eligibility criteria for STAMPEDE, and 29 (7.5%) fulfilled the criteria for DSS due to being older, having higher body mass index, and lower HbA1c. UCLA patients experienced slightly less weight loss than patients in the RCTs but had similar diabetes control. The 313 (81%) patients not eligible for study entry into either RCT had similar long-term diabetes control as those who were eligible for the RCTs.

Conclusions

Even though only a very small proportion of patients undergoing bariatric surgery met the eligibility criteria for the 2 major RCTs, most patients in this contemporary cohort had similar outcomes. Diabetes outcomes from STAMPEDE and DSS generalize to most patients undergoing bariatric surgery for diabetes control.

Invited Commentary: Combining Information to Answer Epidemiologic Questions About a Target Population

Epidemiologists are attempting to address research questions of increasing complexity by developing novel methods for combining information from diverse sources. Cole et al. (Am J Epidemiol. 2023;192(3)467-474) provide 2 examples of the process of combining information to draw inferences about a population proportion. In this commentary, we consider combining information to learn about a target population as an epidemiologic activity and distinguish it from more conventional meta-analyses. We examine possible rationales for combining information and discuss broad methodological considerations, with an emphasis on study design, assumptions, and sources of uncertainty.

Selection Bias in Health Research: Quantifying, Eliminating, or Exacerbating Health Disparities?

Purpose of review

To summarize recent literature on selection bias in disparities research addressing either descriptive or causal questions, with examples from dementia research.

Recent findings

Defining a clear estimand, including the target population, is essential to assess whether generalizability bias or collider-stratification bias are threats to inferences. Selection bias in disparities research can result from sampling strategies, differential inclusion pipelines, loss to follow-up, and competing events. If competing events occur, several potentially relevant estimands can be estimated under different assumptions, with different interpretations. The apparent magnitude of a disparity can differ substantially based on the chosen estimand. Both randomized and observational studies may misrepresent health disparities or heterogeneity in treatment effects if they are not based on a known sampling scheme.

Conclusion

Researchers have recently made substantial progress in conceptualization and methods related to selection bias. This progress will improve the relevance of both descriptive and causal health disparities research.

Visualizing External Validity: Graphical Displays to Inform the Extension of Treatment Effects from Trials to Clinical Practice

BACKGROUND

In the presence of effect measure modification, estimates of treatment effects from randomized controlled trials may not be valid in clinical practice settings. The development and application of quantitative approaches for extending treatment effects from trials to clinical practice settings is an active area of research.

METHODS

In this article, we provide researchers with a practical roadmap and four visualizations to assist in variable selection for models to extend treatment effects observed in trials to clinical practice settings and to assess model specification and performance. We apply this roadmap and visualizations to an example extending the effects of adjuvant chemotherapy (5-fluorouracil vs. plus oxaliplatin) for colon cancer from a trial population to a population of individuals treated in community oncology practices in the United States.

RESULTS

The first visualization screens for potential effect measure modifiers to include in models extending trial treatment effects to clinical practice populations. The second visualization displays a measure of covariate overlap between the clinical practice populations and the trial population. The third and fourth visualizations highlight considerations for model specification and influential observations. The conceptual roadmap describes how the output from the visualizations helps interrogate the assumptions required to extend treatment effects from trials to target populations.

CONCLUSIONS

The roadmap and visualizations can inform practical decisions required for quantitatively extending treatment effects from trials to clinical practice settings.

Estimating Subgroup Effects in Generalizability and Transportability Analyses

Methods for extending-generalizing or transporting-inferences from a randomized trial to a target population involve conditioning on a large set of covariates that is sufficient for rendering the randomized and nonrandomized groups exchangeable. Yet, decision makers are often interested in examining treatment effects in subgroups of the target population defined in terms of only a few discrete covariates. Here, we propose methods for estimating subgroup-specific potential outcome means and average treatment effects in generalizability and transportability analyses, using outcome model--based (g-formula), weighting, and augmented weighting estimators. We consider estimating subgroup-specific average treatment effects in the target population and its nonrandomized subset, and we provide methods that are appropriate both for nested and non-nested trial designs. As an illustration, we apply the methods to data from the Coronary Artery Surgery Study (North America, 1975-1996) to compare the effect of surgery plus medical therapy versus medical therapy alone for chronic coronary artery disease in subgroups defined by history of myocardial infarction.

Comparing Lung Cancer Screening Strategies in a Nationally Representative US Population Using Transportability Methods for the National Lung Cancer Screening Trial

Importance

The National Lung Screening Trial (NLST) found that screening for lung cancer with low-dose computed tomography (CT) reduced lung cancer-specific and all-cause mortality compared with chest radiography. It is uncertain whether these results apply to a nationally representative target population.

Objective

To extend inferences about the effects of lung cancer screening strategies from the NLST to a nationally representative target population of NLST-eligible US adults.

Design, Setting, and Participants

This comparative effectiveness study included NLST data from US adults at 33 participating centers enrolled between August 2002 and April 2004 with follow-up through 2009 along with National Health Interview Survey (NHIS) cross-sectional household interview survey data from 2010. Eligible participants were adults aged 55 to 74 years, and were current or former smokers with at least 30 pack-years of smoking (former smokers were required to have quit within the last 15 years). Transportability analyses combined baseline covariate, treatment, and outcome data from the NLST with covariate data from the NHIS and reweighted the trial data to the target population. Data were analyzed from March 2020 to May 2023.

Interventions

Low-dose CT or chest radiography screening with a screening assessment at baseline, then yearly for 2 more years.

Main Outcomes and Measures

For the outcomes of lung-cancer specific and all-cause death, mortality rates, rate differences, and ratios were calculated at a median (25th percentile and 75th percentile) follow-up of 5.5 (5.2-5.9) years for lung cancer-specific mortality and 6.5 (6.1-6.9) years for all-cause mortality.

Results

The transportability analysis included 51 274 NLST participants and 685 NHIS participants representing the target population (of approximately 5 700 000 individuals after survey-weighting). Compared with the target population, NLST participants were younger (median [25th percentile and 75th percentile] age, 60 [57 to 65] years vs 63 [58 to 67] years), had fewer comorbidities (eg, heart disease, 6551 of 51 274 [12.8%] vs 1 025 951 of 5 739 532 [17.9%]), and were more educated (bachelor's degree or higher, 16 349 of 51 274 [31.9%] vs 859 812 of 5 739 532 [15.0%]). In the target population, for lung cancer-specific mortality, the estimated relative rate reduction was 18% (95% CI, 1% to 33%) and the estimated absolute rate reduction with low-dose CT vs chest radiography was 71 deaths per 100 000 person-years (95% CI, 4 to 138 deaths per 100 000 person-years); for all-cause mortality the estimated relative rate reduction was 6% (95% CI, -2% to 12%). In the NLST, for lung cancer-specific mortality, the estimated relative rate reduction was 21% (95% CI, 9% to 32%) and the estimated absolute rate reduction was 67 deaths per 100 000 person-years (95% CI, 27 to 106 deaths per 100 000 person-years); for all-cause mortality, the estimated relative rate reduction was 7% (95% CI, 0% to 12%).

Conclusions and Relevance

Estimates of the comparative effectiveness of low-dose CT screening compared with chest radiography in a nationally representative target population were similar to those from unweighted NLST analyses, particularly on the relative scale. Increased uncertainty around effect estimates for the target population reflects large differences in the observed characteristics of trial participants and the target population.

How Generalizable Are Findings from a Community-Based Prospective Cohort Study? Extending Estimates from the Adult Changes in Thought Study to Its Source Population

Background

The Adult Changes in Thought (ACT) study is a cohort of Kaiser Permanente Washington members ages 65+ that began in 1994.

Objective

We wanted to know how well ACT participants represented all older adults in the region, and how well ACT findings on eye disease and its relationship with Alzheimer's disease generalized to all older adults in the Seattle Metropolitan Region.

Methods

We used participation weights derived from pooling ACT and Behavioral Risk Factor Surveillance System (BRFSS) data to estimate prevalences of common eye diseases and their associations with Alzheimer's disease incidence. Cox proportional hazards models accounted for age, education, smoking, sex, and APOE genotype. Confidence intervals for weighted analyses were bootstrapped to account for error in estimating the weights.

Results

ACT participants were fairly similar to older adults in the region. The largest differences were more self-reported current cholesterol medication use in BRFSS and higher proportions with low education in ACT. Incorporating the weights had little impact on prevalence estimates for age-related macular degeneration or glaucoma. Weighted estimates were slightly higher for diabetic retinopathy (weighted 5.7% (95% Confidence Interval 4.3, 7.1); unweighted 4.1% (3.6, 4.6)) and cataract history (weighted 51.8% (49.6, 54.3); unweighted 48.6% (47.3, 49.9)). The weighted hazard ratio for recent diabetic retinopathy diagnosis and Alzheimer's disease was 1.84 (0.34, 4.29), versus 1.32 (0.87, 2.00) in unweighted ACT.

Conclusions

Most, but not all, associations were similar after participation weighting. Even in community-based cohorts, extending inferences to broader populations may benefit from evaluation with participation weights.

External validity in distributed data networks

PURPOSE

While much has been written about how distributed networks address internal validity, external validity is rarely discussed. We aimed to define key terms related to external validity, discuss how they relate to distributed networks, and identify how three networks (the US Food and Drug Administration's Sentinel System, the Canadian Network for Observational Drug Effect Studies [CNODES], and the National Patient Centered Clinical Research Network [PCORnet]) deal with external validity.

METHODS

We define external validity, target populations, target validity, generalizability, and transportability and describe how each relates to distributed networks. We then describe Sentinel, CNODES, and PCORnet and how each approaches these concepts, including a sample case study.

RESULTS

Each network approaches external validity differently. As its target population is US citizens and it includes only US data, Sentinel primarily worries about lack of external validity by not including some segments of the population. The fact that CNODES includes Canadian, United States, and United Kingdom data forces them to seriously consider whether the United States and United Kingdom data will be transportable to Canadian citizens when they meta-analyze database-specific estimates. PCORnet, with its focus on study-specific cohorts and pragmatic trials, conducts more case-by-case explorations of external validity for each new analytic data set it generates.

CONCLUSIONS

There is no one-size-fits-all approach to external validity within distributed networks. With these networks and comparisons between their findings becoming a key part of pharmacoepidemiology, there is a need to adapt tools for improving external validity to the distributed network setting.

Assessing the representativeness of cluster randomized trials: Evidence from two large pragmatic trials in United States nursing homes

BACKGROUND/AIMS

When the randomized clusters in a cluster randomized trial are selected based on characteristics that influence treatment effectiveness, results from the trial may not be directly applicable to the target population. We used data from two large nursing home-based pragmatic cluster randomized trials to compare nursing home and resident characteristics in randomized facilities to eligible non-randomized and ineligible facilities.

METHODS

We linked data from the high-dose influenza vaccine trial and the Music & Memory Pragmatic TRIal for Nursing Home Residents with ALzheimer's Disease (METRICaL) to nursing home assessments and Medicare fee-for-service claims. The target population for the high-dose trial comprised Medicare-certified nursing homes; the target population for the METRICaL trial comprised nursing homes in one of four US-based nursing home chains. We used standardized mean differences to compare facility and individual characteristics across the three groups and logistic regression to model the probability of nursing home trial participation.

RESULTS

In the high-dose trial, 4476 (29%) of the 15,502 nursing homes in the target population were eligible for the trial, of which 818 (18%) were randomized. Of the 1,361,122 residents, 91,179 (6.7%) were residents of randomized facilities, 463,703 (34.0%) of eligible non-randomized facilities, and 806,205 (59.3%) of ineligible facilities. In the METRICaL trial, 160 (59%) of the 270 nursing homes in the target population were eligible for the trial, of which 80 (50%) were randomized. Of the 20,262 residents, 973 (34.4%) were residents of randomized facilities, 7431 (36.7%) of eligible non-randomized facilities, and 5858 (28.9%) of ineligible facilities. In the high-dose trial, randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (132.5 vs 145.9 and 91.9, respectively), for-profit status (91.8% vs 66.8% and 68.8%), belonging to a nursing home chain (85.8% vs 49.9% and 54.7%), and presence of a special care unit (19.8% vs 25.9% and 14.4%). In the METRICaL trial randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (103.7 vs 110.5 and 67.0), resource-poor status (4.6% vs 10.0% and 18.8%), and presence of a special care unit (26.3% vs 33.8% and 10.9%). In both trials, the characteristics of residents in randomized facilities were similar across the three groups.

CONCLUSION

In both trials, facility-level characteristics of randomized nursing homes differed considerably from those of eligible non-randomized and ineligible facilities, while there was little difference in resident-level characteristics across the three groups. Investigators should assess the characteristics of clusters that participate in cluster randomized trials, not just the individuals within the clusters, when examining the applicability of trial results beyond participating clusters.

Generalizability of the Necrotizing Enterocolitis Surgery Trial to the Target Population of Eligible Infants

OBJECTIVE

The objective of this study was to evaluate whether infants randomized in the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network Necrotizing Enterocolitis Surgery Trial differed from eligible infants and whether differences affected the generalizability of trial results.

STUDY DESIGN

Secondary analysis of infants enrolled in Necrotizing Enterocolitis Surgery Trial (born 2010-2017, with follow-up through 2019) at 20 US academic medical centers and an observational data set of eligible infants through 2013. Infants born ≤1000 g and diagnosed with necrotizing enterocolitis or spontaneous intestinal perforation requiring surgical intervention at ≤8 weeks were eligible. The target population included trial-eligible infants (randomized and nonrandomized) born during the first half of the study with available detailed preoperative data. Using model-based weighting methods, we estimated the effect of initial laparotomy vs peritoneal drain had the target population been randomized.

RESULTS

The trial included 308 randomized infants. The target population included 382 (156 randomized and 226 eligible, non-randomized) infants. Compared with the target population, fewer randomized infants had necrotizing enterocolitis (31% vs 47%) or died before discharge (27% vs 41%). However, incidence of the primary composite outcome, death or neurodevelopmental impairment, was similar (69% vs 72%). Effect estimates for initial laparotomy vs drain weighted to the target population were largely unchanged from the original trial after accounting for preoperative diagnosis of necrotizing enterocolitis (adjusted relative risk [95% CI]: 0.85 [0.71-1.03] in target population vs 0.81 [0.64-1.04] in trial) or spontaneous intestinal perforation (1.02 [0.79-1.30] vs 1.11 [0.95-1.31]).

CONCLUSION

Despite differences between randomized and eligible infants, estimated treatment effects in the trial and target population were similar, supporting the generalizability of trial results.

TRIAL REGISTRATION

ClinicalTrials.gov ID: NCT01029353.

Estimating the area under the ROC curve when transporting a prediction model to a target population

We propose methods for estimating the area under the receiver operating characteristic (ROC) curve (AUC) of a prediction model in a target population that differs from the source population that provided the data used for original model development. If covariates that are associated with model performance, as measured by the AUC, have a different distribution in the source and target populations, then AUC estimators that only use data from the source population will not reflect model performance in the target population. Here, we provide identification results for the AUC in the target population when outcome and covariate data are available from the sample of the source population, but only covariate data are available from the sample of the target population. In this setting, we propose three estimators for the AUC in the target population and show that they are consistent and asymptotically normal. We evaluate the finite-sample performance of the estimators using simulations and use them to estimate the AUC in a nationally representative target population from the National Health and Nutrition Examination Survey for a lung cancer risk prediction model developed using source population data from the National Lung Screening Trial.

Comparative Effectiveness of Percutaneous Microaxial Left Ventricular Assist Device vs Intra-Aortic Balloon Pump or No Mechanical Circulatory Support in Patients With Cardiogenic Shock

Importance

Recent studies have produced inconsistent findings regarding the outcomes of the percutaneous microaxial left ventricular assist device (LVAD) during acute myocardial infarction with cardiogenic shock (AMICS).

Objective

To compare the percutaneous microaxial LVAD vs alternative treatments among patients presenting with AMICS using observational analyses of administrative data.

Design, Setting, and Participants

This comparative effectiveness research study used Medicare fee-for-service claims of patients admitted with AMICS undergoing percutaneous coronary intervention from October 1, 2015, through December 31, 2019. Treatment strategies were compared using (1) inverse probability of treatment weighting to estimate the effect of different baseline treatments in the overall population; (2) instrumental variable analysis to determine the effectiveness of the percutaneous microaxial LVAD among patients whose treatment was influenced by cross-sectional institutional practice patterns; (3) an instrumented difference-in-differences analysis to determine the effectiveness of treatment among patients whose treatment was influenced by longitudinal changes in institutional practice patterns; and (4) a grace period approach to determine the effectiveness of initiating the percutaneous microaxial LVAD within 2 days of percutaneous coronary intervention. Analysis took place between March 2021 and December 2022.

Interventions

Percutaneous microaxial LVAD vs alternative treatments (including medical therapy and intra-aortic balloon pump).

Main Outcomes and Measures

Thirty-day all-cause mortality and readmissions.

Results

Of 23 478 patients, 14 264 (60.8%) were male and the mean (SD) age was 73.9 (9.8) years. In the inverse probability of treatment weighting analysis and grace period approaches, treatment with percutaneous microaxial LVAD was associated with a higher risk-adjusted 30-day mortality (risk difference, 14.9%; 95% CI, 12.9%-17.0%). However, patients receiving the percutaneous microaxial LVAD had a higher frequency of factors associated with severe illness, suggesting possible confounding by measures of illness severity not available in the data. In the instrumental variable analysis, 30-day mortality was also higher with percutaneous microaxial LVAD, but patient and hospital characteristics differed across levels of the instrumental variable, suggesting possible confounding by unmeasured variables (risk difference, 13.5%; 95% CI, 3.9%-23.2%). In the instrumented difference-in-differences analysis, the association between the percutaneous microaxial LVAD and mortality was imprecise, and differences in trends in characteristics between hospitals with different percutaneous microaxial LVAD use suggested potential assumption violations.

Conclusions

In observational analyses comparing the percutaneous microaxial LVAD to alternative treatments among patients with AMICS, the percutaneous microaxial LVAD was associated with worse outcomes in some analyses, while in other analyses, the association was too imprecise to draw meaningful conclusions. However, the distribution of patient and institutional characteristics between treatment groups or groups defined by institutional differences in treatment use, including changes in use over time, combined with clinical knowledge of illness severity factors not captured in the data, suggested violations of key assumptions that are needed for valid causal inference with different observational analyses. Randomized clinical trials of mechanical support devices will allow valid comparisons across candidate treatment strategies and help resolve ongoing controversies.

Extending prediction models for use in a new target population with failure time outcomes

Prediction models are often built and evaluated using data from a population that differs from the target population where model-derived predictions are intended to be used in. In this article, we present methods for evaluating model performance in the target population when some observations are right censored. The methods assume that outcome and covariate data are available from a source population used for model development and covariates, but no outcome data, are available from the target population. We evaluate the finite sample performance of the proposed estimators using simulations and apply the methods to transport a prediction model built using data from a lung cancer screening trial to a nationally representative population of participants eligible for lung cancer screening.

Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population

Extending (i.e., generalizing or transporting) causal inferences from a randomized trial to a target population requires assumptions that randomized and nonrandomized individuals are exchangeable conditional on baseline covariates. These assumptions are made on the basis of background knowledge, which is often uncertain or controversial, and need to be subjected to sensitivity analysis. We present simple methods for sensitivity analyses that directly parameterize violations of the assumptions using bias functions and do not require detailed background knowledge about specific unknown or unmeasured determinants of the outcome or modifiers of the treatment effect. We show how the methods can be applied to non-nested trial designs, where the trial data are combined with a separately obtained sample of nonrandomized individuals, as well as to nested trial designs, where the trial is embedded within a cohort sampled from the target population.

Efficient and robust methods for causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a target population

We present methods for causally interpretable meta-analyses that combine information from multiple randomized trials to draw causal inferences for a target population of substantive interest. We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results for transporting causal inferences from a collection of independent randomized trials to a new target population in which experimental data may not be available. We propose an estimator for the potential outcome mean in the target population under each treatment studied in the trials. The estimator uses covariate, treatment, and outcome data from the collection of trials, but only covariate data from the target population sample. We show that it is doubly robust in the sense that it is consistent and asymptotically normal when at least one of the models it relies on is correctly specified. We study the finite sample properties of the estimator in simulation studies and demonstrate its implementation using data from a multicenter randomized trial.

Improving trial generalizability using observational studies

Complementary features of randomized controlled trials (RCTs) and observational studies (OSs) can be used jointly to estimate the average treatment effect of a target population. We propose a calibration weighting estimator that enforces the covariate balance between the RCT and OS, therefore improving the trial-based estimator's generalizability. Exploiting semiparametric efficiency theory, we propose a doubly robust augmented calibration weighting estimator that achieves the efficiency bound derived under the identification assumptions. A nonparametric sieve method is provided as an alternative to the parametric approach, which enables the robust approximation of the nuisance functions and data-adaptive selection of outcome predictors for calibration. We establish asymptotic results and confirm the finite sample performances of the proposed estimators by simulation experiments and an application on the estimation of the treatment effect of adjuvant chemotherapy for early-stage non-small-cell lung patients after surgery.

John J, Kono N, Henderson VW, Mack WJ. Generalizability of cognitive results from clinical trial participants to an older adult population: Addressing external validity

Introduction

Study inclusion criteria and recruitment practices limit the generalizability of randomized-controlled trial (RCT) results. Statistical modeling could enhance generalizability of outcomes. To illustrate this, the cognition-depression relationship was assessed with and without adjustment relative to the target population of older women.

Methods

Randomized participants from four RCTs and non-randomized participants from two cohorts were included in this study. Prediction models estimated probability of being randomized into trials from target populations. These probabilities were used for inverse odds weighting relative to target populations. Weighted linear regression was used to assess the depression-cognition relationship.

Results

There was no depression-cognition relationship in the combined randomized sample. After applying weights relative to a representative cohort, negative relationships were observed. After applying weights relative to a non-representative cohort, bias of estimates increased.

Discussion

Quantitative approaches to transportability using representative samples may explain the absence of a-priori established relationships in RCTs.

Improving efficiency of inference in clinical trials with external control data

Suppose we are interested in the effect of a treatment in a clinical trial. The efficiency of inference may be limited due to small sample size. However, external control data are often available from historical studies. Motivated by an application to Helicobacter pylori infection, we show how to borrow strength from such data to improve efficiency of inference in the clinical trial. Under an exchangeability assumption about the potential outcome mean, we show that the semiparametric efficiency bound for estimating the average treatment effect can be reduced by incorporating both the clinical trial data and external controls. We then derive a doubly robust and locally efficient estimator. The improvement in efficiency is prominent especially when the external control data set has a large sample size and small variability. Our method allows for a relaxed overlap assumption, and we illustrate with the case where the clinical trial only contains a treated group. We also develop doubly robust and locally efficient approaches that extrapolate the causal effect in the clinical trial to the external population and the overall population. Our results also offer a meaningful implication for trial design and data collection. We evaluate the finite-sample performance of the proposed estimators via simulation. In the Helicobacter pylori infection application, our approach shows that the combination treatment has potential efficacy advantages over the triple therapy.

Illustration of 2 Fusion Designs and Estimators

"Fusion" study designs combine data from different sources to answer questions that could not be answered (as well) by subsets of the data. Studies that augment main study data with validation data, as in measurement-error correction studies or generalizability studies, are examples of fusion designs. Fusion estimators, here solutions to stacked estimating functions, produce consistent answers to identified research questions using data from fusion designs. In this paper, we describe a pair of examples of fusion designs and estimators, one where we generalize a proportion to a target population and one where we correct measurement error in a proportion. For each case, we present an example motivated by human immunodeficiency virus research and summarize results from simulation studies. Simulations demonstrate that the fusion estimators provide approximately unbiased results with appropriate 95% confidence interval coverage. Fusion estimators can be used to appropriately combine data in answering important questions that benefit from multiple sources of information.

Transporting a Prediction Model for Use in a New Target Population

We considered methods for transporting a prediction model for use in a new target population, both when outcome and covariate data for model development are available from a source population that has a different covariate distribution compared with the target population and when covariate data (but not outcome data) are available from the target population. We discuss how to tailor the prediction model to account for differences in the data distribution between the source population and the target population. We also discuss how to assess the model's performance (e.g., by estimating the mean squared prediction error) in the target population. We provide identifiability results for measures of model performance in the target population for a potentially misspecified prediction model under a sampling design where the source and the target population samples are obtained separately. We introduce the concept of prediction error modifiers that can be used to reason about tailoring measures of model performance to the target population. We illustrate the methods in simulated data and apply them to transport a prediction model for lung cancer diagnosis from the National Lung Screening Trial to the nationally representative target population of trial-eligible individuals in the National Health and Nutrition Examination Survey.

Regression-based estimation of heterogeneous treatment effects when extending inferences from a randomized trial to a target population

Most work on extending (generalizing or transporting) inferences from a randomized trial to a target population has focused on estimating average treatment effects (i.e., averaged over the target population's covariate distribution). Yet, in the presence of strong effect modification by baseline covariates, the average treatment effect in the target population may be less relevant for guiding treatment decisions. Instead, the conditional average treatment effect (CATE) as a function of key effect modifiers may be a more useful estimand. Recent work on estimating target population CATEs using baseline covariate, treatment, and outcome data from the trial and covariate data from the target population only allows for the examination of heterogeneity over distinct subgroups. We describe flexible pseudo-outcome regression modeling methods for estimating target population CATEs conditional on discrete or continuous baseline covariates when the trial is embedded in a sample from the target population (i.e., in nested trial designs). We construct pointwise confidence intervals for the CATE at a specific value of the effect modifiers and uniform confidence bands for the CATE function. Last, we illustrate the methods using data from the Coronary Artery Surgery Study (CASS) to estimate CATEs given history of myocardial infarction and baseline ejection fraction value in the target population of all trial-eligible patients with stable ischemic heart disease.

No short-term mortality from benzodiazepine use post-acute ischemic stroke after accounting for bias

BACKGROUND AND OBJECTIVES

Older adults receive benzodiazepines for agitation, anxiety, and insomnia after acute ischemic stroke (AIS). No trials have been conducted to determine if benzodiazepine use affects poststroke mortality in the elderly.

METHODS

We examined the association between initiating benzodiazepines within 1 week after AIS and 30-day mortality. We included patients ≥65 years, admitted for new nonsevere AIS (NIH-Stroke-Severity[NIHSS]≤ 20), 2014-2020, with no recorded benzodiazepine use in the previous 3 months and no contraindication for use. We linked a stroke registry to electronic health records, used inverse-probability weighting to address confounding, and estimated the risk difference (RD). A process of cloning, weighting, and censoring was used to avoid immortal time bias.

RESULTS

Among 2,584 patients, 389 received benzodiazepines. The crude 30-day mortality risk from treatment initiation was 212/1,000 among patients who received benzodiazepines, while the 30-day mortality was 34/1,000 among those who did not. When follow-up was aligned on day of AIS admission and immortal time was assigned to the two groups, the estimated risks were 27/1,000 and 22/1,000, respectively. Upon further adjustment for confounders, the RD was 5 (-12 to 19) deaths/1,000 patients.

CONCLUSION

The observed higher 30-day mortality associated with benzodiazepine initiation within 7 days was largely due to bias.

Data Integration in Causal Inference

Integrating data from multiple heterogeneous sources has become increasingly popular to achieve a large sample size and diverse study population. This paper reviews development in causal inference methods that combines multiple datasets collected by potentially different designs from potentially heterogeneous populations. We summarize recent advances on combining randomized clinical trial with external information from observational studies or historical controls, combining samples when no single sample has all relevant variables with application to two-sample Mendelian randomization, distributed data setting under privacy concerns for comparative effectiveness and safety research using real-world data, Bayesian causal inference, and causal discovery methods.

Targeting keystone species helps restore the dysbiosis of butyrate-producing bacteria in nonalcoholic fatty liver disease

The dysbiosis of the gut microbiome is one of the pathogenic factors of nonalcoholic fatty liver disease (NAFLD) and also affects the treatment and intervention of NAFLD. Among gut microbiomes, keystone species that regulate the integrity and stability of an ecological community have become the potential intervention targets for NAFLD. Here, we collected stool samples from 22 patients with nonalcoholic steatohepatitis (NASH), 25 obese patients, and 16 healthy individuals from New York for 16S rRNA gene sequencing. An algorithm was implemented to identify keystone species based on causal inference theories and dynamic intervention simulation. External validation was performed in an independent cohort from California. Eight keystone species in the gut of NAFLD, represented by Porphyromonas loveana, Alistipes indistinctus, and Dialister pneumosintes, were identified, which could efficiently restore the microbial composition of the NAFLD toward a normal gut microbiome with 92.3% recovery. These keystone species regulate intestinal amino acid metabolism and acid-base environment to promote the growth of the butyrate-producing Lachnospiraceae and Ruminococcaceae species that are significantly reduced in NAFLD patients. Our findings demonstrate the importance of keystone species in restoring the microbial composition toward a normal gut microbiome, suggesting a novel potential microbial treatment for NAFLD.

Transfer learning of individualized treatment rules from experimental to real-world data

Individualized treatment effect lies at the heart of precision medicine. Interpretable individualized treatment rules (ITRs) are desirable for clinicians or policymakers due to their intuitive appeal and transparency. The gold-standard approach to estimating the ITRs is randomized experiments, where subjects are randomized to different treatment groups and the confounding bias is minimized to the extent possible. However, experimental studies are limited in external validity because of their selection restrictions, and therefore the underlying study population is not representative of the target real-world population. Conventional learning methods of optimal interpretable ITRs for a target population based only on experimental data are biased. On the other hand, real-world data (RWD) are becoming popular and provide a representative sample of the target population. To learn the generalizable optimal interpretable ITRs, we propose an integrative transfer learning method based on weighting schemes to calibrate the covariate distribution of the experiment to that of the RWD. Theoretically, we establish the risk consistency for the proposed ITR estimator. Empirically, we evaluate the finite-sample performance of the transfer learner through simulations and apply it to a real data application of a job training program.

The population-attributable fraction for time-to-event data

Background

Even though the population-attributable fraction (PAF) is a well-established metric, it is often incorrectly estimated or interpreted not only in clinical application, but also in statistical research articles. The risk of bias is especially high in more complex time-to-event data settings.

Methods

We explain how the PAF can be defined, identified and estimated in time-to-event settings with competing risks and time-dependent exposures. By using multi-state methodology and inverse probability weighting, we demonstrate how to reduce or completely avoid severe types of biases including competing risks bias, immortal time bias and confounding due to both baseline and time-varying patient characteristics.

Results

The method is exemplarily applied to a real data set. Moreover, we estimate the number of deaths that were attributable to ventilator-associated pneumonia in France in the year 2016. The example demonstrates how, under certain simplifying assumptions, PAF estimates can be extrapolated to a target population of interest.

Conclusions

Defining and estimating the PAF in advanced time-to-event settings within a framework that unifies causal and multi-state modelling enables to tackle common sources of bias and allows straightforward implementation with standard software packages.

Special considerations in randomized trials investigating neonatal surgical treatments

Randomized controlled trials (RCTs) are challenging, but are the studies most likely to change practice and benefit patients. RCTs investigating neonatal surgical therapies are rare. The Necrotizing Enterocolitis Surgery Trial (NEST) was the first surgical RCT conducted by the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Neonatal Research Network (NRN), and multiple lessons were learned. NEST was conducted over a 7.25-year enrollment period and the primary outcome was death or neurodevelopmental impairment (NDI) at 18-22 months corrected age. Surgical investigators designing clinical trials involving neonatal surgical treatments have many considerations to include, including how to study eligible but non-randomized patients, heterogeneity of treatment effect, use of frequentist and Bayesian analyses, assessment of generalizability, and anticipating criticisms during peer review. Surgeons are encouraged to embrace these challenges and seek innovative methods to acquire evidence that will be used to improve patient outcomes.

Benefits and Risks Associated With Continuation of Anti-Tumor Necrosis Factor After 24 Weeks of Pregnancy in Women With Inflammatory Bowel Disease : A Nationwide Emulation Trial

BACKGROUND

Continuation of biologics for inflammatory disorders during pregnancy is still a difficult decision. Many women with inflammatory bowel diseases (IBDs) stop anti-tumor necrosis factor (anti-TNF) treatment after 24 weeks.

OBJECTIVE

To evaluate the benefits and risks of anti-TNF continuation after 24 weeks of pregnancy for mothers with IBD and their offspring.

DESIGN

Target trial emulation between 2010 and 2020.

SETTING

Nationwide population-based study using the Système National des Données de Santé.

PATIENTS

All pregnancies with birth exposed to anti-TNF between conception and 24 weeks of pregnancy in women with IBD.

INTERVENTION

Continuation of anti-TNF after 24 weeks of pregnancy.

MEASUREMENTS

Occurrence of maternal IBD relapse up to 6 months after pregnancy, adverse pregnancy outcomes, and serious infections in the offspring during the first 5 years of life was compared according to anti-TNF continuation after 24 weeks of pregnancy using inverse probability-weighted marginal models.

RESULTS

A total of 5293 pregnancies were included; among them, anti-TNF treatment was discontinued before 24 weeks for 2890 and continued beyond 24 weeks for 2403. Continuation of anti-TNF was associated with decreased frequencies of maternal IBD relapse (35.8% vs. 39.0%; adjusted risk ratio [aRR], 0.93 [95% CI, 0.86 to 0.99]) and prematurity (7.6% vs. 8.9%; aRR, 0.82 [CI, 0.68 to 0.99]). No difference according to anti-TNF continuation was found regarding stillbirths (0.4% vs. 0.2%; aRR, 2.16 [CI, 0.64 to 7.81]), small weight for gestational age births (13.1% vs. 12.9%; aRR, 1.01 [CI, 0.88 to 1.17]), and serious infections in the offspring (54.2 vs. 50.2 per 1000 person-years; adjusted hazard ratio, 1.08 [CI, 0.94 to 1.25]).

LIMITATION

Algorithms rather than clinical data were used to identify patients with IBD, pregnancies, and serious infections.

CONCLUSION

Continuation of anti-TNF after 24 weeks of pregnancy appears beneficial regarding IBD activity and prematurity, while not affecting neonatal outcomes and serious infections in the offspring.

PRIMARY FUNDING SOURCE

None.

Profile Matching for the Generalization and Personalization of Causal Inferences

We introduce profile matching, a multivariate matching method for randomized experiments and observational studies that finds the largest possible unweighted samples across multiple treatment groups that are balanced relative to a covariate profile. This covariate profile can represent a specific population or a target individual, facilitating the generalization and personalization of causal inferences. For generalization, because the profile often amounts to summary statistics for a target population, profile matching does not always require accessing individual-level data, which may be unavailable for confidentiality reasons. For personalization, the profile comprises the characteristics of a single individual. Profile matching achieves covariate balance by construction, but unlike existing approaches to matching, it does not require specifying a matching ratio, as this is implicitly optimized for the data. The method can also be used for the selection of units for study follow-up, and it readily applies to multivalued treatments with many treatment categories. We evaluate the performance of profile matching in a simulation study of the generalization of a randomized trial to a target population. We further illustrate this method in an exploratory observational study of the relationship between opioid use and mental health outcomes. We analyze these relationships for three covariate profiles representing: (i) sexual minorities, (ii) the Appalachian United States, and (iii) the characteristics of a hypothetical vulnerable patient. The method can be implemented via the new function profmatch in the designmatch package for R, for which we provide a step-by-step tutorial.

Using Numerical Methods to Design Simulations: Revisiting the Balancing Intercept

In this paper, we consider methods for generating draws of a binary random variable whose expectation conditional on covariates follows a logistic regression model with known covariate coefficients. We examine approximations for finding a "balancing intercept," that is, a value for the intercept of the logistic model that leads to a desired marginal expectation for the binary random variable. We show that a recently proposed analytical approximation can produce inaccurate results, especially when targeting more extreme marginal expectations or when the linear predictor of the regression model has high variance. We then formulate the balancing intercept as a solution to an integral equation, implement a numerical approximation for solving the equation based on Monte Carlo methods, and show that the approximation works well in practice. Our approach to the basic problem of the balancing intercept provides an example of a broadly applicable strategy for formulating and solving problems that arise in the design of simulation studies used to evaluate or teach epidemiologic methods.