Invited Commentary: Combining Information to Answer Epidemiologic Questions About a Target Population

Epidemiologists are attempting to address research questions of increasing complexity by developing novel methods for combining information from diverse sources. Cole et al. (Am J Epidemiol. 2023;192(3)467-474) provide 2 examples of the process of combining information to draw inferences about a population proportion. In this commentary, we consider combining information to learn about a target population as an epidemiologic activity and distinguish it from more conventional meta-analyses. We examine possible rationales for combining information and discuss broad methodological considerations, with an emphasis on study design, assumptions, and sources of uncertainty.

Transporting survival of an HIV clinical trial to the external target populations

Due to the heterogeneity of the randomized controlled trial (RCT) and external target populations, the estimated treatment effect from the RCT is not directly applicable to the target population. For example, the patient characteristics of the ACTG 175 HIV trial are significantly different from that of the three external target populations of interest: US early-stage HIV patients, Thailand HIV patients, and southern Ethiopia HIV patients. This paper considers several methods to transport the treatment effect from the ACTG 175 HIV trial to the target populations beyond the trial population. Most transport methods focus on continuous and binary outcomes; on the contrary, we derive and discuss several transport methods for survival outcomes: an outcome regression method based on a Cox proportional hazard (PH) model, an inverse probability weighting method based on the models for treatment assignment, sampling score, and censoring, and a doubly robust method that combines both methods, called the augmented calibration weighting (ACW) method. However, as the PH assumption was found to be incorrect for the ACTG 175 trial, the methods that depend on the PH assumption may lead to the biased quantification of the treatment effect. To account for the violation of the PH assumption, we extend the ACW method with the linear spline-based hazard regression model that does not require the PH assumption. Applying the aforementioned methods for transportability, we explore the effect of PH assumption, or the violation thereof, on transporting the survival results from the ACTG 175 trial to various external populations.

Leveraging External Validation Data: The Challenges of Transporting Measurement Error Parameters

Approaches to address measurement error frequently rely on validation data to estimate measurement error parameters (e.g., sensitivity and specificity). Acquisition of validation data can be costly, thus secondary use of existing data for validation is attractive. To use these external validation data, however, we may need to address systematic differences between these data and the main study sample. Here, we derive estimators of the risk and the risk difference that leverage external validation data to account for outcome misclassification. If misclassification is differential with respect to covariates that themselves are differentially distributed in the validation and study samples, the misclassification parameters are not immediately transportable. We introduce two ways to account for such covariates: (1) standardize by these covariates or (2) iteratively model the outcome. If conditioning on a covariate for transporting the misclassification parameters induces bias of the causal effect (e.g., M-bias), the former but not the latter approach is biased. We provide proof of identification, describe estimation using parametric models, and assess performance in simulations. We also illustrate implementation to estimate the risk of preterm birth and the effect of maternal HIV infection on preterm birth. Measurement error should not be ignored and it can be addressed using external validation data via transportability methods.

Transporting results in an observational epidemiology setting: purposes, methods, and applied example

In the medical domain, substantial effort has been invested in generating internally valid estimates in experimental as well as observational studies, but limited effort has been made in testing generalizability, or external validity. Testing the external validity of scientific findings is nevertheless crucial for the application of knowledge across populations. In particular, transporting estimates obtained from observational studies requires the combination of methods for causal inference and methods to transport the effect estimates in order to minimize biases inherent to observational studies and to account for differences between the study and target populations. In this paper, the conceptual framework and assumptions behind transporting results from a population-based study population to a target population is described in an observational setting. An applied example to life-course epidemiology, where internal validity was constructed for illustrative purposes, is shown by using the targeted maximum likelihood estimator.

Fusing trial data for treatment comparisons: Single vs multi-span bridging

While randomized controlled trials (RCTs) are critical for establishing the efficacy of new therapies, there are limitations regarding what comparisons can be made directly from trial data. RCTs are limited to a small number of comparator arms and often compare a new therapeutic to a standard of care which has already proven efficacious. It is sometimes of interest to estimate the efficacy of the new therapy relative to a treatment that was not evaluated in the same trial, such as a placebo or an alternative therapy that was evaluated in a different trial. Such dual-study comparisons are challenging because of potential differences between trial populations that can affect the outcome. In this article, two bridging estimators are considered that allow for comparisons of treatments evaluated in different trials, accounting for measured differences in trial populations. A "multi-span" estimator leverages a shared arm between two trials, while a "single-span" estimator does not require a shared arm. A diagnostic statistic that compares the outcome in the standardized shared arms is provided. The two estimators are compared in simulations, where both estimators demonstrate minimal empirical bias and nominal confidence interval coverage when the identification assumptions are met. The estimators are applied to data from the AIDS Clinical Trials Group 320 and 388 to compare the efficacy of two-drug vs four-drug antiretroviral therapy on CD4 cell counts among persons with advanced HIV. The single-span approach requires weaker identification assumptions and was more efficient in simulations and the application.

Transporting and implementing a caregiver-mediated intervention for toddlers with autism in Goa, India: evidence from the social ABCs

Introduction

Autism is a global health priority with an urgent need for evidence-based, resource-efficient, scalable supports that are feasible for implementation in low- and middle-income countries (LMICs). Initiating supports in the toddler years has potential to significantly impact child and family outcomes. The current paper describes the feasibility and outcomes associated with a Canadian-developed caregiver-mediated intervention for toddlers (the Social ABCs), delivered through a clinical service in Goa, India.

Methods

Clinical staff at the Sethu Centre for Child Development and Family Guidance in Goa, India, were trained by the Canadian program development team and delivered the program to families seen through their clinic. Using a retrospective chart review, we gathered information about participating families and used a pre-post design to examine change over time.

Results

Sixty-four families were enrolled (toddler mean age = 28.5 months; range: 19-35), of whom 55 (85.94%) completed the program. Video-coded data revealed that parents learned the strategies (implementation fidelity increased from M = 45.42% to 76.77%, p < .001, with over 90% of caregivers attaining at least 70% fidelity). Toddler responsivity to their caregivers (M = 7.00% vs. 46.58%) and initiations per minute (M = 1.16 vs. 3.49) increased significantly, p's < .001. Parents also reported significant improvements in child behaviour/skills (p < .001), and a non-significant trend toward reduced parenting stress (p = .056).

Discussion

Findings corroborate the emerging evidence supporting the use of caregiver-mediated models in LMICs, adding evidence that such supports can be provided in the very early years (i.e., under three years of age) when learning may be optimized.

In-Home Cannabis Smoking Among a Cannabis-Using Convenience Sample from the Global Drug Survey: With Weighted Estimates for U.S. Respondents

Introduction: Secondhand and thirdhand tobacco smoke exposure most often occur at home, but little is known about occurrences of in-home cannabis smoking. We ascertained in-home cannabis smoking reported by all cannabis-using (i.e., used in the last 12 months) respondents to the Global Drug Survey (GDS; international-GDS sample), and among U.S. cannabis-using respondents (US-GDS sample). Materials and Methods: We used data collected November 2019-January 2020 for the 2020 GDS, an annual anonymous, cross-sectional survey; respondents were 16+ years old, from 191 countries. We estimated any and daily in-home cannabis smoking in the last 30 days among international-GDS respondents (n=63,797), using mixed effects logistic regression. US-GDS respondents (n=6,580) were weighted to the covariate distribution of the nationally representative 2018 National Survey on Drug Use and Health cannabis-using sample, using inverse odds probability weighting, to make estimates more generalizable to the U.S. cannabis-using population. Results: For the international-GDS cannabis-using respondents, any in-home cannabis smoking was reported by 63.9% of men, 61.9% of women, and 68.6% of nonbinary people; and by age (<25 years old=62.7%, 25-34 years old=65.0%, and 35+ years old=62.8%). Daily in-home cannabis smoking was highest among nonbinary (28.7%) and respondents 35+ years of age (28.0%). For the weighted US-GDS cannabis-using respondents, any in-home cannabis smoking was reported by 49.8% of males and 61.2% of females; and by age (<25 years old=62.6%, 25-34 years old=41.8%, 35+ years old=57.9%). Weighted daily in-home smoking was 23.2% among males and 37.1% among females; by age (<25 years old=34.8%, 25-34 years old=27.8%, and 35+ years old=21.6%). Conclusions: There was high daily cannabis smoking in homes of international-GDS and US-GDS respondents who used cannabis in the last 12 months. In part, due to cannabis legalization, the number of users worldwide has increased over the past decade. Criminal stigma historically associated with cannabis continues to drive those users indoors. In this context, our findings support further investigation of cannabis use behavior to understand how often people are exposed to secondhand and thirdhand cannabis smoke and the consequences of that exposure.

Two-stage targeted maximum likelihood estimation for mixed aggregate and individual participant data analysis with an application to multidrug resistant tuberculosis

In this study, we develop a new method for the meta-analysis of mixed aggregate data (AD) and individual participant data (IPD). The method is an adaptation of inverse probability weighted targeted maximum likelihood estimation (IPW-TMLE), which was initially proposed for two-stage sampled data. Our methods are motivated by a systematic review investigating treatment effectiveness for multidrug resistant tuberculosis (MDR-TB) where the available data include IPD from some studies but only AD from others. One complication in this application is that participants with MDR-TB are typically treated with multiple antimicrobial agents where many such medications were not observed in all studies considered in the meta-analysis. We focus here on the estimation of the expected potential outcome while intervening on a specific medication but not intervening on any others. Our method involves the implementation of a TMLE that transports the estimation from studies where the treatment is observed to the full target population. A second weighting component adjusts for the studies with missing (inaccessible) IPD. We demonstrate the properties of the proposed method and contrast it with alternative approaches in a simulation study. We finally apply this method to estimate treatment effectiveness in the MDR-TB case study.

Cluster Randomized Trials Designed to Support Generalizable Inferences

When planning a cluster randomized trial, evaluators often have access to an enumerated cohort representing the target population of clusters. Practicalities of conducting the trial, such as the need to oversample clusters with certain characteristics in order to improve trial economy or support inferences about subgroups of clusters, may preclude simple random sampling from the cohort into the trial, and thus interfere with the goal of producing generalizable inferences about the target population. We describe a nested trial design where the randomized clusters are embedded within a cohort of trial-eligible clusters from the target population and where clusters are selected for inclusion in the trial with known sampling probabilities that may depend on cluster characteristics (e.g., allowing clusters to be chosen to facilitate trial conduct or to examine hypotheses related to their characteristics). We develop and evaluate methods for analyzing data from this design to generalize causal inferences to the target population underlying the cohort. We present identification and estimation results for the expectation of the average potential outcome and for the average treatment effect, in the entire target population of clusters and in its non-randomized subset. In simulation studies, we show that all the estimators have low bias but markedly different precision. Cluster randomized trials where clusters are selected for inclusion with known sampling probabilities that depend on cluster characteristics, combined with efficient estimation methods, can precisely quantify treatment effects in the target population, while addressing objectives of trial conduct that require oversampling clusters on the basis of their characteristics.

Transporting randomized trial results to estimate counterfactual survival functions in target populations

When the distributions of treatment effect modifiers differ between a randomized trial and an external target population, the sample average treatment effect in the trial may be substantially different from the target population average treatment, and accurate estimation of the latter requires adjusting for the differential distribution of effect modifiers. Despite the increasingly rich literature on transportability, little attention has been devoted to methods for transporting trial results to estimate counterfactual survival functions in target populations, when the primary outcome is time to event and subject to right censoring. In this article, we study inverse probability weighting and doubly robust estimators to estimate counterfactual survival functions and the target average survival treatment effect in the target population, and provide their respective approximate variance estimators. We focus on a common scenario where the target population information is observed only through a complex survey, and elucidate how the survey weights can be incorporated into each estimator we considered. Simulation studies are conducted to examine the finite-sample performances of the proposed estimators in terms of bias, efficiency and coverage, under both correct and incorrect model specifications. Finally, we apply the proposed method to assess transportability of the results in the Action to Control Cardiovascular Risk in Diabetes-Blood Pressure (ACCORD-BP) trial to all adults with Diabetes in the United States.

Estimating Subgroup Effects in Generalizability and Transportability Analyses

Methods for extending-generalizing or transporting-inferences from a randomized trial to a target population involve conditioning on a large set of covariates that is sufficient for rendering the randomized and nonrandomized groups exchangeable. Yet, decision makers are often interested in examining treatment effects in subgroups of the target population defined in terms of only a few discrete covariates. Here, we propose methods for estimating subgroup-specific potential outcome means and average treatment effects in generalizability and transportability analyses, using outcome model--based (g-formula), weighting, and augmented weighting estimators. We consider estimating subgroup-specific average treatment effects in the target population and its nonrandomized subset, and we provide methods that are appropriate both for nested and non-nested trial designs. As an illustration, we apply the methods to data from the Coronary Artery Surgery Study (North America, 1975-1996) to compare the effect of surgery plus medical therapy versus medical therapy alone for chronic coronary artery disease in subgroups defined by history of myocardial infarction.

Transportability Without Positivity: A Synthesis of Statistical and Simulation Modeling

Studies designed to estimate the effect of an action in a randomized or observational setting often do not represent a random sample of the desired target population. Instead, estimates from that study can be transported to the target population. However, transportability methods generally rely on a positivity assumption, such that all relevant covariate patterns in the target population are also observed in the study sample. Strict eligibility criteria, particularly in the context of randomized trials, may lead to violations of this assumption. Two common approaches to address positivity violations are restricting the target population and restricting the relevant covariate set. As neither of these restrictions is ideal, we instead propose a synthesis of statistical and simulation models to address positivity violations. We propose corresponding g-computation and inverse probability weighting estimators. The restriction and synthesis approaches to addressing positivity violations are contrasted with a simulation experiment and an illustrative example in the context of sexually transmitted infection testing uptake. In both cases, the proposed synthesis approach accurately addressed the original research question when paired with a thoughtfully selected simulation model. Neither of the restriction approaches was able to accurately address the motivating question. As public health decisions must often be made with imperfect target population information, model synthesis is a viable approach given a combination of empirical data and external information based on the best available knowledge.

Conditional cross-design synthesis estimators for generalizability in Medicaid

While much of the causal inference literature has focused on addressing internal validity biases, both internal and external validity are necessary for unbiased estimates in a target population of interest. However, few generalizability approaches exist for estimating causal quantities in a target population that is not well-represented by a randomized study but is reflected when additionally incorporating observational data. To generalize to a target population represented by a union of these data, we propose a novel class of conditional cross-design synthesis estimators that combine randomized and observational data, while addressing their estimates' respective biases-lack of overlap and unmeasured confounding. These methods enable estimating the causal effect of managed care plans on health care spending among Medicaid beneficiaries in New York City, which requires obtaining estimates for the 7% of beneficiaries randomized to a plan and 93% who choose a plan, who do not resemble randomized beneficiaries. Our new estimators include outcome regression, propensity weighting, and double robust approaches. All use the covariate overlap between the randomized and observational data to remove potential unmeasured confounding bias. Applying these methods, we find substantial heterogeneity in spending effects across managed care plans. This has major implications for our understanding of Medicaid, where this heterogeneity has previously been hidden. Additionally, we demonstrate that unmeasured confounding rather than lack of overlap poses a larger concern in this setting.

Interpreting Randomized Controlled Trials

This article describes rationales and limitations for making inferences based on data from randomized controlled trials (RCTs). We argue that obtaining a representative random sample from a patient population is impossible for a clinical trial because patients are accrued sequentially over time and thus comprise a convenience sample, subject only to protocol entry criteria. Consequently, the trial's sample is unlikely to represent a definable patient population. We use causal diagrams to illustrate the difference between random allocation of interventions within a clinical trial sample and true simple or stratified random sampling, as executed in surveys. We argue that group-specific statistics, such as a median survival time estimate for a treatment arm in an RCT, have limited meaning as estimates of larger patient population parameters. In contrast, random allocation between interventions facilitates comparative causal inferences about between-treatment effects, such as hazard ratios or differences between probabilities of response. Comparative inferences also require the assumption of transportability from a clinical trial's convenience sample to a targeted patient population. We focus on the consequences and limitations of randomization procedures in order to clarify the distinctions between pairs of complementary concepts of fundamental importance to data science and RCT interpretation. These include internal and external validity, generalizability and transportability, uncertainty and variability, representativeness and inclusiveness, blocking and stratification, relevance and robustness, forward and reverse causal inference, intention to treat and per protocol analyses, and potential outcomes and counterfactuals.

Estimating the area under the ROC curve when transporting a prediction model to a target population

We propose methods for estimating the area under the receiver operating characteristic (ROC) curve (AUC) of a prediction model in a target population that differs from the source population that provided the data used for original model development. If covariates that are associated with model performance, as measured by the AUC, have a different distribution in the source and target populations, then AUC estimators that only use data from the source population will not reflect model performance in the target population. Here, we provide identification results for the AUC in the target population when outcome and covariate data are available from the sample of the source population, but only covariate data are available from the sample of the target population. In this setting, we propose three estimators for the AUC in the target population and show that they are consistent and asymptotically normal. We evaluate the finite-sample performance of the estimators using simulations and use them to estimate the AUC in a nationally representative target population from the National Health and Nutrition Examination Survey for a lung cancer risk prediction model developed using source population data from the National Lung Screening Trial.

Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population

Extending (i.e., generalizing or transporting) causal inferences from a randomized trial to a target population requires assumptions that randomized and nonrandomized individuals are exchangeable conditional on baseline covariates. These assumptions are made on the basis of background knowledge, which is often uncertain or controversial, and need to be subjected to sensitivity analysis. We present simple methods for sensitivity analyses that directly parameterize violations of the assumptions using bias functions and do not require detailed background knowledge about specific unknown or unmeasured determinants of the outcome or modifiers of the treatment effect. We show how the methods can be applied to non-nested trial designs, where the trial data are combined with a separately obtained sample of nonrandomized individuals, as well as to nested trial designs, where the trial is embedded within a cohort sampled from the target population.

Efficient and robust methods for causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a target population

We present methods for causally interpretable meta-analyses that combine information from multiple randomized trials to draw causal inferences for a target population of substantive interest. We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results for transporting causal inferences from a collection of independent randomized trials to a new target population in which experimental data may not be available. We propose an estimator for the potential outcome mean in the target population under each treatment studied in the trials. The estimator uses covariate, treatment, and outcome data from the collection of trials, but only covariate data from the target population sample. We show that it is doubly robust in the sense that it is consistent and asymptotically normal when at least one of the models it relies on is correctly specified. We study the finite sample properties of the estimator in simulation studies and demonstrate its implementation using data from a multicenter randomized trial.

Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?

Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods-lasso regression and Bayesian Additive Regression Trees (BART)-using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modelling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.

How Effect Measure Choice Influences Minimally Sufficient Adjustment Sets for External Validity

Epidemiologic researchers generalizing or transporting effect estimates from a study to a target must account for effect measure modifiers (EMMs) on the scale of interest. Little attention is paid to how the EMMs required may vary depending on the mathematical nuances of each effect measure, however. We defined two types of EMM: marginal EMM, where the effect on the scale of interest differs across levels of a variable; and conditional EMM, where the effect differs conditional on other variables associated with the outcome. These types define three classes of variables: Class 1 (conditional EMM), Class 2 (marginal, but not conditional, EMM), or Class 3 (neither marginal nor conditional EMM). Class 1 variables are necessary to achieve a valid estimate of the RD in a target, while a RR requires Class 1 and Class 2 and an OR requires Class 1, Class 2, and Class 3 (i.e., all variables associated with the outcome). This does not mean that fewer variables are required for an externally valid RD (because variables may not modify effects on all scale) but does suggest researchers should consider the scale of the effect measure when identifying EMM necessary for an externally valid treatment effect estimate.

Using Trial and Observational Data to Assess Effectiveness: Trial Emulation, Transportability, Benchmarking, and Joint Analysis

Comparisons between randomized trial analyses and observational analyses that attempt to address similar research questions have generated many controversies in epidemiology and the social sciences. There has been little consensus on when such comparisons are reasonable, what their implications are for the validity of observational analyses, or whether trial and observational analyses can be integrated to address effectiveness questions. Here, we consider methods for using observational analyses to complement trial analyses when assessing treatment effectiveness. First, we review the framework for designing observational analyses that emulate target trials and present an evidence map of its recent applications. We then review approaches for estimating the average treatment effect in the target population underlying the emulation: using observational analyses of the emulation data alone; and using transportability analyses to extend inferences from a trial to the target population. We explain how comparing treatment effect estimates from the emulation against those from the trial can provide evidence on whether observational analyses can be trusted to deliver valid estimates of effectiveness - a process we refer to as benchmarking - and, in some cases, allow the joint analysis of the trial and observational data. We illustrate different approaches using a simplified example of a pragmatic trial and its emulation in registry data. We conclude that synthesizing trial and observational data - in transportability, benchmarking, or joint analyses - can leverage their complementary strengths to enhance learning about comparative effectiveness, through a process combining quantitative methods and epidemiological judgements.

Transporting a Prediction Model for Use in a New Target Population

We considered methods for transporting a prediction model for use in a new target population, both when outcome and covariate data for model development are available from a source population that has a different covariate distribution compared with the target population and when covariate data (but not outcome data) are available from the target population. We discuss how to tailor the prediction model to account for differences in the data distribution between the source population and the target population. We also discuss how to assess the model's performance (e.g., by estimating the mean squared prediction error) in the target population. We provide identifiability results for measures of model performance in the target population for a potentially misspecified prediction model under a sampling design where the source and the target population samples are obtained separately. We introduce the concept of prediction error modifiers that can be used to reason about tailoring measures of model performance to the target population. We illustrate the methods in simulated data and apply them to transport a prediction model for lung cancer diagnosis from the National Lung Screening Trial to the nationally representative target population of trial-eligible individuals in the National Health and Nutrition Examination Survey.

Experimental Study on Performance Optimization of Grouting Backfill Material Based on Mechanically Ground Coal Gangue Utilizing Urea and Quicklime

Previously conducted studies have established that grouting backfill in mining-induced overburden bed separation and mined-out areas with broken rocks provides an efficient strategy to control strata movement and surface subsidence caused by underground mining. Grouting backfill materials (GBMs) based on coal gangue (CG) are highly desirable in coal mining for accessibility to abundant CG and urgent demand for CG disposal. However, CG is generally employed as coarse aggregate due to rather rigid and inert properties, limiting its application in GBMs. Herein, to reduce reliance on fine aggregates, such as fly ash and clay, cemented GBM formulations using ground CG powder as a dominant component were proposed. Urea and quicklime were utilized as additives to optimize slurry transportability and compressive strength. Besides typical grinding without additives, CG powder was also prepared via grinding with urea, intending to enhance the hydrogen bonding (HB) interaction between urea and minerals contained in CG. The effect of grinding time and urea on CG particle size and phase composition was investigated. Then, the dependence of slurry transportability and compressive strength on grinding time, solid concentration, urea, and quicklime dosage were revealed. It has been experimentally proved that grinding for 30~90 min significantly decreased CG particle size and even induced crystal deformation of dolomite and kaolinite. For GBMs, urea improved slurry flowability, possibly caused by decreased water absorption on the CG surface and the release of water encapsulated in hydrated cement particles. Moreover, quicklime strengthened GBM bodies, which could be explained by an accelerated pozzolanic reaction between CG powder and additional CH supplied by quicklime hydration. G60U3-based GBM-B2 with 5% quicklime provided a stable and smooth slurry with a bleeding rate of 1.25%, a slump flow of 205 mm, and a hardened body with a seven-day UCS of 1.51 MPa.

A cautionary note on the use of G-computation in population adjustment

In a recent issue of the Journal; Remiro-Azócar et al. introduce a new method to adjust for population difference between two trials; when the individual patient data (IPD) are only accessible for one study. The proposed method generates the covariate data for the trial without IPD; then using a G-computation approach to transport information about the treatment effect from the other study with IPD to this trial. The authors advocate the use of G-computation over matching-adjusted indirect comparison because (i) the former allows for "useful extrapolation" when there is poor case-mix overlap between populations; and (ii) nonparametric; data-adaptive methods can be used to reduce the risk of (outcome) model misspecification. In this commentary; we provide a different perspective from these arguments. Despite certain disagreements; we believe that the proposed data generation approaches can open new and interesting research directions for population adjustment methodology in the future.

Data Integration in Causal Inference

Integrating data from multiple heterogeneous sources has become increasingly popular to achieve a large sample size and diverse study population. This paper reviews development in causal inference methods that combines multiple datasets collected by potentially different designs from potentially heterogeneous populations. We summarize recent advances on combining randomized clinical trial with external information from observational studies or historical controls, combining samples when no single sample has all relevant variables with application to two-sample Mendelian randomization, distributed data setting under privacy concerns for comparative effectiveness and safety research using real-world data, Bayesian causal inference, and causal discovery methods.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 10

In this latest update we discuss the transportability of comparative effectiveness evidence across countries. We highlight results of a survey indicating that European HTA agencies are reluctant to accept real-world data from other countries, review recent benefit assessments indicating a potential softening of a requirement for the use of local real-world data in Germany, and outline a recent review presenting approaches that can correct for a lack of transportability.

A calibration approach to transportability and data-fusion with observational data

Two important considerations in clinical research studies are proper evaluations of internal and external validity. While randomized clinical trials can overcome several threats to internal validity, they may be prone to poor external validity. Conversely, large prospective observational studies sampled from a broadly generalizable population may be externally valid, yet susceptible to threats to internal validity, particularly confounding. Thus, methods that address confounding and enhance transportability of study results across populations are essential for internally and externally valid causal inference, respectively. These issues persist for another problem closely related to transportability known as data-fusion. We develop a calibration method to generate balancing weights that address confounding and sampling bias, thereby enabling valid estimation of the target population average treatment effect. We compare the calibration approach to two additional doubly robust methods that estimate the effect of an intervention on an outcome within a second, possibly unrelated target population. The proposed methodologies can be extended to resolve data-fusion problems that seek to evaluate the effects of an intervention using data from two related studies sampled from different populations. A simulation study is conducted to demonstrate the advantages and similarities of the different techniques. We also test the performance of the calibration approach in a motivating real data example comparing whether the effect of biguanides vs sulfonylureas-the two most common oral diabetes medication classes for initial treatment-on all-cause mortality described in a historical cohort applies to a contemporary cohort of US Veterans with diabetes.

The Future of Causal Inference

The past several decades have seen exponential growth in causal inference approaches and their applications. In this commentary, we provide our top-10 list of emerging and exciting areas of research in causal inference. These include methods for high-dimensional data and precision medicine, causal machine learning, causal discovery, and others. These methods are not meant to be an exhaustive list; instead, we hope that this list will serve as a springboard for stimulating the development of new research.

Characterizing real world safety profile of oral Janus kinase inhibitors among adult atopic dermatitis patients: evidence transporting from the rheumatoid arthritis population

OBJECTIVE

To address potential safety concerns of Janus Kinase Inhibitors (JAK-Is), we characterized their safety profile in the atopic dermatitis (AD) patient population.

METHODS

In this retrospective observational study, we used propensity score-based methods and a Poisson modeling framework to estimate the incidence of health outcomes of interest (HOI) for the AD patient. To that end, two mutually exclusive cohorts were created using a real world data resource: a rheumatoid arthritis (RA) cohort, where we directly quantify the safety risk of JAK-Is on HOIs, and an AD cohort, that comprises the target population of interest and to whom we transport the results obtained from the RA cohort. The RA cohort included all adults who filled at least one prescription for a JAK-I (tofacitinib, baricitinib, or upadacitinib) between 1 January 2017 and 31 January 2020. The AD cohort consisted of all adults diagnosed with AD during the same period. We first estimated the incidence rate of each HOI in the RA cohort, and then transported the results to the AD population.

RESULTS

The RA and AD cohorts included 5,296 and 261,855 patients, respectively. On average, patients in the AD cohort were younger, more often male, more likely to be Asian, and had higher household income. They also had a lower prevalence of several comorbid conditions including hypertension, chronic kidney disease, obesity, and depression. Overall, the transported incidence rates of the HOIs to the AD cohort were lower than those obtained in the RA cohort by 13-50%.

CONCLUSION

We applied transportability methods to characterize the risk of the HOIs in the AD population and found absolute risks higher than that of the general population. Future work is needed to validate these conclusions in comparable populations.

Challenges in Systematic Reviews and Meta-Analyses of Mediation Analyses

Systematic reviews and meta-analyses of mediation studies are increasingly being implemented in practice. Nonetheless, the methodology for conducting such review and analysis is still in a development phase, with much room for improvement. In this paper, we highlight and discuss challenges that investigators face in systematic reviews and meta-analyses of mediation studies and propose ways of accommodating these in practice.

Emulating a target trial of intensive nurse home visiting in the policy-relevant population using linked administrative data

BACKGROUND

Populations willing to participate in randomized trials may not correspond well to policy-relevant target populations. Evidence of effectiveness that is complementary to randomized trials may be obtained by combining the 'target trial' causal inference framework with whole-of-population linked administrative data.

METHODS

We demonstrate this approach in an evaluation of the South Australian Family Home Visiting Program, a nurse home visiting programme targeting socially disadvantaged families. Using de-identified data from 2004-10 in the ethics-approved Better Evidence Better Outcomes Linked Data (BEBOLD) platform, we characterized the policy-relevant population and emulated a trial evaluating effects on child developmental vulnerability at 5 years (n = 4160) and academic achievement at 9 years (n = 6370). Linkage to seven health, welfare and education data sources allowed adjustment for 29 confounders using Targeted Maximum Likelihood Estimation (TMLE) with SuperLearner. Sensitivity analyses assessed robustness to analytical choices.

RESULTS

We demonstrated how the target trial framework may be used with linked administrative data to generate evidence for an intervention as it is delivered in practice in the community in the policy-relevant target population, and considering effects on outcomes years down the track. The target trial lens also aided in understanding and limiting the increased measurement, confounding and selection bias risks arising with such data. Substantively, we did not find robust evidence of a meaningful beneficial intervention effect.

CONCLUSIONS

This approach could be a valuable avenue for generating high-quality, policy-relevant evidence that is complementary to trials, particularly when the target populations are multiply disadvantaged and less likely to participate in trials.

Improving trial generalizability using observational studies

Complementary features of randomized controlled trials (RCTs) and observational studies (OSs) can be used jointly to estimate the average treatment effect of a target population. We propose a calibration weighting estimator that enforces the covariate balance between the RCT and OS, therefore improving the trial-based estimator's generalizability. Exploiting semiparametric efficiency theory, we propose a doubly robust augmented calibration weighting estimator that achieves the efficiency bound derived under the identification assumptions. A nonparametric sieve method is provided as an alternative to the parametric approach, which enables the robust approximation of the nuisance functions and data-adaptive selection of outcome predictors for calibration. We establish asymptotic results and confirm the finite sample performances of the proposed estimators by simulation experiments and an application on the estimation of the treatment effect of adjuvant chemotherapy for early-stage non-small cell lung patients after surgery. This article is protected by copyright. All rights reserved.

A Bayesian approach for estimating the partial potential impact fraction with exposure measurement error under a main study/internal validation design

The partial potential impact fraction describes the proportion of disease cases that can be prevented if the distribution of modifiable continuous exposures is shifted in a population, while other risk factors are not modified. It is a useful quantity for evaluating the burden of disease in epidemiologic and public health studies. When exposures are measured with error, the partial potential impact fraction estimates may be biased, which necessitates methods to correct for the exposure measurement error. Motivated by the health professionals follow-up study, we develop a Bayesian approach to adjust for exposure measurement error when estimating the partial potential impact fraction under the main study/internal validation study design. We adopt the reclassification approach that leverages the strength of the main study/internal validation study design and clarifies transportability assumptions for valid inference. We assess the finite-sample performance of both the point and credible interval estimators via extensive simulations and apply the proposed approach in the health professionals follow-up study to estimate the partial potential impact fraction for colorectal cancer incidence under interventions exploring shifting the distributions of red meat, alcohol, and/or folate intake.

Transportability From Randomized Trials to Clinical Care: On Initial HIV Treatment With Efavirenz and Suicidal Thoughts or Behaviors

In an analysis of randomized trials, use of efavirenz for treatment of human immunodeficiency virus (HIV) infection was associated with increased suicidal thoughts/behaviors. However, analyses of observational data have found no evidence of increased risk. To assess whether population differences might explain this divergence, we transported the effect of efavirenz use from these trials to a specific target population. Using inverse odds weights and multiple imputation, we transported the effect of efavirenz on suicidal thoughts/behaviors in these randomized trials (participants were enrolled in 2001-2007) to a trials-eligible cohort of US adults initiating antiretroviral therapy while receiving HIV clinical care at medical centers between 1999 and 2015. Overall, 8,291 cohort participants and 3,949 trial participants were eligible. Prescription of antidepressants (19% vs. 13%) and injection drug history (16% vs. 10%) were more frequent in the cohort than in the trial participants. Compared with the effect in trials, the estimated hazard ratio for efavirenz on suicidal thoughts/behaviors was attenuated in our target population (trials: hazard ratio (HR) = 2.3 (95% confidence interval (CI): 1.2, 4.4); transported: HR = 1.8 (95% CI: 0.9, 4.4)), whereas the incidence rate difference was similar (trials: HR = 5.1 (95% CI: 1.6, 8.7); transported: HR = 5.4 (95% CI: -0.4, 11.4)). In our target population, there was greater than 20% attenuation of the hazard ratio estimate as compared with the trials-only estimate. Transporting results from trials to a target population is informative for addressing external validity.

Study Designs for Extending Causal Inferences From a Randomized Trial to a Target Population

In this article, we examine study designs for extending (generalizing or transporting) causal inferences from a randomized trial to a target population. Specifically, we consider nested trial designs, where randomized individuals are nested within a sample from the target population, and nonnested trial designs, including composite data-set designs, where observations from a randomized trial are combined with those from a separately obtained sample of nonrandomized individuals from the target population. We show that the counterfactual quantities that can be identified in each study design depend on what is known about the probability of sampling nonrandomized individuals. For each study design, we examine identification of counterfactual outcome means via the g-formula and inverse probability weighting. Last, we explore the implications of the sampling properties underlying the designs for the identification and estimation of the probability of trial participation.

Transporting Subgroup Analyses of Randomized Controlled Trials for Planning Implementation of New Interventions

Subgroup analyses of randomized controlled trials guide resource allocation and implementation of new interventions by identifying groups of individuals who are likely to benefit most from the intervention. Unfortunately, trial populations are rarely representative of the target populations of public health or clinical interest. Unless the relevant differences between trial and target populations are accounted for, subgroup results from trials might not reflect which groups in the target population will benefit most from the intervention. Transportability provides a rigorous framework for applying results derived in potentially highly selected study populations to external target populations. The method requires that researchers measure and adjust for all variables that 1) modify the effect of interest and 2) differ between the target and trial populations. To date, applications of transportability have focused on the external validity of overall study results and understanding within-trial heterogeneity; however, this approach has not yet been used for subgroup analyses of trials. Through an example from the Iniciativa Profilaxis Pre-Exposición (iPrEx) study (multiple countries, 2007-2010) of preexposure prophylaxis for human immunodeficiency virus, we illustrate how transporting subgroup analyses can produce target-specific subgroup effect estimates and numbers needed to treat. This approach could lead to more tailored and accurate guidance for resource allocation and cost-effectiveness analyses.

Efficient, doubly robust estimation of the effect of dose switching for switchers in a randomized clinical trial

Motivated by a clinical trial conducted by Janssen Pharmaceutica in which a flexible dosing regimen is compared to placebo, we evaluate how switchers in the treatment arm (i.e., patients who were switched to the higher dose) would have fared had they been kept on the low dose. This is done in order to understand whether flexible dosing is potentially beneficial for them. Simply comparing these patients' responses with those of patients who stayed on the low dose does not likely entail a satisfactory evaluation because the latter patients are usually in a better health condition. Because the available information in the considered trial is too limited to enable a reliable adjustment, we will instead transport data from a fixed dosing trial that has been conducted concurrently on the same target, albeit not in an identical patient population. In particular, we propose an estimator that relies on an outcome model, a model for switching, and a propensity score model for the association between study and patient characteristics. The proposed estimator is asymptotically unbiased if either the outcome or the propensity score model is correctly specified, and efficient (under the semiparametric model where the randomization probabilities are known and independent of baseline covariates) when all models are correctly specified. The proposed method for transporting information from an external study is more broadly applicable in studies where a classical confounding adjustment is not possible due to near positivity violation (e.g., studies where switching takes place in a (near) deterministic manner). Monte Carlo simulations and application to the motivating study demonstrate adequate performance.

Fusion designs and estimators for treatment effects

While randomized trials remain the best evidence for treatment effectiveness, lack of generalizability often remains an important concern. Additionally, when new treatments are compared against existing standards of care, the potentially small benefit of the new treatment may be difficult to detect in a trial without extremely large sample sizes and long follow-up times. Recent advances in "data fusion" provide a framework to combine results across studies that are applicable to a given population of interest and allow treatment comparisons that may not be feasible with traditional study designs. We propose a data fusion-based estimator that can be used to combine information from two studies: (1) a study comparing a new treatment to the standard of care in the local population of interest, and (2) a study comparing the standard of care to placebo in a separate, distal population. We provide conditions under which the parameter of interest can be identified from the two studies described and explore properties of the estimator through simulation. Finally, we apply the estimator to estimate the effect of triple- vs monotherapy for the treatment of HIV using data from two randomized trials. The proposed estimator can account for underlying population structures that induce differences in case mix, adherence, and outcome prevalence between the local and distal populations, and the estimator can also account for potentially informative loss to follow-up. Approaches like those detailed here are increasingly important to speed the approval and adoption of effective new therapies by leveraging multiple sources of information.

Implementation of the COVID-19 Vulnerability Index Across an International Network of Health Care Data Sets: Collaborative External Validation Study

Background

SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who require hospitalization from those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision-making during the pandemic. However, the model is at high risk of bias according to the “prediction model risk of bias assessment” criteria, and it has not been externally validated.

Objective

The aim of this study was to externally validate the C-19 index across a range of health care settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases.

Methods

We followed the Observational Health Data Sciences and Informatics (OHDSI) framework for external validation to assess the reliability of the C-19 index. We evaluated the model on two different target populations, 41,381 patients who presented with SARS-CoV-2 at an outpatient or emergency department visit and 9,429,285 patients who presented with influenza or related symptoms during an outpatient or emergency department visit, to predict their risk of hospitalization with pneumonia during the following 0-30 days. In total, we validated the model across a network of 14 databases spanning the United States, Europe, Australia, and Asia.

Results

The internal validation performance of the C-19 index had a C statistic of 0.73, and the calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data, the model obtained C statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US, and South Korean data sets, respectively. The calibration was poor, with the model underestimating risk. When validated on 12 data sets containing influenza patients across the OHDSI network, the C statistics ranged between 0.40 and 0.68.

Conclusions

Our results show that the discriminative performance of the C-19 index model is low for influenza cohorts and even worse among patients with COVID-19 in the United States, Spain, and South Korea. These results suggest that C-19 should not be used to aid decision-making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.

Transporting stochastic direct and indirect effects to new populations

Transported mediation effects may contribute to understanding how interventions work differently when applied to new populations. However, we are not aware of any estimators for such effects. Thus, we propose two doubly robust, efficient estimators of transported stochastic (also called randomized interventional) direct and indirect effects. We demonstrate their finite sample properties in a simulation study. We then apply the preferred substitution estimator to longitudinal data from the Moving to Opportunity Study, a large-scale housing voucher experiment, to transport stochastic indirect effect estimates of voucher receipt in childhood on subsequent risk of mental health or substance use disorder mediated through parental employment across sites, thereby gaining understanding of drivers of the site differences.

Calibrating validation samples when accounting for measurement error in intervention studies

Many lifestyle intervention trials depend on collecting self-reported outcomes, such as dietary intake, to assess the intervention's effectiveness. Self-reported outcomes are subject to measurement error, which impacts treatment effect estimation. External validation studies measure both self-reported outcomes and accompanying biomarkers, and can be used to account for measurement error. However, in order to account for measurement error using an external validation sample, an assumption must be made that the inferences are transportable from the validation sample to the intervention trial of interest. This assumption does not always hold. In this paper, we propose an approach that adjusts the validation sample to better resemble the trial sample, and we also formally investigate when bias due to poor transportability may arise. Lastly, we examine the performance of the methods using simulation, and illustrate them using PREMIER, a lifestyle intervention trial measuring self-reported sodium intake as an outcome, and OPEN, a validation study measuring both self-reported diet and urinary biomarkers.

Generalizing randomized trial findings to a target population using complex survey population data

Randomized trials are considered the gold standard for estimating causal effects. Trial findings are often used to inform policy and programming efforts, yet their results may not generalize well to a relevant target population due to potential differences in effect moderators between the trial and population. Statistical methods have been developed to improve generalizability by combining trials and population data, and weighting the trial to resemble the population on baseline covariates. Large-scale surveys in fields such as health and education with complex survey designs are a logical source for population data; however, there is currently no best practice for incorporating survey weights when generalizing trial findings to a complex survey. We propose and investigate ways to incorporate survey weights in this context. We examine the performance of our proposed estimator through simulations in comparison to estimators that ignore the complex survey design. We then apply the methods to generalize findings from two trials-a lifestyle intervention for blood pressure reduction and a web-based intervention to treat substance use disorders-to their respective target populations using population data from complex surveys. The work highlights the importance in properly accounting for the complex survey design when generalizing trial findings to a population represented by a complex survey sample.

Extending inferences from a randomized trial to a new target population

When treatment effect modifiers influence the decision to participate in a randomized trial, the average treatment effect in the population represented by the randomized individuals will differ from the effect in other populations. In this tutorial, we consider methods for extending causal inferences about time-fixed treatments from a trial to a new target population of nonparticipants, using data from a completed randomized trial and baseline covariate data from a sample from the target population. We examine methods based on modeling the expectation of the outcome, the probability of participation, or both (doubly robust). We compare the methods in a simulation study and show how they can be implemented in software. We apply the methods to a randomized trial nested within a cohort of trial-eligible patients to compare coronary artery surgery plus medical therapy versus medical therapy alone for patients with chronic coronary artery disease. We conclude by discussing issues that arise when using the methods in applied analyses.

Estimating treatment importance in multidrug-resistant tuberculosis using Targeted Learning: An observational individual patient data network meta-analysis

Persons with multidrug-resistant tuberculosis (MDR-TB) have a disease resulting from a strain of tuberculosis (TB) that does not respond to at least isoniazid and rifampicin, the two most effective anti-TB drugs. MDR-TB is always treated with multiple antimicrobial agents. Our data consist of individual patient data from 31 international observational studies with varying prescription practices, access to medications, and distributions of antibiotic resistance. In this study, we develop identifiability criteria for the estimation of a global treatment importance metric in the context where not all medications are observed in all studies. With stronger causal assumptions, this treatment importance metric can be interpreted as the effect of adding a medication to the existing treatments. We then use this metric to rank 15 observed antimicrobial agents in terms of their estimated add-on value. Using the concept of transportability, we propose an implementation of targeted maximum likelihood estimation, a doubly robust and locally efficient plug-in estimator, to estimate the treatment importance metric. A clustered sandwich estimator is adopted to compute variance estimates and produce confidence intervals. Simulation studies are conducted to assess the performance of our estimator, verify the double robustness property, and assess the appropriateness of the variance estimation approach.

Correcting for selective participation in cohort studies using auxiliary register data without identification of non-participants

Aims: Selective participation may hamper the validity of population-based cohort studies. The resulting bias can be alleviated by linking auxiliary register data to both the participants and the non-participants of the study, estimating propensity scores for participation and correcting for participation based on these. However, registry holders may not be allowed to disclose sensitive data on (invited) non-participants. Our aim is to provide guidance on how adequate bias correction can be achieved by using auxiliary register data but without disclosing information that could be linked to the subset of non-participants. Methods: We show how existing methods can be used to estimate generalisation weights under various data disclosure scenarios where invited non-participants are indistinguishable from uninvited ones. We also demonstrate how the methods can be implemented using Nordic register data. Results: Inverse-probability-of-sampling weights estimated within a random sample of the target population in which the non-respondents are disclosed are equivalent in expectation to analogous weights in a scenario where the non-participants and uninvited individuals from the population are indistinguishable. To minimise the risk of disclosure when the entire population is invited to participate, investigators should instead consider inverse-odds-of-sampling weights, a method that has previously been suggested for transporting study results to external populations. Conclusions: Generalisation weights can be estimated from auxiliary register data without disclosing information on invited non-participants.

A novel approach for identifying and addressing case-mix heterogeneity in individual participant data meta-analysis

Case-mix heterogeneity across studies complicates meta-analyses. As a result of this, treatments that are equally effective on patient subgroups may appear to have different effectiveness on patient populations with different case mix. It is therefore important that meta-analyses be explicit for what patient population they describe the treatment effect. To achieve this, we develop a new approach for meta-analysis of randomized clinical trials, which use individual patient data (IPD) from all trials to infer the treatment effect for the patient population in a given trial, based on direct standardization using either outcome regression (OCR) or inverse probability weighting (IPW). Accompanying random-effect meta-analysis models are developed. The new approach enables disentangling heterogeneity due to case mix from that due to beyond case-mix reasons.

Generalizing Intensive Blood Pressure Treatment to Adults With Diabetes Mellitus

BACKGROUND

Controversy over blood pressure (BP) treatment targets for individuals with diabetes is in part due to conflicting perspectives about generalizability of available trial data.

OBJECTIVE

The authors sought to estimate how results from the largest clinical trial of intensive BP treatment among adults with diabetes would generalize to the U.S.

POPULATION

METHODS

The authors used transportability methods to reweight individual patient data from the ACCORD (Action to Control Cardiovascular Risk in Diabetes) BP trial (N = 4,507) of intensive (goal systolic BP <120 mm Hg) versus standard (goal systolic BP <140 mm Hg) treatment to better represent the demographic and clinical risk factors of the U.S. population of adults with diabetes (data from NHANES [National Health and Nutrition Examination Survey] 2005 to 2014, n = 1,943). The primary outcome was the first occurrence of nonfatal myocardial infarction, nonfatal stroke, or cardiovascular death. Analysis used weighted Cox proportional hazards regression models with robust standard errors.

RESULTS

The ACCORD BP sample had less racial/ethnic diversity and more elevated cardiovascular risk factors than the NHANES participants. Weighted results significantly favored intensive BP treatment, unlike unweighted results (hazard ratio for primary outcome in intensive versus standard treatment in weighted analyses: 0.67, 95% confidence interval: 0.49 to 0.91; in unweighted analyses: hazard ratio: 0.88, 95% confidence interval: 0.73 to 1.07). Over 5 years, the weighted results estimate a number needed to treat of 34, and number needed to harm of 55.

CONCLUSIONS

After reweighting to better reflect the U.S. adult population with diabetes, intensive BP therapy was associated with significantly lower risk for cardiovascular events. However, data were limited among racial/ethnic minorities and those with lower cardiovascular risk.

Lessons Learned About Barriers to Implementing School-Based Interventions for Adolescents: Ideas for Enhancing Future Research and Clinical Projects

The majority of youth with mental health problems do not receive treatment, highlighting the critical need to transport evidence-based interventions into community settings, such as schools. Despite being able to reach a large number of adolescents and minority youth, the process of implementing evidence-based interventions to schools is challenging. This paper discusses some expected and unexpected challenges experienced during the implementation of an open trial and a pilot randomized controlled trial examining the acceptability and effectiveness of a school-based preventive intervention for adolescents at risk for internalizing disorders. First, we highlight key programs and findings on preventive interventions for adolescents at risk for depression and anxiety. Next, we provide a brief overview of the preventive intervention we implemented in schools. This provides a context for the section that describes implementation issues and highlights specific challenges and potential solutions for intervention implementation. Finally, the paper offers recommendations for researchers and clinicians interested in implementing school-based mental health services for adolescents.

Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals

We consider methods for causal inference in randomized trials nested within cohorts of trial-eligible individuals, including those who are not randomized. We show how baseline covariate data from the entire cohort, and treatment and outcome data only from randomized individuals, can be used to identify potential (counterfactual) outcome means and average treatment effects in the target population of all eligible individuals. We review identifiability conditions, propose estimators, and assess the estimators' finite-sample performance in simulation studies. As an illustration, we apply the estimators in a trial nested within a cohort of trial-eligible individuals to compare coronary artery bypass grafting surgery plus medical therapy vs. medical therapy alone for chronic coronary artery disease.

Diagnostic Assessment of Assumptions for External Validity: An Example Using Data in Metastatic Colorectal Cancer

BACKGROUND

Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods.

METHODS

We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years.

RESULTS

In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95% confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95% CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95% CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95% CI = -0.09, 0.09).

CONCLUSION

These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.

Composition or Context: Using Transportability to Understand Drivers of Site Differences in a Large-scale Housing Experiment

BACKGROUND

The Moving To Opportunity (MTO) experiment manipulated neighborhood context by randomly assigning housing vouchers to volunteers living in public housing to use to move to lower poverty neighborhoods in five US cities. This random assignment overcomes confounding limitations that challenge other neighborhood studies. However, differences in MTO's effects across the five cities have been largely ignored. Such differences could be due to population composition (e.g., differences in the racial/ethnic distribution) or to context (e.g., differences in the economy).

METHODS

Using a nonparametric omnibus test and a multiply robust, semiparametric estimator for transportability, we assessed the extent to which differences in individual-level compositional characteristics that may act as effect modifiers can account for differences in MTO's effects across sites. We examined MTO's effects on marijuana use, behavioral problems, major depressive disorder, and generalized anxiety disorder among black and Latino adolescent males, where housing voucher receipt was harmful for health in some sites but beneficial in others.

RESULTS

Comparing point estimates, differences in composition partially explained site differences in MTO effects on marijuana use and behavioral problems but did not explain site differences for major depressive disorder or generalized anxiety disorder.

CONCLUSIONS

Our findings provide quantitative, rigorous evidence for the importance of context or unmeasured individual-level compositional variables in modifying MTO's effects.