Comparison of Methods to Generalize Randomized Clinical Trial Results Without Individual-Level Data for the Target Population

A Two-Step Framework for Validating Causal Effect Estimates

BACKGROUND

Comparing causal effect estimates obtained using observational data to those obtained from the gold standard (i.e., randomized controlled trials [RCTs]) helps assess the validity of these estimates. However, comparisons are challenging due to differences between observational data and RCT generated data. The unknown treatment assignment mechanism in the observational data and varying sampling mechanisms between the RCT and the observational data can lead to confounding and sampling bias, respectively.

AIMS

The objective of this study is to propose a two-step framework to validate causal effect estimates obtained from observational data by adjusting for both mechanisms.

MATERIALS AND METHODS

An estimator of causal effects related to the two mechanisms is constructed. A two-step framework for comparing causal effect estimates is derived from the estimator. An R package RCTrep is developed to implement the framework in practice.

RESULTS

A simulation study is conducted to show that using our framework observational data can produce causal effect estimates similar to those of an RCT. A real-world application of the framework to validate treatment effects of adjuvant chemotherapy obtained from registry data is demonstrated.

CONCLUSION

This  study constructs a framework for comparing causal effect estimates between observational data and RCT data, facilitating the assessment of the validity of causal effect estimates obtained from observational data.

Learning about treatment effects in a new target population under transportability assumptions for relative effect measures

Investigators often believe that relative effect measures conditional on covariates, such as risk ratios and mean ratios, are "transportable" across populations. Here, we examine the identification of causal effects in a target population using an assumption that conditional relative effect measures are transportable from a trial to the target population. We show that transportability for relative effect measures is largely incompatible with transportability for difference effect measures, unless the treatment has no effect on average or one is willing to make even stronger transportability assumptions that imply the transportability of both relative and difference effect measures. We then describe how marginal (population-averaged) causal estimands in a target population can be identified under the assumption of transportability of relative effect measures, when we are interested in the effectiveness of a new experimental treatment in a target population where the only treatment in use is the control treatment evaluated in the trial. We extend these results to consider cases where the control treatment evaluated in the trial is only one of the treatments in use in the target population, under an additional partial exchangeability assumption in the target population (i.e., an assumption of no unmeasured confounding in the target population with respect to potential outcomes under the control treatment in the trial). We also develop identification results that allow for the covariates needed for transportability of relative effect measures to be only a small subset of the covariates needed to control confounding in the target population. Last, we propose estimators that can be easily implemented in standard statistical software and illustrate their use using data from a comprehensive cohort study of stable ischemic heart disease.

Standardizing to specific target populations in distributed networks and multisite pharmacoepidemiologic studies

Distributed network studies and multisite studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multisite studies may increase interpretability and improve precision. We simulated a 4-site study, standardized each site using inverse odds weights (IOWs) to resemble the 3 smallest sites or the smallest site, estimated IOW-weighted risk differences (RDs), and combined estimates with inverse variance weights (IVWs). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of 1 site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOWs reduced differences between estimates and increased precision when targeting the 3 smallest sites or the smallest site. In the CPRD Aurum study, the IOW + IVW estimate was also more precise (smallest region: RD = 5.41% [95% CI, 1.03-9.79]; IOW + IVW estimate: RD = 3.25% [95% CI, 3.07-3.43]). When performing pharmacoepidemiologic research in distributed networks or multisite studies in the presence of EMMs, designation of target populations has the potential to improve estimate precision and interpretability. This article is part of a Special Collection on Pharmacoepidemiology.

Entropy balancing for causal generalization with target sample summary information

In this paper, we focus on estimating the average treatment effect (ATE) of a target population when individual-level data from a source population and summary-level data (e.g., first or second moments of certain covariates) from the target population are available. In the presence of the heterogeneous treatment effect, the ATE of the target population can be different from that of the source population when distributions of treatment effect modifiers are dissimilar in these two populations, a phenomenon also known as covariate shift. Many methods have been developed to adjust for covariate shift, but most require individual covariates from a representative target sample. We develop a weighting approach based on the summary-level information from the target sample to adjust for possible covariate shift in effect modifiers. In particular, weights of the treated and control groups within a source sample are calibrated by the summary-level information of the target sample. Our approach also seeks additional covariate balance between the treated and control groups in the source sample. We study the asymptotic behavior of the corresponding weighted estimator for the target population ATE under a wide range of conditions. The theoretical implications are confirmed in simulation studies and a real-data application.

Using Randomized Controlled Trials to Estimate the Effect of Community Interventions for Childhood Asthma

INTRODUCTION

The Centers for Disease Control and Prevention's Controlling Childhood Asthma and Reducing Emergencies initiative aims to prevent 500,000 emergency department (ED) visits and hospitalizations within 5 years among children with asthma through implementation of evidence-based interventions and policies. Methods are needed for calculating the anticipated effects of planned asthma programs and the estimated effects of existing asthma programs. We describe and illustrate a method of using results from randomized control trials (RCTs) to estimate changes in rates of adverse asthma events (AAEs) that result from expanding access to asthma interventions.

METHODS

We use counterfactual arguments to justify a formula for the expected number of AAEs prevented by a given intervention. This formula employs a current rate of AAEs, a measure of the increase in access to the intervention, and the rate ratio estimated in an RCT.

RESULTS

We justified a formula for estimating the effect of expanding access to asthma interventions. For example, if 20% of patients with asthma in a community with 20,540 annual asthma-related ED visits were offered asthma self-management education, ED visits would decrease by an estimated 1,643; and annual hospitalizations would decrease from 2,639 to 617.

CONCLUSION

Our method draws on the best available evidence from RCTs to estimate effects on rates of AAEs in the community of interest that result from expanding access to asthma interventions.

Transporting monovalent rotavirus vaccine efficacy estimates to an external target population: a secondary analysis of data from a randomised controlled trial in Malawi

Oral rotavirus vaccine efficacy estimates from randomised controlled trials are highly variable across settings. Although the randomised study design increases the likelihood of internal validity of findings, results from trials may not always apply outside the context of the study due to differences between trial participants and the target population. Here, we used a weight-based method to transport results from a monovalent rotavirus vaccine clinical trial conducted in Malawi between 2005 and 2008 to a target population of all trial-eligible children in Malawi, represented by data from the 2015-2016 Malawi Demographic and Health Survey (DHS). We reweighted trial participants to reflect the population characteristics described by the Malawi DHS. Vaccine efficacy was estimated for 1008 trial participants after applying these weights such that they represented trial-eligible children in Malawi. We also conducted subgroup analyses to examine the heterogeneous treatment effects by stunting and tuberculosis vaccination status at enrolment. In the original trial, the estimates of one-year vaccine efficacy against severe rotavirus gastroenteritis and any-severity rotavirus gastroenteritis in Malawi were 49.2% (95% CI 15.6%-70.3%) and 32.1% (95% CI 2.5%-53.1%), respectively. After weighting trial participants to represent all trial-eligible children in Malawi, vaccine efficacy increased to 62.2% (95% CI 35.5%-79.0%) against severe rotavirus gastroenteritis and 38.9% (95% CI 11.4%-58.5%) against any-severity rotavirus gastroenteritis. Rotavirus vaccine efficacy may differ between trial participants and target populations when these two populations differ. Differences in tuberculosis vaccination status between the trial sample and DHS population contributed to varying trial and target population vaccine efficacy estimates.

Association of immediate reinsertion of new catheters with subsequent mortality among patients with suspected catheter infection: a cohort study

Background

Central venous catheter (CVC) insertion complications are a prevalent and important problem in the intensive care unit (ICU), and source control by immediate catheter removal is considered urgent in patients with septic shock suspected to be caused by catheter-related bloodstream infection (CRBSI). We sought to determine the impact of immediate reinsertion of a new catheter (IRINC) on mortality among patients after CVC removal for suspected CRBSI.

Methods

A propensity score-matched cohort of patients with suspected CRBSI who underwent IRINC or no IRINC in a 32-bed ICU in a university hospital in China from January 2009 through April 2021. Catheter tip culture and clinical symptoms were used to identify patients with suspected CRBSI. The Kaplan–Meier method was used to analyse 30-day mortality before and after propensity score matching, and adjusted hazard ratios (HRs) and 95% confidence intervals (CIs) for mortality in the matched cohort were estimated with Cox proportional hazards models.

Results

In total, 1,238 patients who had a CVC removed due to suspected CRBSI were identified. Among these patients, 877 (70.8%) underwent IRINC, and 361 (29.2%) did not. Among 682 propensity score-matched patients, IRINC was associated with an increased risk of 30-day mortality (HR, 1.481; 95% CI, 1.028 to 2.134) after multivariable, multilevel adjustment. Kaplan–Meier analysis found that IRINC was associated with the risk of mortality both before matching (P = 0.00096) and after matching (P = 0.018). A competing risk analysis confirmed the results of the propensity score-matched analysis. The attributable risk associated with bloodstream infection was not significantly different (HR, 1.081; 95% CI 0.964 to 1.213) among patients with suspected CRBSI in terms of 30-day mortality compared with that associated with other infections.

Conclusions

In this cohort study, IRINC was associated with higher 30-day mortality compared to delayed CVC or no CVC among patients with suspected CRBSI. A large-sample randomized controlled trial is needed to define the best management for CVC in cases of suspected CRBSI because IRINC may also be associated with noninfectious complications.

Trial registration This study was registered with the China Clinical Trials Registry (URL: http://www.chictr.org.cn/index.aspx) under the following registration number: ChiCTR1900022175.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13613-022-01014-8.

The value of combining individual and small area sociodemographic data for assessing and handling selective participation in cohort studies: Evidence from the Swedish CardioPulmonary bioImage Study

Objectives

To study the value of combining individual- and neighborhood-level sociodemographic data to predict study participation and assess the effects of baseline selection on the distribution of metabolic risk factors and lifestyle factors in the Swedish CardioPulmonary bioImage Study (SCAPIS).

Methods

We linked sociodemographic register data to SCAPIS participants (n = 30,154, ages: 50–64 years) and a random sample of the study’s target population (n = 59,909). We assessed the classification ability of participation models based on individual-level data, neighborhood-level data, and combinations of both. Standardized mean differences (SMD) were used to examine how reweighting the sample to match the population affected the averages of 32 cardiopulmonary risk factors at baseline. Absolute SMDs >0.10 were considered meaningful.

Results

Combining both individual-level and neighborhood-level data gave rise to a model with better classification ability (AUC: 71.3%) than models with only individual-level (AUC: 66.9%) or neighborhood-level data (AUC: 65.5%). We observed a greater change in the distribution of risk factors when we reweighted the participants using both individual and area data. The only meaningful change was related to the (self-reported) frequency of alcohol consumption, which appears to be higher in the SCAPIS sample than in the population. The remaining risk factors did not change meaningfully.

Conclusions

Both individual- and neighborhood-level characteristics are informative in assessing study selection effects. Future analyses of cardiopulmonary outcomes in the SCAPIS cohort can benefit from our study, though the average impact of selection on risk factor distributions at baseline appears small.

Generalizability of heterogeneous treatment effects based on causal forests applied to two randomized clinical trials of intensive glycemic control

Purpose Machine learning is an attractive tool for identifying heterogeneous treatment effects (HTE) of interventions but generalizability of machine learning derived HTE remains unclear. We examined generalizability of HTE detected using causal forests in two similarly designed randomized trials in type II diabetes patients. Methods We evaluated published HTE of intensive versus standard glycemic control on all-cause mortality from the Action to Control Cardiovascular Risk in Diabetes study (ACCORD) in a second trial, the Veterans Affairs Diabetes Trial (VADT). We then applied causal forests to VADT, ACCORD, and pooled data from both studies and compared variable importance and subgroup effects across samples. Results HTE in ACCORD did not replicate in similar subgroups in VADT, but variable importance was correlated between VADT and ACCORD (Kendall's tau-b 0.75). Applying causal forests to pooled individual-level data yielded seven subgroups with similar HTE across both studies, ranging from risk difference of all-cause mortality of -3.9% (95% CI -7.0, -0.8) to 4.7% (95% CI 1.8, 7.5). Conclusions Machine learning detection of HTE subgroups from randomized trials may not generalize across study samples even when variable importance is correlated. Pooling individual-level data may overcome differences in study populations and/or differences in interventions that limit HTE generalizability.

Transposition of cardiovascular outcome trial effects to the real-world population of patients with type 2 diabetes

Background

Transferring results obtained in cardiovascular outcome trials (CVOTs) to the real-world setting is challenging. We herein transposed CVOT results to the population of patients with type 2 diabetes (T2D) seen in routine clinical practice and who may receive the medications tested in CVOTs.

Methods

We implemented the post-stratification approach based on aggregate data of CVOTs and individual data of a target population of diabetic outpatients. We used stratum-specific estimates available from CVOTs to calculate expected effect size for the target population by weighting the average of the stratum-specific treatment effects according to proportions of a given characteristic in the target population. Data are presented as hazard ratio (HR) and 95% confidence intervals.

Results

Compared to the target population (n = 139,708), the CVOT population (n = 95,816) was younger and had a two to threefold greater prevalence of cardiovascular disease. EMPA-REG was the CVOT with the largest variety of details on stratum-specific effects, followed by TECOS, whereas DECLARE and PIONEER-6 had more limited stratum-specific information. The post-stratification HR estimate for 3 point major adverse cardiovascular event (MACE) based on EMPA-REG was 0.88 (0.74–1.03) in the target population, compared to 0.86 (0.74–0.99) in the trial. The HR estimate based on LEADER was 0.88 (0.77–0.99) in the target population compared to 0.87 (0.78–0.97) in the trial. Consistent results were obtained for SUSTAIN-6, EXSCEL, PIONEER-6 and DECLARE. The effect of DPP-4 inhibitors observed in CVOTs remained neutral in the target population.

Conclusions

Based on CVOT stratum-specific effects, cardiovascular protective actions of glucose lowering medications tested in CVOTs are transferrable to a much different real-world population of patients with T2D.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12933-021-01300-y.

The findings of a surgical hip fracture trial were generalizable to the UK national hip fracture database

OBJECTIVE

To estimate the generalizability of treatment effects observed in a randomized trial of hip fracture surgery implants to a broader population of people undergoing hip surgery in the United Kingdom.

STUDY DESIGN AND SETTING

In 2018, the WHiTE-3 trial (n = 958) demonstrated that a modular hemiarthroplasty implant conferred no additional benefit over the traditional monoblock implant for quality of life and length of hospital stay. We compared and weighted the trial sample against two target populations: WHiTE-cohort (n = 2,457) and UK-National Hip Fracture Database (NHFD, n = 190,894), and re-estimate expected treatment effects for the target populations.

RESULTS

Despite differences in baseline characteristics of the trial sample and target populations, the re-estimated treatment effects were comparable. For quality of life, the differences between the trial estimate and WHiTE-cohort and NHFD estimates were 0.01 points on the EuroQol (EQ5D). For length of stay, the difference between the trial estimate and WHiTE-cohort was 0.50 days; and the difference between the trial estimate and NHFD estimate was -0.47 days.

CONCLUSION

This generalizability analysis of the WHiTE-3 trial found that the inferences from the trial can be generalized to a wider population of individuals in the UK NHFD and the WHiTE-cohort who met the inclusion criteria for WHiTE-3.

Effectiveness of adjuvant FOLFOX vs 5FU/LV in adults over age 65 with stage II and III colon cancer using a novel hybrid approach

PURPOSE

Estimates of cancer therapy effects can differ in clinical trials and clinical practice, partly due to underrepresentation of certain patient subgroups in trials. We utilize a hybrid approach, combining clinical trial and real-world data, to estimate the comparative effectiveness of two adjuvant chemotherapy regimens for colon cancer.

METHODS

We identified patients aged 66 and older enrolled in the Multicenter International Study of Oxaliplatin/5FU-LV in the Adjuvant Treatment of Colon Cancer. Similar patients were identified in the Surveillance, Epidemiology, and End Results (SEER)-Medicare database, initiating adjuvant chemotherapy with either 5-fluorouracil (5FU) alone or in combination with oxaliplatin (FOLFOX). We used logistic regression to estimate the likelihood of trial enrollment as a function of age, sex, and substage. Using inverse odds of sampling weights (IOSW), we compared 5-year mortality in patients randomized to FOLFOX vs 5FU using weighted Cox proportional hazards regression, the Nelson-Aalen estimator for cumulative hazards, and bootstrapping for 95% confidence intervals (CIs).

RESULTS

There were 690 trial participants and 3834 SEER-Medicare patients. The SEER-Medicare population was older and had a higher proportion of stage IIIB and IIIC patients than the trial. After controlling for differences between populations, the IOSW 5-year HR was 1.21 (0.89, 1.65), slightly farther from the null than the trial estimate (HR = 1.14, 95%CI: 0.87, 1.49).

CONCLUSIONS

This study supports mounting evidence of little to no incremental reduction in 5-year mortality for FOLFOX vs 5FU in older adults with stage II-III colon cancer, emphasizing the importance of combining clinical trial and real-world data to support such conclusions.

Reweighting Oranges to Apples: Transported RE-LY Trial Versus Nonexperimental Effect Estimates of Anticoagulation in Atrial Fibrillation

BACKGROUND

Results from trials and nonexperimental studies are often directly compared, with little attention paid to differences between study populations. When target and trial population data are available, accounting for these differences through transporting trial results to target populations of interest provides useful perspective. We aimed to compare two-year risk differences (RDs) for ischemic stroke, mortality, and gastrointestinal bleeding in older adults with atrial fibrillation initiating dabigatran and warfarin when using trial transport methods versus nonexperimental methods.

METHODS

We identified Medicare beneficiaries who initiated warfarin or dabigatran from a 20% nationwide sample. To transport treatment effects observed in the randomized evaluation of long-term anticoagulation trial, we applied inverse odds weights to standardize estimates to two Medicare target populations of interest, initiators of: (1) dabigatran and (2) warfarin. Separately, we conducted a nonexperimental study in the Medicare populations using standardized morbidity ratio weighting to control measured confounding.

RESULTS

Comparing dabigatran to warfarin, estimated two-year RDs for ischemic stroke were similar with trial transport and nonexperimental methods. However, two-year mortality RDs were closer to the null when using trial transport versus nonexperimental methods for the dabigatran target population (transported RD: -0.57%; nonexperimental RD: -1.9%). Estimated gastrointestinal bleeding RDs from trial transport (dabigatran initiator RD: 1.8%; warfarin initiator RD: 1.9%) appeared more harmful than nonexperimental results (dabigatran initiator RD: 0.14%; warfarin initiator RD: 0.57%).

CONCLUSIONS

Differences in study populations can and should be considered quantitatively to ensure results are relevant to populations of interest, particularly when comparing trial with nonexperimental findings. See video abstract: http://links.lww.com/EDE/B703.

Reweighting Randomized Controlled Trial Evidence to Better Reflect Real Life - A Case Study of the Innovative Medicines Initiative

Evidence from randomized controlled trials available for timely health technology assessments of new pharmacological treatments and regulatory decision making may not be generalizable to local patient populations, often resulting in decisions being made under uncertainty. In recent years, several reweighting approaches have been explored to address this important question of generalizability to a target population. We present a case study of the Innovative Medicines Initiative to illustrate the inverse propensity score reweighting methodology, which may allow us to estimate the expected treatment benefit if a clinical trial had been run in a broader real‐world target population. We learned that identifying treatment effect modifiers, understanding and managing differences between patient characteristic data sets, and balancing the closeness of trial and target patient populations with effective sample size are key to successfully using this methodology and potentially mitigating some of this uncertainty around local decision making.

Enrolment criteria for diabetes cardiovascular outcome trials do not inform on generalizability to clinical practice: The case of glucagon-like peptide-1 receptor agonists

AIM

To evaluate the generalizability of cardiovascular outcome trials (CVOTs) on glucagon-like peptide-1 receptor agonists (GLP-1RAs), we assessed what proportion of real-world patients with type 2 diabetes (T2D) constitute true CVOT-like populations.

MATERIALS AND METHODS

We applied inclusion/exclusion (I/E) criteria of each GLP-1RA CVOT to a cross-sectional database of 281 380 T2D patients from Italian diabetes outpatient clinics. We calculated the proportion of patients eligible for each CVOT and compared their clinical characteristics with those of trial patients. In addition, we used a Bayesian network-based method to sample the greatest subsets of real-world patients yielding true CVOT-like populations.

RESULTS

Between 98 725 and 124 164 T2D patients could be evaluated for CVOT eligibility. After excluding patients who were already on GLP-1RAs and applying I/E criteria, 35.8% of patients would be eligible for REWIND, 34.1% for PIONEER-6, 13.4% for EXSCEL, 10.1% for SUSTAIN-6, 9.5% for HARMONY and 9.4% for LEADER. Overall, 45.4% of patients could be eligible for at least one of the CVOTs. These patients, however, were extremely different to trial patients in most of the clinical characteristics, including demographics, concomitant medications and complications. The greatest CVOT-like subsets of real-world patients were 0.5% for SUSTAIN-6, 1.0% for EXSCEL, 1.2% for LEADER, 1.8% for PIONEER-6 and 7.9% for REWIND.

CONCLUSIONS

A very small proportion of real-world patients constitute true CVOT-like populations. These findings question whether any meaningful information can be drawn from applying trial enrolment criteria to real-world T2D patients.

Recovery of original individual person data (IPD) inferences from empirical IPD summaries only: Applications to distributed computing under disclosure constraints

There are many settings where individual person data (IPD) are not available, due to privacy or technical reasons, and one must work with IPD proxies, such as summary statistics, to approximate original IPD inferences, that is, the results of statistical analyses that would ideally have been performed on individual-level data. For instance, in a distributed computing setting, as implemented in the DataSHIELD software framework, different centers can only share IPD proxies to obtain pooled IPD inferences. Such privacy requirements limit the scope of statistical investigation. For example, it can be challenging to perform between-center random-effect regression models. To increase modeling freedom we propose a method that only uses simple nondisclosive summaries of the original IPD as input, such as empirical marginal moments and correlation matrices, and generates artificial data compatible with those summary features. Specifically, data are generated from a Gaussian copula with marginal and joint components specified by the above summaries. The goal is to reproduce original IPD features in the artificial data, such that original IPD inferences are recovered from the artificial data. In an application example, and through simulations, we show that we can recover estimates of a multivariable IPD random-effect logistic regression, from artificial data generated via the Gaussian copula using the above IPD summaries, suggesting the proposed approach provides a generally applicable strategy for distributed computing settings with data protection constraints.

Correcting for selective participation in cohort studies using auxiliary register data without identification of non-participants

Aims: Selective participation may hamper the validity of population-based cohort studies. The resulting bias can be alleviated by linking auxiliary register data to both the participants and the non-participants of the study, estimating propensity scores for participation and correcting for participation based on these. However, registry holders may not be allowed to disclose sensitive data on (invited) non-participants. Our aim is to provide guidance on how adequate bias correction can be achieved by using auxiliary register data but without disclosing information that could be linked to the subset of non-participants. Methods: We show how existing methods can be used to estimate generalisation weights under various data disclosure scenarios where invited non-participants are indistinguishable from uninvited ones. We also demonstrate how the methods can be implemented using Nordic register data. Results: Inverse-probability-of-sampling weights estimated within a random sample of the target population in which the non-respondents are disclosed are equivalent in expectation to analogous weights in a scenario where the non-participants and uninvited individuals from the population are indistinguishable. To minimise the risk of disclosure when the entire population is invited to participate, investigators should instead consider inverse-odds-of-sampling weights, a method that has previously been suggested for transporting study results to external populations. Conclusions: Generalisation weights can be estimated from auxiliary register data without disclosing information on invited non-participants.