Generalizing Evidence from Randomized Trials using Inverse Probability of Sampling Weights

Robustifying Trial-Derived Optimal Treatment Rules for A Target Population

Treatment rules based on individual patient characteristics that are easy to interpret and disseminate are important in clinical practice. Properly planned and conducted randomized clinical trials are used to construct individualized treatment rules. However, it is often a concern that trial participants lack representativeness, so it limits the applicability of the derived rules to a target population. In this work, we use data from a single trial study to propose a two-stage procedure to derive a robust and parsimonious rule to maximize the benefit in the target population. The procedure allows a wide range of possible covariate distributions in the target population, with minimal assumptions on the first two moments of the covariate distribution. The practical utility and favorable performance of the methodology are demonstrated using extensive simulations and a real data application.

Generalizing the per-protocol treatment effect: The case of ACTG A5095

BACKGROUND

Intention-to-treat comparisons of randomized trials provide asymptotically consistent estimators of the effect of treatment assignment, without regard to compliance. However, decision makers often wish to know the effect of a per-protocol comparison. Moreover, decision makers may also wish to know the effect of treatment assignment or treatment protocol in a user-specified target population other than the sample in which the trial was fielded. Here, we aimed to generalize results from the ACTG A5095 trial to the US recently HIV-diagnosed target population.

METHODS

We first replicated the published conventional intention-to-treat estimate (2-year risk difference and hazard ratio) comparing a four-drug antiretroviral regimen to a three-drug regimen in the A5095 trial. We then estimated the intention-to-treat effect that accounted for informative dropout and the per-protocol effect that additionally accounted for protocol deviations by constructing inverse probability weights. Furthermore, we employed inverse odds of sampling weights to generalize both intention-to-treat and per-protocol effects to a target population comprising US individuals with HIV diagnosed during 2008-2014.

RESULTS

Of 761 subjects in the analysis, 82 dropouts (36 in the three-drug arm and 46 in the four-drug arm) and 59 protocol deviations (25 in the three-drug arm and 34 in the four-drug arm) occurred during the first 2 years of follow-up. A total of 169 subjects incurred virologic failure or death. The 2-year risks were similar both in the trial and in the US HIV-diagnosed target population for estimates from the conventional intention-to-treat, dropout-weighted intention-to-treat, and per-protocol analyses. In the US target population, the 2-year conventional intention-to-treat risk difference (unit: %) for virologic failure or death comparing the four-drug arm to the three-drug arm was -0.4 (95% confidence interval: -6.2, 5.1), while the hazard ratio was 0.97 (95% confidence interval: 0.70, 1.34); the 2-year risk difference was -0.9 (95% confidence interval: -6.9, 5.3) for the dropout-weighted intention-to-treat comparison (hazard ratio = 0.95, 95% confidence interval: 0.68, 1.32) and -0.7 (95% confidence interval: -6.7, 5.5) for the per-protocol comparison (hazard ratio = 0.96, 95% confidence interval: 0.69, 1.34).

CONCLUSION

No benefit of four-drug antiretroviral regimen over three-drug regimen was found from the conventional intention-to-treat, dropout-weighted intention-to-treat or per-protocol estimates in the trial sample or target population.

Correlating eligibility criteria generalizability and adverse events using Big Data for patients and clinical trials

Randomized controlled trials can benefit from proactive assessment of how well their participant selection strategies during the design of eligibility criteria can influence the study generalizability. In this paper, we present a quantitative metric called generalizability index for study traits 2.0 (GIST 2.0) to assess the a priori generalizability (based on population representativeness) of a clinical trial by accounting for the dependencies among multiple eligibility criteria. The metric was evaluated on 16 sepsis trials identified from ClinicalTrials.gov, with their adverse event reports extracted from the trial results sections. The correlation between GIST scores and adverse events was analyzed. We found that the GIST 2.0 score was significantly correlated with total adverse events and serious adverse events (weighted correlation coefficients of 0.825 and 0.709, respectively, with P < 0.01). This study exemplifies the promising use of Big Data in electronic health records and ClinicalTrials.gov for optimizing eligibility criteria design for clinical studies.

GIST 2.0: A scalable multi-trait metric for quantifying population representativeness of individual clinical studies

The design of randomized controlled clinical studies can greatly benefit from iterative assessments of population representativeness of eligibility criteria. We propose a multi-trait metric - GIST 2.0 that can compute the a priori generalizability based on the population representativeness of a clinical study by explicitly modeling the dependencies among all eligibility criteria. We evaluate this metric on twenty clinical studies of two diseases and analyze how a study's eligibility criteria affect its generalizability (collectively and individually). We statistically analyze the effects of trial setting, trait selection and trait summarizing technique on GIST 2.0. Finally we provide theoretical as well as empirical validations for the expected properties of GIST 2.0.

The effect of antiretroviral therapy on all-cause mortality, generalized to persons diagnosed with HIV in the USA, 2009-11

BACKGROUND

Although antiretroviral therapy (ART) is known to be protective against HIV-related mortality, the expected magnitude of effect is unclear because existing estimates of the effect of ART may not directly generalize to recently HIV-diagnosed persons.

METHODS

In this study, we estimated 5-year mortality risks for immediate versus no ART initiation among patients (n = 12,547) in the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) using the complement of adjusted Kaplan-Meier survival functions. We subsequently standardized estimates to persons diagnosed with HIV in the USA between 2009 and 2011, who were enumerated using national surveillance data.

RESULTS

The 5-year mortality, had all patients in the CNICS immediately initiated ART, was 10.6% [95% confidence interval (CI): 9.3%, 11.9%] compared with 28.3% (95% CI: 19.1%, 37.5%) had ART initiation been delayed at least 5 years. The 5-year mortality risk difference due to ART among patients in the CNICS was -17.7% (95% CI: -27.0%, -8.4%). Based on methods for generalizing an estimate from a study sample to a different target population, the expected risk difference due to ART initiation among recently HIV-diagnosed persons in the USA was -19.1% (95% CI: -30.5%, -7.8%).

CONCLUSIONS

Immediate ART initiation substantially lowers mortality among persons in the CNICS and this benefit is expected to be similar among persons recently diagnosed with HIV in the USA. We demonstrate a method by which concerns about generalizability can be addressed and evaluated quantitatively.