Assessing the use of observational methods and real-world data to emulate ongoing randomized controlled trials

BACKGROUND/AIMS

There has been growing interest in better understanding the potential of observational research methods in medical product evaluation and regulatory decision-making. Previously, we used linked claims and electronic health record data to emulate two ongoing randomized controlled trials, characterizing the populations and results of each randomized controlled trial prior to publication of its results. Here, our objective was to compare the populations and results from the emulated trials with those of the now-published randomized controlled trials.

METHODS

This study compared participants' demographic and clinical characteristics and study results between the emulated trials, which used structured data from OptumLabs Data Warehouse, and the published PRONOUNCE and GRADE trials. First, we examined the feasibility of implementing the baseline participant characteristics included in the published PRONOUNCE and GRADE trials' using real-world data and classified each variable as ascertainable, partially ascertainable, or not ascertainable. Second, we compared the emulated trials and published randomized controlled trials for baseline patient characteristics (concordance determined using standardized mean differences <0.20) and results of the primary and secondary endpoints (concordance determined by direction of effect estimates and statistical significance).

RESULTS

The PRONOUNCE trial enrolled 544 participants, and the emulated trial included 2226 propensity score-matched participants. In the PRONOUNCE trial publication, one of the 32 baseline participant characteristics was listed as an exclusion criterion on ClinicalTrials.gov but was ultimately not used. Among the remaining 31 characteristics, 9 (29.0%) were ascertainable, 11 (35.5%) were partially ascertainable, and 10 (32.2%) were not ascertainable using structured data from OptumLabs. For one additional variable, the PRONOUNCE trial did not provide sufficient detail to allow its ascertainment. Of the nine variables that were ascertainable, values in the emulated trial and published randomized controlled trial were discordant for 6 (66.7%). The primary endpoint of time from randomization to the first major adverse cardiovascular event and secondary endpoints of nonfatal myocardial infarction and stroke were concordant between the emulated trial and published randomized controlled trial. The GRADE trial enrolled 5047 participants, and the emulated trial included 7540 participants. In the GRADE trial publication, 8 of 34 (23.5%) baseline participant characteristics were ascertainable, 14 (41.2%) were partially ascertainable, and 11 (32.4%) were not ascertainable using structured data from OptumLabs. For one variable, the GRADE trial did not provide sufficient detail to allow for ascertainment. Of the eight variables that were ascertainable, values in the emulated trial and published randomized controlled trial were discordant for 4 (50.0%). The primary endpoint of time to hemoglobin A1c ≥7.0% was mostly concordant between the emulated trial and the published randomized controlled trial.

CONCLUSION

Despite challenges, observational methods and real-world data can be leveraged in certain important situations for a more timely evaluation of drug effectiveness and safety in more diverse and representative patient populations.

A Descriptive Cohort Study of Drug Utilization Patterns Among Patients Hospitalized With Coronavirus Disease 2019 in the United States, January 2021-February 2022

Background

There is a dearth of drug utilization studies for coronavirus disease 2019 (COVID-19) treatments in 2021 and beyond after the introduction of vaccines and updated guidelines; such studies are needed to contextualize ongoing COVID-19 treatment effectiveness studies during these time periods. This study describes utilization patterns for corticosteroids, interleukin-6 (IL-6) inhibitors, Janus kinase inhibitors, and remdesivir among hospitalized adults with COVID-19, over the entire hospitalization, and within hospitalization periods categorized by respiratory support requirements.

Methods

This descriptive cohort study included United States adults hospitalized with COVID-19 admitted from 1 January 2021 through 1 February 2022; data included HealthVerity claims and hospital chargemaster. The number and distribution of patients were reported for the first 3 drug regimen lines initiated.

Results

The cohort included 51 066 patients; the most common initial drug regimens were corticosteroids (23.4%), corticosteroids plus remdesivir (25.1%), and remdesivir (4.4%). IL-6 inhibitors and Janus kinase inhibitors were included in later drug regimens and were more commonly administered with both corticosteroids and remdesivir than with corticosteroids alone. IL-6 inhibitors were more commonly administered than Janus kinase inhibitors when patients received high-flow oxygen or ventilation.

Conclusions

These findings provide important context for comparative studies of COVID-19 treatments with study periods extending into 2021 and later. While prescribing generally aligned with National Institutes of Health COVID-19 treatment guidelines during this period, these findings suggest that prescribing preference, potential confounding by indication, and confounding by prior/concomitant use of other therapeutics should be considered in the design and interpretation of comparative studies.

Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials

Importance

Nonrandomized studies using insurance claims databases can be analyzed to produce real-world evidence on the effectiveness of medical products. Given the lack of baseline randomization and measurement issues, concerns exist about whether such studies produce unbiased treatment effect estimates.

Objective

To emulate the design of 30 completed and 2 ongoing randomized clinical trials (RCTs) of medications with database studies using observational analogues of the RCT design parameters (population, intervention, comparator, outcome, time [PICOT]) and to quantify agreement in RCT-database study pairs.

Design, Setting, and Participants

New-user cohort studies with propensity score matching using 3 US claims databases (Optum Clinformatics, MarketScan, and Medicare). Inclusion-exclusion criteria for each database study were prespecified to emulate the corresponding RCT. RCTs were explicitly selected based on feasibility, including power, key confounders, and end points more likely to be emulated with real-world data. All 32 protocols were registered on ClinicalTrials.gov before conducting analyses. Emulations were conducted from 2017 through 2022.

Exposures

Therapies for multiple clinical conditions were included.

Main Outcomes and Measures

Database study emulations focused on the primary outcome of the corresponding RCT. Findings of database studies were compared with RCTs using predefined metrics, including Pearson correlation coefficients and binary metrics based on statistical significance agreement, estimate agreement, and standardized difference.

Results

In these highly selected RCTs, the overall observed agreement between the RCT and the database emulation results was a Pearson correlation of 0.82 (95% CI, 0.64-0.91), with 75% meeting statistical significance, 66% estimate agreement, and 75% standardized difference agreement. In a post hoc analysis limited to 16 RCTs with closer emulation of trial design and measurements, concordance was higher (Pearson r, 0.93; 95% CI, 0.79-0.97; 94% meeting statistical significance, 88% estimate agreement, 88% standardized difference agreement). Weaker concordance occurred among 16 RCTs for which close emulation of certain design elements that define the research question (PICOT) with data from insurance claims was not possible (Pearson r, 0.53; 95% CI, 0.00-0.83; 56% meeting statistical significance, 50% estimate agreement, 69% standardized difference agreement).

Conclusions and Relevance

Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.

Emulating the GRADE trial using real world data: retrospective comparative effectiveness study

OBJECTIVE

To emulate the GRADE (Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness Study) trial using real world data before its publication. GRADE directly compared second line glucose lowering drugs for their ability to lower glycated hemoglobin A_1c (HbA_1c).

DESIGN

Observational study.

SETTING

OptumLabs® Data Warehouse (OLDW), a nationwide claims database in the US, 25 January 2010 to 30 June 2019.

PARTICIPANTS

Adults with type 2 diabetes and HbA_1c 6.8-8.5% while using metformin monotherapy, identified according to the GRADE trial specifications, who also used glimepiride, liraglutide, sitagliptin, or insulin glargine.

MAIN OUTCOME MEASURES

The primary outcome was time to HbA_1c ≥7.0%. Secondary outcomes were time to HbA_1c >7.5%, incident microvascular complications, incident macrovascular complications, adverse events, all cause hospital admissions, and all cause mortality. Propensity scores were estimated using the gradient boosting machine method, and inverse propensity score weighting was used to emulate randomization of the treatment groups, which were then compared using Cox proportional hazards regression.

RESULTS

8252 people were identified (19.7% of adults starting the study drugs in OLDW) who met eligibility criteria for the GRADE trial (glimepiride arm=4318, liraglutide arm=690, sitagliptin arm=2993, glargine arm=251). The glargine arm was excluded from analyses owing to small sample size. Median times to HbA_1c ≥7.0% were 442 days (95% confidence interval 394 to 480 days) for glimepiride, 764 (741 to not calculable) days for liraglutide, and 427 (380 to 483) days for sitagliptin. Liraglutide was associated with lower risk of reaching HbA_1c ≥7.0% compared with glimepiride (hazard ratio 0.57, 95% confidence interval 0.43 to 0.75) and sitagliptin (0.55, 0.41 to 0.73). Results were consistent for the secondary outcome of time to HbA_1c >7.5%. No significant differences were observed among treatment groups for the remaining secondary outcomes.

CONCLUSIONS

In this emulation of the GRADE trial, liraglutide was statistically significantly more effective at maintaining glycemic control than glimepiride or sitagliptin when added to metformin monotherapy. Generating timely evidence on medical treatments using real world data as a complement to prospective trials is of value.

Real-World Evidence Prediction of a Phase IV Oncology Trial: Comparative Degarelix vs Leuprolide Safety

BACKGROUND

Medical and regulatory communities are increasingly interested in the utility of real-world evidence (RWE) for answering questions pertaining to drug safety and effectiveness but concerns about validity remain. A principled approach to conducting RWE studies may alleviate concerns and increase confidence in findings. This study sought to predict the findings from the PRONOUNCE trial using a principled approach to generating RWE.

METHODS

This propensity-score (PS) matched observational cohort study utilized 3 claims databases to compare the occurrence of major adverse cardiovascular events (MACE) among initiators of degarelix vs. leuprolide. Patients were included if they had history of prostate cancer and atherosclerotic cardiovascular disease. Subjects were excluded if they didn't have continuous database enrollment in the year prior to treatment initiation, were exposed to androgen deprivation therapy or experienced an acute cardiovascular event within 30 days prior to treatment initiation, or had a history or risk factors of QT prolongation.

RESULTS

There were 12,448 leuprolide and 1,969 degarelix study-eligible patients before matching, with 1,887 in each arm after PS-matching. The results for MACE comparing degarelix to leuprolide in the observational analysis (hazard ratio= 1.35; 95% confidence interval = 0.94-1.93) was consistent with the subsequently released PRONOUNCE result (hazard ratio = 1.28; 95% confidence interval = 0.59-2.79).

CONCLUSIONS

This study successfully predicted the result of a comparative cardiovascular safety trial in the oncology setting. Although the findings are encouraging, limitations of measuring cancer stage and tumor progression are representative of challenges in attempting to generalize whether claims-based RWE can be used as actionable evidence.

Summary of Literature on Pediatric Real-world Evidence and Effectiveness

This cross-sectional study investigates the use of real-world data and real-world evidence in informing medicine efficacy in pediatric literature.

Real-world Cardiovascular Outcomes Associated With Degarelix vs Leuprolide for Prostate Cancer Treatment

IMPORTANCE

With a growing interest in the use of real-world evidence for regulatory decision-making, it is important to understand whether real-world data can be used to emulate the results of randomized clinical trials.

OBJECTIVE

To use electronic health record and administrative claims data to emulate the ongoing PRONOUNCE trial (A Trial Comparing Cardiovascular Safety of Degarelix Versus Leuprolide in Patients With Advanced Prostate Cancer and Cardiovascular Disease).

DESIGN, SETTING, AND PARTICIPANTS

This retrospective, propensity-matched cohort study included adult men with a diagnosis of prostate cancer and cardiovascular disease who initiated either degarelix or leuprolide between December 24, 2008, and June 30, 2019. Participants were commercially insured individuals and Medicare Advantage beneficiaries included in a large US administrative claims database.

EXPOSURES

Degarelix or leuprolide.

MAIN OUTCOMES AND MEASURES

The primary end point was time to first occurrence of a major adverse cardiovascular event (MACE), defined as death due to any cause, myocardial infarction, or stroke, analogous to the PRONOUNCE trial. Secondary end points were time to death due to any cause, myocardial infarction, stroke, and angina. Cox proportional hazards regression was used to evaluate primary and secondary end points.

RESULTS

A total of 32 172 men initiated degarelix or leuprolide for prostate cancer; of them, 9490 (29.5%) had cardiovascular disease, and 7800 (24.2%) met the PRONOUNCE trial eligibility criteria and were included in this study. Overall, 165 participants (2.1%) were Asian, 1390 (17.8%) were Black, 663 (8.5%) were Hispanic, and 5258 (67.4%) were White. The mean (SD) age was 74.4 (7.4) years. Among 2226 propensity score-matched patients, no significant difference was observed in the risk of MACE for patients taking degarelix vs those taking leuprolide (10.18 vs 8.60 events per 100 person-years; hazard ratio [HR], 1.18; 95% CI, 0.86-1.61). Degarelix was associated with a higher risk of death from any cause (HR, 1.48; 95% CI, 1.01-2.18) but not of myocardial infarction (HR, 1.16; 95% CI, 0.60-2.25), stroke (HR, 0.92; 95% CI, 0.45-1.85), or angina (HR, 1.36; 95% CI, 0.43-4.27).

CONCLUSIONS AND RELEVANCE

In this emulation of a clinical trial of men with cardiovascular disease undergoing treatment for prostate cancer, degarelix was not associated with a lower risk of cardiovascular events than leuprolide. Comparison of these data with PRONOUNCE trial results, when published, will help enhance our understanding of the appropriate role of using real-world data to emulate clinical trials.

Emulating Randomized Clinical Trials With Nonrandomized Real-World Evidence Studies: First Results From the RCT DUPLICATE Initiative

BACKGROUND

Regulators are evaluating the use of noninterventional real-world evidence (RWE) studies to assess the effectiveness of medical products. The RCT DUPLICATE initiative (Randomized, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology) uses a structured process to design RWE studies emulating randomized, controlled trials (RCTs) and compare results. We report findings of the first 10 trial emulations, evaluating cardiovascular outcomes of antidiabetic or antiplatelet medications.

METHODS

We selected 3 active-controlled and 7 placebo-controlled RCTs for replication. Using patient-level claims data from US commercial and Medicare payers, we implemented inclusion and exclusion criteria, selected primary end points, and comparator populations to emulate those of each corresponding RCT. Within the trial-mimicking populations, we conducted propensity score matching to control for >120 preexposure confounders. All study measures were prospectively defined and protocols registered before hazard ratios and 95% CIs were computed. Success criteria for the primary analysis were prespecified for each replication.

RESULTS

Despite attempts to emulate RCT design as closely as possible, differences between the RCT and corresponding RWE study populations remained. The regulatory conclusions were equivalent in 6 of 10. The RWE emulations achieved a hazard ratio estimate that was within the 95% CI from the corresponding RCT in 8 of 10 studies. In 9 of 10, either the regulatory or estimate agreement success criteria were fulfilled. The largest differences in effect estimates were found for RCTs where second-generation sulfonylureas were used as a proxy for placebo regarding cardiovascular effects. Nine of 10 replications had a standardized difference between effect estimates of <2, which suggests differences within expected random variation.

CONCLUSIONS

Agreement between RCT and RWE findings varies depending on which agreement metric is used. Interim findings indicate that selection of active comparator therapies with similar indications and use patterns enhances the validity of RWE. Even in the context of active comparators, concordance between RCT and RWE findings is not guaranteed, partially because trials are not emulated exactly. More trial emulations are needed to understand how often and in what contexts RWE findings match RCTs. Registration: URL: https://www.clinicaltrials.gov; Unique identifiers: NCT03936049, NCT04215523, NCT04215536, NCT03936010, NCT03936036, NCT03936062, NCT03936023, NCT03648424, NCT04237935, NCT04237922.

Depression of retinal glutamate transporter function leads to elevated intravitreal glutamate levels and ganglion cell death

PURPOSE

Elevated levels of extracellular glutamate have been implicated in the pathophysiology of neuronal loss in both central nervous system and ophthalmic disorders, including glaucoma. This increase in glutamate may result from a failure of glutamate transporters (molecules that ordinarily regulate extracellular glutamate; E:xcitatory A:mino A:cid T:ransporter; EAAT). Elevated glutamate levels can also lead to alterations in glutamate receptor expression. It was hypothesized that selective blockade of glutamate transporters would be toxic to retinal ganglion cells.

METHODS

Glutamate transporters were blocked either pharmacologically or with subtype-specific antisense oligonucleotides against EAAT1. Glutamate levels, transporter levels and ganglion cell survival were assayed.

RESULTS

Pharmacological inhibition of glutamate transporters with either an EAAT2 specific inhibitor or a nonspecific inhibitor of all the subtypes of transporters was toxic to ganglion cells. Treatment with oligonucleotides against the glutamate transporter EAAT1 decreased the levels of expression of the transporter, increased vitreal glutamate, and was toxic to ganglion cells.

CONCLUSIONS

These results demonstrate that normal function of EAAT1 and EAAT2 is necessary for retinal ganglion cell survival and plays an important role in retinal excitotoxicity. Manipulation of retinal glutamate transporter expression may become a useful tool in understanding retinal neuronal loss.