Claims-based cardiovascular outcome identification for clinical research: Results from 7 large randomized cardiovascular clinical trials

Comparison of Methods of Clinical Trial Emulation Utilizing Data From the Comparison of AMD Treatment Trial (CATT) and the IRIS® Registry

Purpose

We used exact matching and inverse propensity score weighting (IPSW) using real-world data (RWD) from the American Academy of Ophthalmology IRIS® Registry (Intelligent Research in Sight) to emulate the 2 pro re nata (prn) treatment arms from the Comparison of AMD Treatment Trial (CATT) and to compare the outcomes of the RWD arms to the 2 monthly treatment arms from the clinical trial.

Design

Retrospective cohort study utilizing deidentified electronic health record registry data and patient-level deidentified clinical trial data.

Subjects

All treatment-naive patient eyes with neovascular age-related macular degeneration treated with ranibizumab or bevacizumab only for 1 year from either the CATT or the IRIS Registry.

Methods

Patients were identified in the IRIS Registry between October 1, 2015 and December 31, 2019. After all nonimaging-based inclusion and exclusion criteria from the CATT were applied, patient eyes receiving bevacizumab or ranibizumab only on a prn basis were identified as the eligible cohort. Exact matching and ISPW was applied based on age, gender, and baseline visual acuity.

Main Outcome Measures

Mean change in visual acuity, in approximated ETDRS letters, between baseline and 1 year for the IRIS Registry prn treatment arms generated by exact matching and IPSW.

Results

We identified 427 eyes treated with ranibizumab prn and 771 eyes treated with bevacizumab prn. Using exact matching, 98% (n = 281) of CATT patient eyes in the bevacizumab monthly treatment arm and 87% (n = 261) of CATT patient eyes in the ranibizumab monthly treatment arm were matched to a patient eye in the IRIS Registry. For the ranibizumab prn treatment arm, patient eyes generated using exact matching gained 1.9 letters and those generated using IPSW gained 2.8 letters (exact matching: 1.9 letters ± 14.0 vs. IPSW: 2.8 letters ± 15.0 letters, P = 0.43). For the bevacizumab prn treatment arm, patient eyes generated using exact matching gained 2.4 letters and those generated using IPSW gained 2.1 letters (exact matching: 2.4 letters ± 15.4 vs. IPSW: 2.1 letters ± 16.0 letters, P = 0.79).

Conclusions

Both exact matching and IPSW produced similar results in emulating the prn treatment arms of the CATT using IRIS Registry data and patient-level clinical trial data. Similar to prior real-world studies, the clinical outcomes were significantly worse in the IRIS Registry treatment arms compared with the clinical trial.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Ascertaining Nonfatal Endpoints in Clinical Trials: Central Adjudication Versus Patient Insurance Claims

BACKGROUND

The 21st Century Cures Act allows the US Food and Drug Administration (FDA) to utilize real-world data (RWD) to create real-world evidence (RWE) for new indications or post approval study requirements. We compared central adjudication with two insurance claims data sources to understand how endpoint accuracy differences impact RWE results.

METHODS

We developed a decision analytic model to compare differences in efficacy (all-cause death, stroke and myocardial infarction) and safety (bleeding requiring transfusion) results for a simulated acute coronary syndrome antiplatelet therapy clinical trial. Endpoint accuracy metrics were derived from previous studies that compared centrally-adjudicated and insurance claims-based clinical trial endpoints.

RESULTS

Efficacy endpoint results per 100 patients were similar for the central adjudication model (intervention event rate, 11.3; control, 13.7; difference, 2.4) and the prospective claims data collection model (intervention event rate, 11.2; control 13.6; difference, 2.3). However, the retrospective claims linking model's efficacy results were larger (intervention event rate, 14.6; control, 18.0; difference, 3.4). True positive event rate results (intervention, control and difference) for both insurance claims-based models were less than the central adjudication model due to false negative events. Differences in false positive event rates were responsible for differences in efficacy results for the two insurance claims-based models.

CONCLUSION

Efficacy endpoint results differed by data source. Investigators need guidance to determine which data sources produce regulatory-grade RWE.

Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review

OBJECTIVE

Real-world data (RWD), defined as routinely collected healthcare data, can be a potential catalyst for addressing challenges faced in clinical trials. We performed a scoping review of database-specific RWD applications within clinical trial contexts, synthesizing prominent uses and themes.

MATERIALS AND METHODS

Querying 3 biomedical literature databases, research articles using electronic health records, administrative claims databases, or clinical registries either within a clinical trial or in tandem with methodology related to clinical trials were included. Articles were required to use at least 1 US RWD source. All abstract screening, full-text screening, and data extraction was performed by 1 reviewer. Two reviewers independently verified all decisions.

RESULTS

Of 2020 screened articles, 89 qualified: 59 articles used electronic health records, 29 used administrative claims, and 26 used registries. Our synthesis was driven by the general life cycle of a clinical trial, culminating into 3 major themes: trial process tasks (51 articles); dissemination strategies (6); and generalizability assessments (34). Despite a diverse set of diseases studied, <10% of trials using RWD for trial process tasks evaluated medications or procedures (5/51). All articles highlighted data-related challenges, such as missing values.

DISCUSSION

Database-specific RWD have been occasionally leveraged for various clinical trial tasks. We observed underuse of RWD within conducted medication or procedure trials, though it is subject to the confounder of implicit report of RWD use.

CONCLUSION

Enhanced incorporation of RWD should be further explored for medication or procedure trials, including better understanding of how to handle related data quality issues to facilitate RWD use.

Accuracy of Cardiovascular Trial Outcome Ascertainment and Treatment Effect Estimates from Routine Health Data: A Systematic Review and Meta-Analysis

BACKGROUND

Registry-based randomized controlled trials allow for outcome ascertainment using routine health data (RHD). While this method provides a potential solution to the rising cost and complexity of clinical trials, comparative analyses of outcome ascertainment by clinical end point committee (CEC) adjudication compared with RHD sources are sparse. Among cardiovascular trials, we set out to systematically compare the incidence of cardiovascular events and estimated randomized treatment effects ascertained from RHD versus traditional clinical evaluation and adjudication.

METHODS

We searched MEDLINE (1976 to August 2020) for studies where outcome ascertainment was performed by both RHD and CEC adjudication to compare the incidence of cardiovascular events and treatment effects. We derived ratios of hazard ratios to compare treatment effects from RHD and CEC adjudication. We pooled ratios of hazard ratios using an inverse variance random-effects meta-analysis.

RESULTS

Nine studies (1988-2020; 32 156 patients) involving 10 randomized control trials compared outcome ascertainment with RHD and CEC in patients with or at risk of cardiovascular disease. There was a high degree of agreement and interrater reliability between CEC and RHD outcome determination for all-cause mortality (agreement percentage: 98.4%-100% and κ: 0.95-1.0) and cardiovascular mortality (agreement percentage: 97.8%-99.9% and κ: 0.66-0.99). For myocardial infarction, the κ values ranged from 0.67-0.98, and for stroke the values ranged from 0.52-0.89. In contrast, the κ value for peripheral artery disease was low (κ: 0.27). There was little difference in the randomized treatment effect derived from CEC and RHD ascertainment of events based on the ratios of hazard ratio, with pooled ratios of hazard ratios ranging from 0.93 (95% CI, 0.63-1.39) for cardiovascular mortality to 1.27 (95% CI, 0.67-2.41) for stroke.

CONCLUSIONS

Clinical outcome ascertainment using retrospectively acquired RHD displayed high levels of agreement with CEC adjudication for identifying all-cause mortality and cardiovascular outcomes. Importantly, cardiovascular treatment effects in randomized control trials determined from RHD and CEC resulted in similar point estimates. Overall, our review supports the use of RHD as a potential alternative source for clinical outcome ascertainment in cardiovascular trials. Validation studies with prospectively planned linkage are warranted.

Utilization of Medical Codes for Hypotension in Shock Patients: A Retrospective Analysis

Purpose

To evaluate the utilization of hypotension diagnosis codes by shock type and year in known hypotensive patients.

Patients and Methods

Retrospective analysis of the Medicare fee-for-service claims database. Patients with a shock diagnosis code between 2011 and 2017 were identified using the International Classification of Diseases, Ninth and Tenth Revision, Clinical Modification (ICD-9-CM and ICD-10-CM). Based on specific ICD codes corresponding to each shock type, patients were classified into four mutually exclusive cohorts: cardiogenic shock, hypovolemic shock, septic shock, and other/unspecified shock. Annual proportion and counts of cases with at least one hypotension ICD code for each shock cohort were generated to produce 7-year medical code utilization trends. A Cochran-Armitage test for trend was performed to evaluate the statistical significance.

Results

A total of 2,200,275 shock patients were analyzed, 13.3% (n=292,192) of which received a hypotension code. Hypovolemic shock cases were the most likely to receive a hypotension code (18.02%, n=46,544), while septic shock cases had the lowest rate (11.48%, n=158,348). The proportion of patients with hypotension codes for other cohorts were 18.0% (n=46,544) for hypovolemic shock and 16.9% (n=32,024) for other/unspecified shock. The presence of hypotension codes decreased by 0.9% between 2011 and 2014, but significantly increased from 10.6% in 2014 to 17.9% in 2017 (p <0.0001, Z=−105.05).

Conclusion

Hypotension codes are remarkably underutilized in known hypotensive patients. Patients, providers, and researchers are likely to benefit from improved hypotension coding practices.

Probabilistic Linkage of Randomized Controlled Trial Data to Administrative Claims: A Case Study of Patients from Baricitinib Clinical Trials

Introduction

The aim of this work is to assess the feasibility of probabilistically linking randomized controlled trial (RCT) data to claims data in a real-world setting to inform future rheumatoid arthritis (RA) research.

Methods

This retrospective cohort study utilized IQVIA’s Patient Centric Medical Claims (Dx) Database, IQVIA’s Longitudinal Prescription Claims (LRx) Database, and Lilly’s baricitinib RCT data from a sample of patients that consented to the linkage of their de-identified insurance claims to their de-identified RCT data. Patients were initially matched on age, gender, and three-digit ZIP code of the provider and further matched according to a point scoring system using additional clinical variables.

Results

A total of 245 patients from 49 US clinical trial sites were eligible for the study and 78 (31.8%) of these patients consented to participate. Of the 78 consented patients, 69 (88%) were successfully matched on age, gender, and three-digit ZIP code of the provider. Of the 69 patients successfully matched on age, gender, and three-digit ZIP code of the provider, 44 (63.8%) had at least one sufficient match using the point scoring system. Of these 44, 23 (52.3%) patients matched at a ratio of one RCT patient to one Dx/LRx patient, 11 (25.0%) at a ratio of 1:2, 7 (15.9%) at a ratio of 1:3 and three (6.8%) at a ratio of 1:4 or greater. To further improve match ratios, a variable hierarchy was applied to the 18 RCT patients with 2–3 matches. Overall, 38 of the 78 (48.7%) consented RCT patients were successfully matched 1:1 to claims database patients.

Conclusions

This probabilistic linkage methodology demonstrates the feasibility, at a moderate linkage rate, of linking patients from RCTs to real-world data, which can provide a means to assess additional information not usually collected within or following a clinical trial.

Methods for safety and endpoint ascertainment: identification of adverse events through scrutiny of negatively adjudicated events

Background

The primary goal of phase 2 and 3 clinical trials is to evaluate the safety and effectiveness of therapeutic interventions, and efficient and reproducible ascertainment of important clinical events, either as clinical outcome events (COEs) or adverse events (AEs), is critical. Clinical outcomes require consistency and clinical judgment, so these events are often adjudicated centrally by clinical events classification (CEC) physician reviewers using standardized definitions. In contrast, AEs are reported by sites to the trial coordinating center based on common reporting criteria set by regulatory authorities and trial sponsors. These different requirements have led to the development of separate tracks for COE and AE review.

Main body

Potential COEs that fail to meet standardized definitions for CEC adjudication – i.e. negatively adjudicated events (NAE) – may meet criteria for AEs. Trial oversight practices require the sponsor to process AEs regardless of how the AEs are submitted; therefore, review of NAEs may be necessary to ensure that important AEs do not go unreported. The Duke Clinical Research Institute (DCRI) developed and implemented a process for scrutinizing NAEs to detect potential missed serious AEs. Initial experience with this process across two trials suggests that approximately 0.2% of NAEs are serious unexpected AEs that were not otherwise reported and another 1.5% are serious expected AEs.

Conclusions

Given their infrequent concealment of serious AEs in two large trials assessing cardiovascular outcomes, routine scrutiny of NAEs to identify AEs is not recommended at this time, though it may be useful in some trials and should be carefully considered by the trial team. Closer integration of data across safety surveillance and endpoint adjudication systems may enable scrutiny of NAEs when indicated while limiting complexity associated with this process.

Emerging opportunities to harness real world data: An introduction to data sources, concepts, and applications

While randomized controlled trials (RCTs) are the gold standard for comparative effectiveness research, they are unable to provide the answers to all pertinent clinical and research questions. Real world evidence (RWE), that is, clinical evidence obtained outside RCTs and often through routine clinical practice, offers the potential to conduct observational studies that accelerate advances in care, improve outcomes for patients, and provide important insights that can answer important questions. Once appropriate information technology is available, real world data can be cost-effective to generate. RWE serves a vital role in the evaluation of treatment strategies for which there are no RCTs and for describing patterns of care. RWE also serves as an important adjunct to RCTs and can be used to determine if benefits seen in RCTs extend to clinical practice, provide insight into the findings of RCTs, generate hypotheses for future RCTs, and inform the design of future RCTs. These potential benefits must be balanced against some of the important limitations of RWE, including variable data quality, lack of granularity for important clinical variables, and the potential for bias and confounding. By using appropriate analytic techniques and study design, these limitations can be minimized but not eliminated. Going forward, RWE studies may be enhanced by using rigorous data quality standards, incorporating randomization, developing more prospective registries, and better leveraging data from electronic health records.