A modular, prospective, semi-automated drug safety monitoring system for use in a distributed data environment

Assessing the Benefits and Harms of Pharmacotherapy in Older Adults with Frailty: Insights from Pharmacoepidemiologic Studies of Routine Health Care Data

The objective of this review is to summarize and appraise the research methodology, emerging findings, and future directions in pharmacoepidemiologic studies assessing the benefits and harms of pharmacotherapies in older adults with different levels of frailty. Older adults living with frailty are at elevated risk for poor health outcomes and adverse effects from pharmacotherapy. However, current evidence is limited due to the under-enrollment of frail older adults and the lack of validated frailty assessments in clinical trials. Recent advancements in measuring frailty in administrative claims and electronic health records (database-derived frailty scores) have enabled researchers to identify patients with frailty and to evaluate the heterogeneity of treatment effects by patients' frailty levels using routine health care data. When selecting a database-derived frailty score, researchers must consider the type of data (e.g., different coding systems), the length of the predictor assessment period, the extent of validation against clinically validated frailty measures, and the possibility of surveillance bias arising from unequal access to care. We reviewed 13 pharmacoepidemiologic studies published on PubMed from 2013 to 2023 that evaluated the benefits and harms of cardiovascular medications, diabetes medications, anti-neoplastic agents, antipsychotic medications, and vaccines by frailty levels. These studies suggest that, while greater frailty is positively associated with adverse treatment outcomes, older adults with frailty can still benefit from pharmacotherapy. Therefore, we recommend routine frailty subgroup analyses in pharmacoepidemiologic studies. Despite data and design limitations, the findings from such studies may be informative to tailor pharmacotherapy for older adults across the frailty spectrum.

A methodological review of the high-dimensional propensity score in comparative-effectiveness and safety-of-interventions research finds incomplete reporting relative to algorithm development and robustness

OBJECTIVES

The use of secondary databases has become popular for evaluating the effectiveness and safety of interventions in real-life settings. However, the absence of important confounders in these databases is challenging. To address this issue, the high-dimensional propensity score (hdPS) algorithm was developed in 2009. This algorithm uses proxy variables for mitigating confounding by combining information available across several healthcare dimensions. This study assessed the methodology and reporting of the hdPS in comparative effectiveness and safety research.

STUDY DESIGN AND SETTING

In this methodological review, we searched PubMed and Google Scholar from July 2009 to May 2022 for studies that used the hdPS for evaluating the effectiveness or safety of healthcare interventions. Two reviewers independently extracted study characteristics and assessed how the hdPS was applied and reported. Risk of bias was evaluated with the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) tool.

RESULTS

In total, 136 studies met the inclusion criteria; the median publication year was 2018 (Q1-Q3 2016-2020). The studies included 192 datasets, mostly North American databases (n = 132, 69%). The hdPS was used in primary analysis in 120 studies (88%). Dimensions were defined in 101 studies (74%), with a median of 5 (Q1-Q3 4-6) dimensions included. A median of 500 (Q1-Q3 200-500) empirically identified covariates were selected. Regarding hdPS reporting, only 11 studies (8%) reported all recommended items. Most studies (n = 81, 60%) had a moderate overall risk of bias.

CONCLUSION

There is room for improvement in the reporting of hdPS studies, especially regarding the transparency of methodological choices that underpin the construction of the hdPS.

The State of Use and Utility of Negative Controls in Pharmacoepidemiologic Studies

Uses of real-world data in drug safety and effectiveness studies are often challenged by various sources of bias. We undertook a systematic search of the published literature through September 2020 to evaluate the state of use and utility of negative controls to address bias in pharmacoepidemiologic studies. Two reviewers independently evaluated study eligibility and abstracted data. Our search identified 184 eligible studies for inclusion. Cohort studies (115, 63%) and administrative data (114, 62%) were, respectively, the most common study design and data type used. Most studies used negative control outcomes (91, 50%), and for most studies the target source of bias was unmeasured confounding (93, 51%). We identified 4 utility domains of negative controls: 1) bias detection (149, 81%), 2) bias correction (16, 9%), 3) P-value calibration (8, 4%), and 4) performance assessment of different methods used in drug safety studies (31, 17%). The most popular methodologies used were the 95% confidence interval and P-value calibration. In addition, we identified 2 reference sets with structured steps to check the causality assumption of the negative control. While negative controls are powerful tools in bias detection, we found many studies lacked checking the underlying assumptions. This article is part of a Special Collection on Pharmacoepidemiology.

Surveillance of Antidepressant Safety (SADS): Active Signal Detection of Serious Medical Events Following SSRI and SNRI Initiation Using Big Healthcare Data

Introduction

The current process for generating evidence in pharmacovigilance has several limitations, which often lead to delays in the evaluation of drug-associated risks.

Objectives

In this study, we proposed and tested a near real-time epidemiological surveillance system using sequential, cumulative analyses focusing on the detection and preliminary risk quantification of potential safety signals following initiation of selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine reuptake inhibitors (SNRIs).

Methods

We emulated an active surveillance system in an historical setting by conducting repeated annual cohort studies using nationwide Danish healthcare data (1996–2016). Outcomes were selected from the European Medicines Agency's Designated Medical Event list, summaries of product characteristics, and the literature. We followed patients for a maximum of 6 months from treatment initiation to the event of interest or censoring. We performed Cox regression analyses adjusted for standard sets of covariates. Potential safety signals were visualized using heat maps and cumulative hazard ratio (HR) plots over time.

Results

In the total study population, 969,667 new users were included and followed for 461,506 person-years. We detected potential safety signals with incidence rates as low as 0.9 per 10,000 person-years. Having eight different exposure drugs and 51 medical events, we identified 31 unique combinations of potential safety signals with a positive association to the event of interest in the exposed group. We proposed that these signals were designated for further evaluation once they appeared in a prospective setting. In total, 21 (67.7%) of these were not present in the current summaries of product characteristics.

Conclusion

The study demonstrated the feasibility of performing epidemiological surveillance using sequential, cumulative analyses. Larger populations are needed to evaluate rare events and infrequently used antidepressants.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40264-021-01110-x.

Relative Performance of Propensity Score Matching Strategies for Subgroup Analyses

Postapproval drug safety studies often use propensity scores (PSs) to adjust for a large number of baseline confounders. These studies may involve examining whether treatment safety varies across subgroups. There are many ways a PS could be used to adjust for confounding in subgroup analyses. These methods have trade-offs that are not well understood. We conducted a plasmode simulation to compare relative performance of 5 methods involving PS matching for subgroup analysis, including methods frequently used in applied literature whose performance has not been previously directly compared. These methods varied as to whether the overall PS, subgroup-specific PS, or no rematching was used in subgroup analysis as well as whether subgroups were fully nested within the main analytical cohort. The evaluated PS subgroup matching methods performed similarly in terms of balance, bias, and precision in 12 simulated scenarios varying size of the cohort, prevalence of exposure and outcome, strength of relationships between baseline covariates and exposure, the true effect within subgroups, and the degree of confounding within subgroups. Each had strengths and limitations with respect to other performance metrics that could inform choice of method.

Automated data-adaptive analytics for electronic healthcare data to study causal treatment effects

BACKGROUND

Decision makers in health care increasingly rely on nonrandomized database analyses to assess the effectiveness, safety, and value of medical products. Health care data scientists use data-adaptive approaches that automatically optimize confounding control to study causal treatment effects. This article summarizes relevant experiences and extensions.

METHODS

The literature was reviewed on the uses of high-dimensional propensity score (HDPS) and related approaches for health care database analyses, including methodological articles on their performance and improvement. Articles were grouped into applications, comparative performance studies, and statistical simulation experiments.

RESULTS

The HDPS algorithm has been referenced frequently with a variety of clinical applications and data sources from around the world. The appeal of HDPS for database research rests in 1) its superior performance in situations of unobserved confounding through proxy adjustment, 2) its predictable efficiency in extracting confounding information from a given data source, 3) its ability to automate estimation of causal treatment effects to the extent achievable in a given data source, and 4) its independence of data source and coding system. Extensions of the HDPS approach have focused on improving variable selection when exposure is sparse, using free text information and time-varying confounding adjustment.

CONCLUSION

Semiautomated and optimized confounding adjustment in health care database analyses has proven successful across a wide range of settings. Machine-learning extensions further automate its use in estimating causal treatment effects across a range of data scenarios.

Real-World Data Analytics Fit for Regulatory Decision-Making

Healthcare database analyses (claims, electronic health records) have been identified by various regulatory initiatives, including the 21^st Century Cures Act and Prescription Drug User Fee Act ("PDUFA"), as useful supplements to randomized clinical trials to generate evidence on the effectiveness, harm, and value of medical products in routine care. Specific applications include accelerated drug approval pathways and secondary indications for approved medical products. Such real-world data ("RWD") analyses reflect how medical products impact health outside a highly controlled research environment. A constant stream of data from the routine operation of modern healthcare systems that can be analyzed in rapid cycles enables incremental evidence development for regulatory decision-making. Key evidentiary needs by regulators include 1) monitoring of medication performance in routine care, including the effectiveness, safety and value; 2) identifying new patient strata in which a drug may have added value or unacceptable harms; and 3) monitoring targeted utilization. Four broad requirements have been proposed to enable successful regulatory decision-making based on healthcare database analyses (collectively, "MVET"): Meaningful evidence that provides relevant and context-informed evidence sufficient for interpretation, drawing conclusions, and making decisions; valid evidence that meets scientific and technical quality standards to allow causal interpretations; expedited evidence that provides incremental evidence that is synchronized with the decision-making process; and transparent evidence that is audible, reproducible, robust, and ultimately trusted by decision-makers. Evidence generation systems that satisfy MVET requirements to a high degree will contribute to effective regulatory decision-making. Rapid-cycle analytics of healthcare databases is maturing at a time when regulatory overhaul increasingly demands such evidence. Governance, regulations, and data quality are catching up as the utility of this resource is demonstrated in multiple contexts.

Evaluation of the US Food and Drug Administration sentinel analysis tools in confirming previously observed drug-outcome associations: The case of clindamycin and Clostridium difficile infection

PURPOSE

The Food and Drug Administration's Sentinel System developed parameterized, reusable analytic programs for evaluation of medical product safety. Research on outpatient antibiotic exposures, and Clostridium difficile infection (CDI) with non-user reference groups led us to expect a higher rate of CDI among outpatient clindamycin users vs penicillin users. We evaluated the ability of the Cohort Identification and Descriptive Analysis and Propensity Score Matching tools to identify a higher rate of CDI among clindamycin users.

METHODS

We matched new users of outpatient dispensings of oral clindamycin or penicillin from 13 Data Partners 1:1 on propensity score and followed them for up to 60 days for development of CDI. We used Cox proportional hazards regression stratified by Data Partner and matched pair to compare CDI incidence.

RESULTS

Propensity score models at 3 Data Partners had convergence warnings and a limited range of predicted values. We excluded these Data Partners despite adequate covariate balance after matching. From the 10 Data Partners where these models converged without warnings, we identified 807 919 new clindamycin users and 8 815 441 new penicillin users eligible for the analysis. The stratified analysis of 807 769 matched pairs included 840 events among clindamycin users and 290 among penicillin users (hazard ratio 2.90, 95% confidence interval 2.53, 3.31).

CONCLUSIONS

This evaluation produced an expected result and identified several potential enhancements to the Propensity Score Matching tool. This study has important limitations. CDI risk may have been related to factors other than the inherent properties of the drugs, such as duration of use or subsequent exposures.

Safety and Effectiveness of Direct Oral Anticoagulants Versus Vitamin K Antagonists: Pilot Implementation of a Near-Real-Time Monitoring Program in Italy

Background

Real‐time monitoring is used to the ends of postmarketing observational research on newly marketed drugs. We implemented a pilot near‐real‐time monitoring program on the test case of oral anticoagulants. Specifically, we evaluated the safety and effectiveness of direct oral anticoagulants compared to vitamin K antagonists in nonvalvular atrial fibrillation secondary prevention during 2013‐2015 in the Lazio Region, Italy.

Methods and Results

A cohort study was conducted using a sequential propensity‐score–matched new user parallel‐cohort design. Sequential analyses were performed using Cox models. Overall, 10 742 patients contributed to the analyses. Compared with vitamin K antagonists, direct oral anticoagulant use was associated with a reduction of all‐cause mortality (0.81; 95% confidence interval [CI] 0.66‐0.99), cardiovascular mortality (0.71; 95% CI 0.54‐0.93), myocardial infarction (0.67; 95% CI 0.43‐1.04), ischemic stroke (0.87; 95% CI 0.52‐1.45), hemorrhagic stroke (0.25; 95% CI 0.07‐0.88), and with a nonsignificant increase of gastrointestinal bleeding (1.26; 95% CI 0.69‐2.30).

Conclusions

The present pilot study is a cornerstone to develop real‐time monitoring for new drugs in our region.

Prospective Postmarketing Surveillance of Acute Myocardial Infarction in New Users of Saxagliptin: A Population-Based Study

OBJECTIVE

The cardiovascular safety of saxagliptin, a dipeptidyl-peptidase 4 inhibitor, compared with other antihyperglycemic treatments is not well understood. We prospectively examined the association between saxagliptin use and acute myocardial infarction (AMI).

RESEARCH DESIGN AND METHODS

We identified patients aged ≥18 years, starting from the approval date of saxagliptin in 2009 and continuing through August 2014, using data from 18 Mini-Sentinel data partners. We conducted seven sequential assessments comparing saxagliptin separately with sitagliptin, pioglitazone, second-generation sulfonylureas, and long-acting insulin, using disease risk score (DRS) stratification and propensity score (PS) matching to adjust for potential confounders. Sequential testing kept the overall chance of a false-positive signal below 0.05 (one-sided) for each pairwise comparison.

RESULTS

We identified 82,264 saxagliptin users and more than 1.5 times as many users of each comparator. At the end of surveillance, the DRS-stratified hazard ratios (HRs) (95% CI) were 1.08 (0.90-1.28) in the comparison with sitagliptin, 1.11 (0.87-1.42) with pioglitazone, 0.79 (0.64-0.98) with sulfonylureas, and 0.57 (0.46-0.70) with long-acting insulin. The corresponding PS-matched HRs were similar. Only one interim analysis of 168 analyses met criteria for a safety signal: the PS-matched saxagliptin-pioglitazone comparison from the fifth sequential analysis, which yielded an HR of 1.63 (1.12-2.37). This association diminished in subsequent analyses.

CONCLUSIONS

We did not find a higher AMI risk in saxagliptin users compared with users of other selected antihyperglycemic agents during the first 5 years after U.S. Food and Drug Administration approval of the drug.

Development and application of two semi-automated tools for targeted medical product surveillance in a distributed data network

PURPOSE OF REVIEW

An important component of the Food and Drug Administration's Sentinel Initiative is the active post-market risk identification and analysis (ARIA) system, which utilizes semi-automated, parameterized computer programs to implement propensity-score adjusted and self-controlled risk interval designs to conduct targeted surveillance of medical products in the Sentinel Distributed Database. In this manuscript, we review literature relevant to the development of these programs and describe their application within the Sentinel Initiative.

RECENT FINDINGS

These quality-checked and publicly available tools have been successfully used to conduct rapid, replicable, and targeted safety analyses of several medical products. In addition to speed and reproducibility, use of semi-automated tools allows investigators to focus on decisions regarding key methodological parameters. We also identified challenges associated with the use of these methods in distributed and prospective datasets like the Sentinel Distributed Database, namely uncertainty regarding the optimal approach to estimating propensity scores in dynamic data among data partners of heterogeneous size.

SUMMARY

Future research should focus on the methodological challenges raised by these applications as well as developing new modular programs for targeted surveillance of medical products.

The Potential Return on Public Investment in Detecting Adverse Drug Effects

BACKGROUND

Many countries lack fully functional pharmacovigilance programs, and public budgets allocated to pharmacovigilance in industrialized countries remain low due to resource constraints and competing priorities.

OBJECTIVE

Using 3 case examples, we sought to estimate the public health and economic benefits resulting from public investment in active pharmacovigilance programs to detect adverse drug effects.

RESEARCH DESIGN

We assessed 3 examples in which early signals of safety hazards were not adequately recognized, resulting in continued exposure of a large number of patients to these drugs when safer and effective alternative treatments were available. The drug examples studied were rofecoxib, cerivastatin, and troglitazone. Using an individual patient simulation model and the health care system perspective, we estimated the potential costs that could have been averted by early systematic detection of safety hazards through the implementation of active surveillance programs.

RESULTS

We found that earlier drug withdrawal made possible by active safety surveillance would most likely have resulted in savings in direct medical costs of $773-$884 million for rofecoxib, $3-$10 million for cerivastatin, and $38-$63 million for troglitazone in the United States through the prevention of adverse events. By contrast, the yearly public investment in Food and Drug Administration initiated population-based pharmacovigilance activities in the United States is about $42.5 million at present.

CONCLUSION

These examples illustrate a critical and economically justifiable role for active adverse effect surveillance in protecting the health of the public.

Continuous Post-Market Sequential Safety Surveillance with Minimum Events to Signal

The CDC Vaccine Safety Datalink project has pioneered the use of near real-time post-market vaccine safety surveillance for the rapid detection of adverse events. Doing weekly analyses, continuous sequential methods are used, allowing investigators to evaluate the data near-continuously while still maintaining the correct overall alpha level. With continuous sequential monitoring, the null hypothesis may be rejected after only one or two adverse events are observed. In this paper, we explore continuous sequential monitoring when we do not allow the null to be rejected until a minimum number of observed events have occurred. We also evaluate continuous sequential analysis with a delayed start until a certain sample size has been attained. Tables with exact critical values, statistical power and the average times to signal are provided. We show that, with the first option, it is possible to both increase the power and reduce the expected time to signal, while keeping the alpha level the same. The second option is only useful if the start of the surveillance is delayed for logistical reasons, when there is a group of data available at the first analysis, followed by continuous or near-continuous monitoring thereafter.

Leveraging Food and Drug Administration Adverse Event Reports for the Automated Monitoring of Electronic Health Records in a Pediatric Hospital

The objective of this study was to determine whether the Food and Drug Administration’s Adverse Event Reporting System (FAERS) data set could serve as the basis of automated electronic health record (EHR) monitoring for the adverse drug reaction (ADR) subset of adverse drug events. We retrospectively collected EHR entries for 71 909 pediatric inpatient visits at Cincinnati Children’s Hospital Medical Center. Natural language processing (NLP) techniques were used to identify positive diseases/disorders and signs/symptoms (DDSSs) from the patients’ clinical narratives. We downloaded all FAERS reports submitted by medical providers and extracted the reported drug-DDSS pairs. For each patient, we aligned the drug-DDSS pairs extracted from their clinical notes with the corresponding drug-DDSS pairs from the FAERS data set to identify Drug-Reaction Pair Sentences (DRPSs). The DRPSs were processed by NLP techniques to identify ADR-related DRPSs. We used clinician annotated, real-world EHR data as reference standard to evaluate the proposed algorithm. During evaluation, the algorithm achieved promising performance and showed great potential in identifying ADRs accurately for pediatric patients.

Successful Comparison of US Food and Drug Administration Sentinel Analysis Tools to Traditional Approaches in Quantifying a Known Drug-Adverse Event Association

The US Food and Drug Administration's Sentinel system has developed the capability to conduct active safety surveillance of marketed medical products in a large network of electronic healthcare databases. We assessed the extent to which the newly developed, semiautomated Sentinel Propensity Score Matching (PSM) tool could produce the same results as a customized protocol-driven assessment, which found an adjusted hazard ratio (HR) of 3.04 (95% confidence interval [CI], 2.81-3.27) comparing angioedema in patients initiating angiotensin-converting enzyme (ACE) inhibitors vs. beta-blockers. Using data from 13 Data Partners between 1 January 2008, and 30 September 2013, the PSM tool identified 2,211,215 eligible ACE inhibitor and 1,673,682 eligible beta-blocker initiators. The tool produced an HR of 3.14 (95% CI, 2.86-3.44). This comparison provides initial evidence that Sentinel analytic tools can produce findings similar to those produced by a highly customized protocol-driven assessment.

Transparency and Reproducibility of Observational Cohort Studies Using Large Healthcare Databases

The scientific community and decision-makers are increasingly concerned about transparency and reproducibility of epidemiologic studies using longitudinal healthcare databases. We explored the extent to which published pharmacoepidemiologic studies using commercially available databases could be reproduced by other investigators. We identified a nonsystematic sample of 38 descriptive or comparative safety/effectiveness cohort studies. Seven studies were excluded from reproduction, five because of violation of fundamental design principles, and two because of grossly inadequate reporting. In the remaining studies, >1,000 patient characteristics and measures of association were reproduced with a high degree of accuracy (median differences between original and reproduction <2% and <0.1). An essential component of transparent and reproducible research with healthcare databases is more complete reporting of study implementation. Once reproducibility is achieved, the conversation can be elevated to assess whether suboptimal design choices led to avoidable bias and whether findings are replicable in other data sources.

Prospective Benefit-Risk Monitoring of New Drugs for Rapid Assessment of Net Favorability in Electronic Health Care Data

BACKGROUND

Benefit-risk assessment (BRA) methods can combine measures of benefits and risks into a single value.

OBJECTIVES

To examine BRA metrics for prospective monitoring of new drugs in electronic health care data.

METHODS

Using two electronic health care databases, we emulated prospective monitoring of three drugs (rofecoxib vs. nonselective nonsteroidal anti-inflammatory drugs, prasugrel vs. clopidogrel, and denosumab vs. bisphosphonates) using a sequential propensity score-matched cohort design. We applied four BRA metrics: number needed to treat and number needed to harm; incremental net benefit (INB) with maximum acceptable risk; INB with relative-value-adjusted life-years; and INB with quality-adjusted life-years (QALYs). We determined whether and when the bootstrapped 99% confidence interval (CI) for each metric excluded zero, indicating net favorability of one drug over the other.

RESULTS

For rofecoxib, all four metrics yielded a negative value, suggesting net favorability of nonselective nonsteroidal anti-inflammatory drugs over rofecoxib, and the 99% CI for all but the number needed to treat and number needed to harm excluded the null during follow-up. For prasugrel, only the 99% CI for INB-QALY excluded the null, but trends in values over time were similar across the four metrics, suggesting overall net favorability of prasugrel versus clopidogrel. The 99% CI for INB-relative-value-adjusted life-years and INB-QALY excluded the null in the denosumab example, suggesting net favorability of denosumab over bisphosphonates.

CONCLUSIONS

Prospective benefit-risk monitoring can be used to determine net favorability of a new drug in electronic health care data. In three examples, existing BRA metrics produced qualitatively similar results but differed with respect to alert generation.

Developing alerting thresholds for prospective drug safety monitoring

BACKGROUND

Current methods for prospective drug safety monitoring focus on determining whether and when to generate safety alerts indicating that a new drug may be less safe than a comparator. Approaches are needed to develop safety thresholds that can be used to define whether a new drug is no less than or equally safe as the comparator.

OBJECTIVES

Our aim is to develop a framework for determining which safety statements can be made about a new drug and when they can be made during prospective monitoring.

METHODS

We developed a two-pronged approach to establish safety thresholds for active monitoring. First, we adapted concepts from setting margins in non-inferiority (NI) trials ("NI approach"). Second, we summarized NI margins used in published randomized trials and reviewed publicly available data from the US FDA's website to identify the type and magnitude of evidence used in regulatory decisions involving withdrawals and black box warnings between 2009 and 2013 ("benchmark approach"). We applied the framework to a case study of dabigatran versus warfarin and major bleed.

RESULTS

We provide formulas on both risk ratio and risk difference scales for the NI approach that are analogous to threshold setting in NI trials but based on point estimates and using a maximum tolerable increase rather than a preservation factor. Using this approach, we established a safety threshold for the dabigatran case study that was within range of the findings from the benchmark approach (1.18 to 7.30). Comparing the safety threshold with post-approval studies of dabigatran versus warfarin indicated that no safety statement can be made.

CONCLUSIONS

The proposed framework expands the safety statements that can be made in current prospective drug safety monitoring systems. Copyright © 2015 John Wiley & Sons, Ltd.