Empirical performance of a new user cohort method: lessons for developing a risk identification and analysis system

Foundational model aided automatic high-throughput drug screening using self-controlled cohort study

Background

Developing medicine from scratch to governmental authorization and detecting adverse drug reactions (ADR) have barely been economical, expeditious, and risk-averse investments. The availability of large-scale observational healthcare databases and the popularity of large language models offer an unparalleled opportunity to enable automatic high-throughput drug screening for both repurposing and pharmacovigilance.

Objectives

To demonstrate a general workflow for automatic high-throughput drug screening with the following advantages: (i) the association of various exposure on diseases can be estimated; (ii) both repurposing and pharmacovigilance are integrated; (iii) accurate exposure length for each prescription is parsed from clinical texts; (iv) intrinsic relationship between drugs and diseases are removed jointly by bioinformatic mapping and large language model - ChatGPT; (v) causal-wise interpretations for incidence rate contrasts are provided.

Methods

Using a self-controlled cohort study design where subjects serve as their own control group, we tested the intention-to-treat association between medications on the incidence of diseases. Exposure length for each prescription is determined by parsing common dosages in English free text into a structured format. Exposure period starts from initial prescription to treatment discontinuation. A same exposure length preceding initial treatment is the control period. Clinical outcomes and categories are identified using existing phenotyping algorithms. Incident rate ratios (IRR) are tested using uniformly most powerful (UMP) unbiased tests.

Results

We assessed 3,444 medications on 276 diseases on 6,613,198 patients from the Clinical Practice Research Datalink (CPRD), an UK primary care electronic health records (EHR) spanning from 1987 to 2018. Due to the built-in selection bias of self-controlled cohort studies, ingredients-disease pairs confounded by deterministic medical relationships are removed by existing map from RxNorm and nonexistent maps by calling ChatGPT. A total of 16,901 drug-disease pairs reveals significant risk reduction, which can be considered as candidates for repurposing, while a total of 11,089 pairs showed significant risk increase, where drug safety might be of a concern instead.

Conclusions

This work developed a data-driven, nonparametric, hypothesis generating, and automatic high-throughput workflow, which reveals the potential of natural language processing in pharmacoepidemiology. We demonstrate the paradigm to a large observational health dataset to help discover potential novel therapies and adverse drug effects. The framework of this study can be extended to other observational medical databases.

Comparative safety and effectiveness of angiotensin converting enzyme inhibitors and thiazides and thiazide-like diuretics under strict monotherapy

Previous work comparing safety and effectiveness outcomes for new initiators of angiotensin converting-enzyme inhibitors (ACEi) and thiazides demonstrated more favorable outcomes for thiazides, although cohort definitions allowed for addition of a second antihypertensive medication after a week of monotherapy. Here, we modify the monotherapy definition, imposing exit from cohorts upon addition of another antihypertensive medication. We determine hazard ratios (HR) for 55 safety and effectiveness outcomes over six databases and compare results to earlier findings. We find, for all primary outcomes, statistically significant differences in effectiveness between ACEi and thiazides were not replicated (HRs: 1.11, 1.06, 1.12 for acute myocardial infarction, hospitalization with heart failure and stroke, respectively). While statistical significance is similarly lost for several safety outcomes, the safety profile of thiazides remains more favorable. Our results indicate a less striking difference in effectiveness of thiazides compared to ACEi and reflect some sensitivity to the monotherapy cohort definition modification.

The State of Use and Utility of Negative Controls in Pharmacoepidemiologic Studies

Uses of real-world data in drug safety and effectiveness studies are often challenged by various sources of bias. We undertook a systematic search of the published literature through September 2020 to evaluate the state of use and utility of negative controls to address bias in pharmacoepidemiologic studies. Two reviewers independently evaluated study eligibility and abstracted data. Our search identified 184 eligible studies for inclusion. Cohort studies (115, 63%) and administrative data (114, 62%) were, respectively, the most common study design and data type used. Most studies used negative control outcomes (91, 50%), and for most studies the target source of bias was unmeasured confounding (93, 51%). We identified 4 utility domains of negative controls: 1) bias detection (149, 81%), 2) bias correction (16, 9%), 3) P-value calibration (8, 4%), and 4) performance assessment of different methods used in drug safety studies (31, 17%). The most popular methodologies used were the 95% confidence interval and P-value calibration. In addition, we identified 2 reference sets with structured steps to check the causality assumption of the negative control. While negative controls are powerful tools in bias detection, we found many studies lacked checking the underlying assumptions. This article is part of a Special Collection on Pharmacoepidemiology.

Comparative effectiveness of first-line antihypertensive drug classes on the maintenance of estimated glomerular filtration rate (eGFR) in real world primary care

Renin-angiotensin system inhibitors (RASi), particularly angiotensin-converting enzyme inhibitors (ACEIs) and angiotensin II receptor blockers (ARBs), are commonly used in the treatment of hypertension and are recommended for kidney protection. Uncertainty remains about the effectiveness of RASi being used as first-line antihypertensive therapy on eGFR maintenance compared to its alternatives, especially for those with no or early-stage chronic kidney disease (CKD). We conducted a retrospective cohort study of 19,499 individuals (mean age 64.1, 43.5% males) from primary care in Singapore with 4.5 median follow-up years. The study cohort included newly diagnosed individuals with hypertension (whose eGFR was mainly in CKD stages G1-G2) and initiated on ACEIs, ARBs, beta-blockers (BBs), calcium channel blockers (CCBs) or diuretics (Ds) as first-line antihypertensive monotherapy. We compared the estimated glomerular filtration rate (eGFR) curve before/after the drug initiation over time of patients under different drug classes and analyzed the time to declining to a more advanced stage CKD. Inverse probability of treatment weighting (IPTW) was used to adjust for baseline confounding factors. Two key findings were observed. First, after initiating antihypertensive drugs, the eGFR almost maintained the same as the baseline in the first follow-up year, compared with dropping 3 mL/min/1.73 m2 per year before drug initiation. Second, ARBs were observed to be slightly inferior to ACEIs (HR = 1.14, 95% CI = (1.04, 1.23)) and other antihypertensive agents (HR = 1.10, 95% CI = (1.01, 1.20)) in delaying eGFR decline to a more advanced CKD stage in the study population. Our results showed that initiating antihypertensive agents can significantly maintain eGFR for those newly diagnosed patients with hypertension. However, RASi may not be superior to other antihypertensive agents in maintaining eGFR levels for non-CKD or early stages CKD patients.

Beta-2 adrenergic receptor agonism alters astrocyte phagocytic activity and has potential applications to psychiatric disease

Schizophrenia is a debilitating condition necessitating more efficacious therapies. Previous studies suggested that schizophrenia development is associated with aberrant synaptic pruning by glial cells. We pursued an interdisciplinary approach to understand whether therapeutic reduction in glial cell-specifically astrocytic-phagocytosis might benefit neuropsychiatric patients. We discovered that beta-2 adrenergic receptor (ADRB2) agonists reduced phagocytosis using a high-throughput, phenotypic screen of over 3200 compounds in primary human fetal astrocytes. We used protein interaction pathways analysis to associate ADRB2, to schizophrenia and endocytosis. We demonstrated that patients with a pediatric exposure to salmeterol, an ADRB2 agonist, had reduced in-patient psychiatry visits using a novel observational study in the electronic health record. We used a mouse model of inflammatory neurodegenerative disease and measured changes in proteins associated with endocytosis and vesicle-mediated transport after ADRB2 agonism. These results provide substantial rationale for clinical consideration of ADRB2 agonists as possible therapies for patients with schizophrenia.

Effects of Antidepressants on COVID-19 Outcomes: Retrospective Study on Large-Scale Electronic Health Record Data

BACKGROUND

Antidepressants exert an anticholinergic effect in varying degrees, and various classes of antidepressants can produce a different effect on immune function. While the early use of antidepressants has a notional effect on COVID-19 outcomes, the relationship between the risk of COVID-19 severity and the use of antidepressants has not been properly investigated previously owing to the high costs involved with clinical trials. Large-scale observational data and recent advancements in statistical analysis provide ample opportunity to virtualize a clinical trial to discover the detrimental effects of the early use of antidepressants.

OBJECTIVE

We primarily aimed to investigate electronic health records for causal effect estimation and use the data for discovering the causal effects of early antidepressant use on COVID-19 outcomes. As a secondary aim, we developed methods for validating our causal effect estimation pipeline.

METHODS

We used the National COVID Cohort Collaborative (N3C), a database aggregating health history for over 12 million people in the United States, including over 5 million with a positive COVID-19 test. We selected 241,952 COVID-19-positive patients (age >13 years) with at least 1 year of medical history. The study included a 18,584-dimensional covariate vector for each person and 16 different antidepressants. We used propensity score weighting based on the logistic regression method to estimate causal effects on the entire data. Then, we used the Node2Vec embedding method to encode SNOMED-CT (Systematized Nomenclature of Medicine-Clinical Terms) medical codes and applied random forest regression to estimate causal effects. We used both methods to estimate causal effects of antidepressants on COVID-19 outcomes. We also selected few negatively effective conditions for COVID-19 outcomes and estimated their effects using our proposed methods to validate their efficacy.

RESULTS

The average treatment effect (ATE) of using any one of the antidepressants was -0.076 (95% CI -0.082 to -0.069; P<.001) with the propensity score weighting method. For the method using SNOMED-CT medical embedding, the ATE of using any one of the antidepressants was -0.423 (95% CI -0.382 to -0.463; P<.001).

CONCLUSIONS

We applied multiple causal inference methods with novel application of health embeddings to investigate the effects of antidepressants on COVID-19 outcomes. Additionally, we proposed a novel drug effect analysis-based evaluation technique to justify the efficacy of the proposed method. This study offers causal inference methods on large-scale electronic health record data to discover the effects of common antidepressants on COVID-19 hospitalization or a worse outcome. We found that common antidepressants may increase the risk of COVID-19 complications and uncovered a pattern where certain antidepressants were associated with a lower risk of hospitalization. While discovering the detrimental effects of these drugs on outcomes could guide preventive care, identification of beneficial effects would allow us to propose drug repurposing for COVID-19 treatment.

A standardized framework for risk-based assessment of treatment effect heterogeneity in observational healthcare databases

Treatment effects are often anticipated to vary across groups of patients with different baseline risk. The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement focused on baseline risk as a robust predictor of treatment effect and provided guidance on risk-based assessment of treatment effect heterogeneity in a randomized controlled trial. The aim of this study is to extend this approach to the observational setting using a standardized scalable framework. The proposed framework consists of five steps: (1) definition of the research aim, i.e., the population, the treatment, the comparator and the outcome(s) of interest; (2) identification of relevant databases; (3) development of a prediction model for the outcome(s) of interest; (4) estimation of relative and absolute treatment effect within strata of predicted risk, after adjusting for observed confounding; (5) presentation of the results. We demonstrate our framework by evaluating heterogeneity of the effect of thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors on three efficacy and nine safety outcomes across three observational databases. We provide a publicly available R software package for applying this framework to any database mapped to the Observational Medical Outcomes Partnership Common Data Model. In our demonstration, patients at low risk of acute myocardial infarction receive negligible absolute benefits for all three efficacy outcomes, though they are more pronounced in the highest risk group, especially for acute myocardial infarction. Our framework allows for the evaluation of differential treatment effects across risk strata, which offers the opportunity to consider the benefit-harm trade-off between alternative treatments.

Methods for drug safety signal detection using routinely collected observational electronic health care data: A systematic review

PURPOSE

Signal detection is a crucial step in the discovery of post-marketing adverse drug reactions. There is a growing interest in using routinely collected data to complement established spontaneous report analyses. This work aims to systematically review the methods for drug safety signal detection using routinely collected healthcare data and their performance, both in general and for specific types of drugs and outcomes.

METHODS

We conducted a systematic review following the PRISMA guidelines, and registered a protocol in PROSPERO. MEDLINE, EMBASE, PubMed, Web of Science, Scopus, and the Cochrane Library were searched until July 13, 2021.

RESULTS

The review included 101 articles, among which there were 39 methodological works, 25 performance assessment papers, and 24 observational studies. Methods included adaptations from those used with spontaneous reports, traditional epidemiological designs, methods specific to signal detection with real-world data. More recently, implementations of machine learning have been studied in the literature. Twenty-five studies evaluated method performances, 16 of them using the area under the curve (AUC) for a range of positive and negative controls as their main measure. Despite the likelihood that performance measurement could vary by drug-event pair, only 10 studies reported performance stratified by drugs and outcomes, in a heterogeneous manner. The replicability of the performance assessment results was limited due to lack of transparency in reporting and the lack of a gold standard reference set.

CONCLUSIONS

A variety of methods have been described in the literature for signal detection with routinely collected data. No method showed superior performance in all papers and across all drugs and outcomes, performance assessment and reporting were heterogeneous. However, there is limited evidence that self-controlled designs, high dimensional propensity scores, and machine learning can achieve higher performances than other methods.

A network paradigm predicts drug synergistic effects using downstream protein-protein interactions

In some cases, drug combinations affect adverse outcome phenotypes by binding the same protein; however, drug-binding proteins are associated through protein-protein interaction (PPI) networks within the cell, suggesting that drug phenotypes may result from long-range network effects. We first used PPI network analysis to classify drugs based on proteins downstream of their targets and next predicted drug combination effects where drugs shared network proteins but had distinct binding proteins (e.g., targets, enzymes, or transporters). By classifying drugs using their downstream proteins, we had an 80.7% sensitivity for predicting rare drug combination effects documented in gold-standard datasets. We further measured the effect of predicted drug combinations on adverse outcome phenotypes using novel observational studies in the electronic health record. We tested predictions for 60 network-drug classes on seven adverse outcomes and measured changes in clinical outcomes for predicted combinations. These results demonstrate a novel paradigm for anticipating drug synergistic effects using proteins downstream of drug targets.

Adjusting for indirectly measured confounding using large-scale propensity score

Confounding remains one of the major challenges to causal inference with observational data. This problem is paramount in medicine, where we would like to answer causal questions from large observational datasets like electronic health records (EHRs) and administrative claims. Modern medical data typically contain tens of thousands of covariates. Such a large set carries hope that many of the confounders are directly measured, and further hope that others are indirectly measured through their correlation with measured covariates. How can we exploit these large sets of covariates for causal inference? To help answer this question, this paper examines the performance of the large-scale propensity score (LSPS) approach on causal analysis of medical data. We demonstrate that LSPS may adjust for indirectly measured confounders by including tens of thousands of covariates that may be correlated with them. We present conditions under which LSPS removes bias due to indirectly measured confounders, and we show that LSPS may avoid bias when inadvertently adjusting for variables (like colliders) that otherwise can induce bias. We demonstrate the performance of LSPS with both simulated medical data and real medical data.

Hip Fracture Risk After Treatment with Tramadol or Codeine: An Observational Study

INTRODUCTION

Hip fractures among older people are a major public health issue, which can impact quality of life and increase mortality within the year after they occur. A recent observational study found an increased risk of hip fracture in subjects who were new users of tramadol compared with codeine. These drugs have somewhat different indications. Tramadol is indicated for moderate to severe pain and can be used for an extended period; codeine is indicated for mild to moderate pain and cough suppression.

OBJECTIVE

In this observational study, we compared the risk of hip fracture in new users of tramadol or codeine, using multiple databases and analytical methods.

METHODS

Using data from the Clinical Practice Research Datalink and three US claims databases, we compared the risk of hip fracture after exposure to tramadol or codeine in subjects aged 50-89 years. To ensure comparability, large-scale propensity scores were used to adjust for confounding.

RESULTS

We observed a calibrated hazard ratio of 1.10 (95% calibrated confidence interval 0.99-1.21) in the Clinical Practice Research Datalink database, and a pooled estimate across the US databases yielded a calibrated hazard ratio of 1.06 (95% calibrated confidence interval 0.97-1.16).

CONCLUSIONS

Our results did not demonstrate a statistically significant difference between subjects treated for pain with tramadol compared with codeine for the outcome of hip fracture risk.

Large-scale evidence generation and evaluation across a network of databases for type 2 diabetes mellitus (LEGEND-T2DM): a protocol for a series of multinational, real-world comparative cardiovascular effectiveness and safety studies

INTRODUCTION

Therapeutic options for type 2 diabetes mellitus (T2DM) have expanded over the last decade with the emergence of cardioprotective novel agents, but without such data for older drugs, leaving a critical gap in our understanding of the relative effects of T2DM agents on cardiovascular risk.

METHODS AND ANALYSIS

The large-scale evidence generations across a network of databases for T2DM (LEGEND-T2DM) initiative is a series of systematic, large-scale, multinational, real-world comparative cardiovascular effectiveness and safety studies of all four major second-line anti-hyperglycaemic agents, including sodium-glucose co-transporter-2 inhibitor, glucagon-like peptide-1 receptor agonist, dipeptidyl peptidase-4 inhibitor and sulfonylureas. LEGEND-T2DM will leverage the Observational Health Data Sciences and Informatics (OHDSI) community that provides access to a global network of administrative claims and electronic health record data sources, representing 190 million patients in the USA and about 50 million internationally. LEGEND-T2DM will identify all adult, patients with T2DM who newly initiate a traditionally second-line T2DM agent. Using an active comparator, new-user cohort design, LEGEND-T2DM will execute all pairwise class-versus-class and drug-versus-drug comparisons in each data source, producing extensive study diagnostics that assess reliability and generalisability through cohort balance and equipoise to examine the relative risk of cardiovascular and safety outcomes. The primary cardiovascular outcomes include a composite of major adverse cardiovascular events and a series of safety outcomes. The study will pursue data-driven, large-scale propensity adjustment for measured confounding, a large set of negative control outcome experiments to address unmeasured and systematic bias.

ETHICS AND DISSEMINATION

The study ensures data safety through a federated analytic approach and follows research best practices, including prespecification and full disclosure of results. LEGEND-T2DM is dedicated to open science and transparency and will publicly share all analytic code from reproducible cohort definitions through turn-key software, enabling other research groups to leverage our methods, data and results to verify and extend our findings.

From random to predictive: a context-specific interaction framework improves selection of drug protein-protein interactions for unknown drug pathways

With high drug attrition, protein-protein interaction (PPI) network models are attractive as efficient methods for predicting drug outcomes by analyzing proteins downstream of drug targets. Unfortunately, these methods tend to overpredict associations and they have low precision and prediction performance; performance is often no better than random (AUROC ~0.5). Typically, PPI models identify ranked phenotypes associated with downstream proteins, yet methods differ in prioritization of downstream proteins. Most methods apply global approaches for assessing all phenotypes. We hypothesized that a per-phenotype analysis could improve prediction performance. We compared two global approaches-statistical and distance-based-and our novel per-phenotype approach, 'context-specific interaction' (CSI) analysis, on severe side effect prediction. We used a novel dataset of adverse events (or designated medical events, DMEs) and discovered that CSI had a 50% improvement over global approaches (AUROC 0.77 compared to 0.51), and a 76-95% improvement in average precision (0.499 compared to 0.284, 0.256). Our results provide a quantitative rationale for considering downstream proteins on a per-phenotype basis when using PPI network methods to predict drug phenotypes.

Computational drug repositioning of atorvastatin for ulcerative colitis

OBJECTIVE

Ulcerative colitis (UC) is a chronic inflammatory disorder with limited effective therapeutic options for long-term treatment and disease maintenance. We hypothesized that a multi-cohort analysis of independent cohorts representing real-world heterogeneity of UC would identify a robust transcriptomic signature to improve identification of FDA-approved drugs that can be repurposed to treat patients with UC.

MATERIALS AND METHODS

We performed a multi-cohort analysis of 272 colon biopsy transcriptome samples across 11 publicly available datasets to identify a robust UC disease gene signature. We compared the gene signature to in vitro transcriptomic profiles induced by 781 FDA-approved drugs to identify potential drug targets. We used a retrospective cohort study design modeled after a target trial to evaluate the protective effect of predicted drugs on colectomy risk in patients with UC from the Stanford Research Repository (STARR) database and Optum Clinformatics DataMart.

RESULTS

Atorvastatin treatment had the highest inverse-correlation with the UC gene signature among non-oncolytic FDA-approved therapies. In both STARR (n = 827) and Optum (n = 7821), atorvastatin intake was significantly associated with a decreased risk of colectomy, a marker of treatment-refractory disease, compared to patients prescribed a comparator drug (STARR: HR = 0.47, P = .03; Optum: HR = 0.66, P = .03), irrespective of age and length of atorvastatin treatment.

DISCUSSION & CONCLUSION

These findings suggest that atorvastatin may serve as a novel therapeutic option for ameliorating disease in patients with UC. Importantly, we provide a systematic framework for integrating publicly available heterogeneous molecular data with clinical data at a large scale to repurpose existing FDA-approved drugs for a wide range of human diseases.

Increasing trust in real-world evidence through evaluation of observational data quality

Objective

Advances in standardization of observational healthcare data have enabled methodological breakthroughs, rapid global collaboration, and generation of real-world evidence to improve patient outcomes. Standardizations in data structure, such as use of common data models, need to be coupled with standardized approaches for data quality assessment. To ensure confidence in real-world evidence generated from the analysis of real-world data, one must first have confidence in the data itself.

Materials and Methods

We describe the implementation of check types across a data quality framework of conformance, completeness, plausibility, with both verification and validation. We illustrate how data quality checks, paired with decision thresholds, can be configured to customize data quality reporting across a range of observational health data sources. We discuss how data quality reporting can become part of the overall real-world evidence generation and dissemination process to promote transparency and build confidence in the resulting output.

Results

The Data Quality Dashboard is an open-source R package that reports potential quality issues in an OMOP CDM instance through the systematic execution and summarization of over 3300 configurable data quality checks.

Discussion

Transparently communicating how well common data model-standardized databases adhere to a set of quality measures adds a crucial piece that is currently missing from observational research.

Conclusion

Assessing and improving the quality of our data will inherently improve the quality of the evidence we generate.

Comparative First-Line Effectiveness and Safety of ACE (Angiotensin-Converting Enzyme) Inhibitors and Angiotensin Receptor Blockers: A Multinational Cohort Study

[Figure: see text].

Comprehensive Comparative Effectiveness and Safety of First-Line β-Blocker Monotherapy in Hypertensive Patients: A Large-Scale Multicenter Observational Study

[Figure: see text].

Framework for identifying drug repurposing candidates from observational healthcare data

OBJECTIVE

Observational medical databases, such as electronic health records and insurance claims, track the healthcare trajectory of millions of individuals. These databases provide real-world longitudinal information on large cohorts of patients and their medication prescription history. We present an easy-to-customize framework that systematically analyzes such databases to identify new indications for on-market prescription drugs.

MATERIALS AND METHODS

Our framework provides an interface for defining study design parameters and extracting patient cohorts, disease-related outcomes, and potential confounders in observational databases. It then applies causal inference methodology to emulate hundreds of randomized controlled trials (RCTs) for prescribed drugs, while adjusting for confounding and selection biases. After correcting for multiple testing, it outputs the estimated effects and their statistical significance in each database.

RESULTS

We demonstrate the utility of the framework in a case study of Parkinson's disease (PD) and evaluate the effect of 259 drugs on various PD progression measures in two observational medical databases, covering more than 150 million patients. The results of these emulated trials reveal remarkable agreement between the two databases for the most promising candidates.

DISCUSSION

Estimating drug effects from observational data is challenging due to data biases and noise. To tackle this challenge, we integrate causal inference methodology with domain knowledge and compare the estimated effects in two separate databases.

CONCLUSION

Our framework enables systematic search for drug repurposing candidates by emulating RCTs using observational data. The high level of agreement between separate databases strongly supports the identified effects.

Comparative safety and effectiveness of alendronate versus raloxifene in women with osteoporosis

Alendronate and raloxifene are among the most popular anti-osteoporosis medications. However, there is a lack of head-to-head comparative effectiveness studies comparing the two treatments. We conducted a retrospective large-scale multicenter study encompassing over 300 million patients across nine databases encoded in the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). The primary outcome was the incidence of osteoporotic hip fracture, while secondary outcomes were vertebral fracture, atypical femoral fracture (AFF), osteonecrosis of the jaw (ONJ), and esophageal cancer. We used propensity score trimming and stratification based on an expansive propensity score model with all pre-treatment patient characteritistcs. We accounted for unmeasured confounding using negative control outcomes to estimate and adjust for residual systematic bias in each data source. We identified 283,586 alendronate patients and 40,463 raloxifene patients. There were 7.48 hip fracture, 8.18 vertebral fracture, 1.14 AFF, 0.21 esophageal cancer and 0.09 ONJ events per 1,000 person-years in the alendronate cohort and 6.62, 7.36, 0.69, 0.22 and 0.06 events per 1,000 person-years, respectively, in the raloxifene cohort. Alendronate and raloxifene have a similar hip fracture risk (hazard ratio [HR] 1.03, 95% confidence interval [CI] 0.94–1.13), but alendronate users are more likely to have vertebral fractures (HR 1.07, 95% CI 1.01–1.14). Alendronate has higher risk for AFF (HR 1.51, 95% CI 1.23–1.84) but similar risk for esophageal cancer (HR 0.95, 95% CI 0.53–1.70), and ONJ (HR 1.62, 95% CI 0.78–3.34). We demonstrated substantial control of measured confounding by propensity score adjustment, and minimal residual systematic bias through negative control experiments, lending credibility to our effect estimates. Raloxifene is as effective as alendronate and may remain an option in the prevention of osteoporotic fracture.

Comparison of Cardiovascular and Safety Outcomes of Chlorthalidone vs Hydrochlorothiazide to Treat Hypertension

IMPORTANCE

Chlorthalidone is currently recommended as the preferred thiazide diuretic to treat hypertension, but no trials have directly compared risks and benefits.

OBJECTIVE

To compare the effectiveness and safety of chlorthalidone and hydrochlorothiazide as first-line therapies for hypertension in real-world practice.

DESIGN, SETTING, AND PARTICIPANTS

This is a Large-Scale Evidence Generation and Evaluation in a Network of Databases (LEGEND) observational comparative cohort study with large-scale propensity score stratification and negative-control and synthetic positive-control calibration on databases spanning January 2001 through December 2018. Outpatient and inpatient care episodes of first-time users of antihypertensive monotherapy in the United States based on 2 administrative claims databases and 1 collection of electronic health records were analyzed. Analysis began June 2018.

EXPOSURES

Chlorthalidone and hydrochlorothiazide.

MAIN OUTCOMES AND MEASURES

The primary outcomes were acute myocardial infarction, hospitalization for heart failure, ischemic or hemorrhagic stroke, and a composite cardiovascular disease outcome including the first 3 outcomes and sudden cardiac death. Fifty-one safety outcomes were measured.

RESULTS

Of 730 225 individuals (mean [SD] age, 51.5 [13.3] years; 450 100 women [61.6%]), 36 918 were dispensed or prescribed chlorthalidone and had 149 composite outcome events, and 693 337 were dispensed or prescribed hydrochlorothiazide and had 3089 composite outcome events. No significant difference was found in the associated risk of myocardial infarction, hospitalized heart failure, or stroke, with a calibrated hazard ratio for the composite cardiovascular outcome of 1.00 for chlorthalidone compared with hydrochlorothiazide (95% CI, 0.85-1.17). Chlorthalidone was associated with a significantly higher risk of hypokalemia (hazard ratio [HR], 2.72; 95% CI, 2.38-3.12), hyponatremia (HR, 1.31; 95% CI, 1.16-1.47), acute renal failure (HR, 1.37; 95% CI, 1.15-1.63), chronic kidney disease (HR, 1.24; 95% CI, 1.09-1.42), and type 2 diabetes mellitus (HR, 1.21; 95% CI, 1.12-1.30). Chlorthalidone was associated with a significantly lower risk of diagnosed abnormal weight gain (HR, 0.73; 95% CI, 0.61-0.86).

CONCLUSIONS AND RELEVANCE

This study found that chlorthalidone use was not associated with significant cardiovascular benefits when compared with hydrochlorothiazide, while its use was associated with greater risk of renal and electrolyte abnormalities. These findings do not support current recommendations to prefer chlorthalidone vs hydrochlorothiazide for hypertension treatment in first-time users was found. We used advanced methods, sensitivity analyses, and diagnostics, but given the possibility of residual confounding and the limited length of observation periods, further study is warranted.

Empirical assessment of case-based methods for drug safety alert identification in the French National Healthcare System database (SNDS): Methodology of the ALCAPONE project

OBJECTIVES

To introduce the methodology of the ALCAPONE project.

BACKGROUND

The French National Healthcare System Database (SNDS), covering 99% of the French population, provides a potentially valuable opportunity for drug safety alert generation. ALCAPONE aimed to assess empirically in the SNDS case-based designs for alert generation related to four health outcomes of interest.

METHODS

ALCAPONE used a reference set adapted from observational medical outcomes partnership (OMOP) and Exploring and Understanding Adverse Drug Reactions (EU-ADR) project, with four outcomes-acute liver injury (ALI), myocardial infarction (MI), acute kidney injury (AKI), and upper gastrointestinal bleeding (UGIB)-and positive and negative drug controls. ALCAPONE consisted of four main phases: (1) data preparation to fit the OMOP Common Data Model and select the drug controls; (2) detection of the selected controls via three case-based designs: case-population, case-control, and self-controlled case series, including design variants (varying risk window, adjustment strategy, etc.); (3) comparison of design variant performance (area under the ROC curve, mean square error, etc.); and (4) selection of the optimal design variants and their calibration for each outcome.

RESULTS

Over 2009-2014, 5225 cases of ALI, 354 109 MI, 12 633 AKI, and 156 057 UGIB were identified using specific definitions. The number of detectable drugs ranged from 61 for MI to 25 for ALI. Design variants generated more than 50 000 points estimates. Results by outcome will be published in forthcoming papers.

CONCLUSIONS

ALCAPONE has shown the interest of the empirical assessment of pharmacoepidemiological approaches for drug safety alert generation and may encourage other researchers to do the same in other databases.

How Confident Are We about Observational Findings in Healthcare: A Benchmark Study

Healthcare professionals increasingly rely on observational healthcare data, such as administrative claims and electronic health records, to estimate the causal effects of interventions. However, limited prior studies raise concerns about the real-world performance of the statistical and epidemiological methods that are used. We present the "OHDSI Methods Benchmark" that aims to evaluate the performance of effect estimation methods on real data. The benchmark comprises a gold standard, a set of metrics, and a set of open source software tools. The gold standard is a collection of real negative controls (drug-outcome pairs where no causal effect appears to exist) and synthetic positive controls (drug-outcome pairs that augment negative controls with simulated causal effects). We apply the benchmark using four large healthcare databases to evaluate methods commonly used in practice: the new-user cohort, self-controlled cohort, case-control, case-crossover, and self-controlled case series designs. The results confirm the concerns about these methods, showing that for most methods the operating characteristics deviate considerably from nominal levels. For example, in most contexts, only half of the 95% confidence intervals we calculated contain the corresponding true effect size. We previously developed an "empirical calibration" procedure to restore these characteristics and we also evaluate this procedure. While no one method dominates, self-controlled methods such as the empirically calibrated self-controlled case series perform well across a wide range of scenarios.

Signal Detection of Potentially Drug-Induced Liver Injury in Children Using Electronic Health Records

Background: This study proposes a quantitative 2-stage procedure to detect potential drug-induced liver injury (DILI) signals in pediatric inpatients using an data warehouse of electronic health records (EHRs). Methods: Eight years of medical data from a constructed database were used. A two-stage procedure was adopted: (i) stage 1: the drugs suspected of inducing DILI were selected and (ii) stage 2: the associations between the drugs and DILI were identified in a retrospective cohort study. Results: 1,196 drugs were filtered initially and 12 drugs were further potentially identified as suspect drugs inducing DILI. Eleven drugs (fluconazole, omeprazole, sulfamethoxazole, vancomycin, granulocyte colony-stimulating factor (G-CSF), acetaminophen, nifedipine, fusidine, oseltamivir, nystatin and meropenem) were showed to be associated with DILI. Of these, two drugs, nystatin [odds ratio[OR]=1.39, 95%CI:1.10-1.75] and G-CSF (OR = 1.91, 95%CI:1.55-2.35), were found to be new potential signals in adults and children. Three drugs [nifedipine [OR = 1.77, 95%CI:1.26-2.46], fusidine [OR = 1.43, 95%CI:1.08-1.86], and oseltamivi r [OR = 1.64, 95%CI:1.23-2.18]] were demonstrated to be new signals in pediatrics. The other drug-DILI associations had been confirmed in previous studies. Conclusions: A quantitative algorithm to detect potential signals of DILI has been described. Our work promotes the application of EHR data in pharmacovigilance and provides candidate drugs for further causality assessment studies.

Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis

BACKGROUND

Uncertainty remains about the optimal monotherapy for hypertension, with current guidelines recommending any primary agent among the first-line drug classes thiazide or thiazide-like diuretics, angiotensin-converting enzyme inhibitors, angiotensin receptor blockers, dihydropyridine calcium channel blockers, and non-dihydropyridine calcium channel blockers, in the absence of comorbid indications. Randomised trials have not further refined this choice.

METHODS

We developed a comprehensive framework for real-world evidence that enables comparative effectiveness and safety evaluation across many drugs and outcomes from observational data encompassing millions of patients, while minimising inherent bias. Using this framework, we did a systematic, large-scale study under a new-user cohort design to estimate the relative risks of three primary (acute myocardial infarction, hospitalisation for heart failure, and stroke) and six secondary effectiveness and 46 safety outcomes comparing all first-line classes across a global network of six administrative claims and three electronic health record databases. The framework addressed residual confounding, publication bias, and p-hacking using large-scale propensity adjustment, a large set of control outcomes, and full disclosure of hypotheses tested.

FINDINGS

Using 4·9 million patients, we generated 22 000 calibrated, propensity-score-adjusted hazard ratios (HRs) comparing all classes and outcomes across databases. Most estimates revealed no effectiveness differences between classes; however, thiazide or thiazide-like diuretics showed better primary effectiveness than angiotensin-converting enzyme inhibitors: acute myocardial infarction (HR 0·84, 95% CI 0·75-0·95), hospitalisation for heart failure (0·83, 0·74-0·95), and stroke (0·83, 0·74-0·95) risk while on initial treatment. Safety profiles also favoured thiazide or thiazide-like diuretics over angiotensin-converting enzyme inhibitors. The non-dihydropyridine calcium channel blockers were significantly inferior to the other four classes.

INTERPRETATION

This comprehensive framework introduces a new way of doing observational health-care science at scale. The approach supports equivalence between drug classes for initiating monotherapy for hypertension-in keeping with current guidelines, with the exception of thiazide or thiazide-like diuretics superiority to angiotensin-converting enzyme inhibitors and the inferiority of non-dihydropyridine calcium channel blockers.

FUNDING

US National Science Foundation, US National Institutes of Health, Janssen Research & Development, IQVIA, South Korean Ministry of Health & Welfare, Australian National Health and Medical Research Council.

Choosing Among Common Data Models for Real-World Data Analyses Fit for Making Decisions About the Effectiveness of Medical Products

Many real-world data analyses use common data models (CDMs) to standardize terminologies for medication use, medical events and procedures, data structures, and interpretations of data to facilitate analyses across data sources. For decision makers, key aspects that influence the choice of a CDM may include (i) adaptability to a specific question; (ii) transparency to reproduce findings, assess validity, and instill confidence in findings; and (iii) ease and speed of use. Organizing CDMs preserve the original information from a data source and have maximum adaptability. Full mapping data models, or preconfigured rules systems, are easy to use, since all raw codes are mapped to medical constructs. Adaptive rule systems grow libraries of reusable measures that can easily adjust to preserve adaptability, expedite analyses, and ensure study-specific transparency.

A Web-Based Clinical System for Cohort Surveillance of Specific Clinical Effectiveness and Safety Outcomes: A Cohort Study of Non-Vitamin K Antagonist Oral Anticoagulants and Warfarin

Background

Conventional systems of drug surveillance lack a seamless workflow, which makes it crucial to have an active drug surveillance system that proactively assesses adverse drug events.

Objective

The aim of this study was to develop a seamless, Web-based workflow for comparing the safety and effectiveness of drugs in a database of electronic medical records.

Methods

We proposed a comprehensive integration process for cohort surveillance using the National Taiwan University Hospital Clinical Surveillance System (NCSS). We studied a practical application of the NCSS that evaluates the drug safety and effectiveness of novel oral anticoagulants (NOACs) and warfarin by cohort tree analysis in an efficient and interoperable platform.

Results

We demonstrated a practical example of investigating the differences in effectiveness and safety between NOACs and warfarin in patients with nonvalvular atrial fibrillation (AF) using the NCSS. We efficiently identified 2357 patients with nonvalvular AF with newly prescribed oral anticoagulants between 2010 and 2015 and further developed 1 main cohort and 2 subcohorts for separately measuring ischemic stroke as the clinical effectiveness outcome and intracranial hemorrhage (ICH) as the safety outcome. In the subcohort of ischemic stroke, NOAC users exhibited a significantly lower risk of ischemic stroke than warfarin users after adjusting for age, sex, comorbidity, and comedication in an intention-to-treat (ITT) analysis (P=.01) but did not exhibit a significantly distinct risk in an as-treated (AT) analysis (P=.12) after the 2-year follow-up. In the subcohort of ICH, NOAC users did not exhibit a different risk of ICH both in ITT (P=.68) and AT analyses (P=.15).

Conclusions

With a seamless and Web-based workflow, the NCSS can serve the critical role of forming associations between evidence and the real world at a medical center in Taiwan.

Evaluating large-scale propensity score performance through real-world and synthetic data experiments

Background

Propensity score adjustment is a popular approach for confounding control in observational studies. Reliable frameworks are needed to determine relative propensity score performance in large-scale studies, and to establish optimal propensity score model selection methods.

Methods

We detail a propensity score evaluation framework that includes synthetic and real-world data experiments. Our synthetic experimental design extends the 'plasmode' framework and simulates survival data under known effect sizes, and our real-world experiments use a set of negative control outcomes with presumed null effect sizes. In reproductions of two published cohort studies, we compare two propensity score estimation methods that contrast in their model selection approach: L1-regularized regression that conducts a penalized likelihood regression, and the 'high-dimensional propensity score' (hdPS) that employs a univariate covariate screen. We evaluate methods on a range of outcome-dependent and outcome-independent metrics.

Results

L1-regularization propensity score methods achieve superior model fit, covariate balance and negative control bias reduction compared with the hdPS. Simulation results are mixed and fluctuate with simulation parameters, revealing a limitation of simulation under the proportional hazards framework. Including regularization with the hdPS reduces commonly reported non-convergence issues but has little effect on propensity score performance.

Conclusions

L1-regularization incorporates all covariates simultaneously into the propensity score model and offers propensity score performance superior to the hdPS marginal screen.

Assessing performance of sequential analysis methods for active drug safety surveillance using observational data

The routine use of sequential methods is well established in clinical studies. Recently, there has been increasing interest in applying these methods to prospectively monitor the safety of newly approved drugs through accrual of real-world data. However, the application to marketed drugs using real-world data has been limited and work is needed to determine which sequential approaches are most suited to such data. In this study, the conditional sequential sampling procedure (CSSP), a group sequential method, was compared with a log-linear model with Poisson distribution (LLMP) through a SAS procedure (PROC GENMOD) combined with an alpha-spending function on two large longitudinal US administrative health claims databases. Relative performance in identifying known drug-outcome associations was examined using a set of 50 well-studied drug-outcome pairs. The study finds that neither method correctly identified all pairs but that LLMP often provides better ability and shorter time for identifying the known drug-outcome associations with superior computational performance when compared with CSSP, albeit with more false positives. With the features of flexible confounding control and ease of implementation, LLMP may be a good alternative or complement to CSSP.

Pharmacovigilance via Baseline Regularization with Large-Scale Longitudinal Observational Data

Several prominent public health hazards [29] that occurred at the beginning of this century due to adverse drug events (ADEs) have raised international awareness of governments and industries about pharmacovigilance (PhV) [6,7], the science and activities to monitor and prevent adverse events caused by pharmaceutical products after they are introduced to the market. A major data source for PhV is large-scale longitudinal observational databases (LODs) [6] such as electronic health records (EHRs) and medical insurance claim databases. Inspired by the Self-Controlled Case Series (SCCS) model [27], arguably the leading method for ADE discovery from LODs, we propose baseline regularization, a regularized generalized linear model that leverages the diverse health profiles available in LODs across different individuals at different times. We apply the proposed method as well as SCCS to the Marshfield Clinic EHR. Experimental results suggest that the proposed method outperforms SCCS under various settings in identifying benchmark ADEs from the Observational Medical Outcomes Partnership ground truth [26].

Case-crossover study to examine the change in postpartum risk of pulmonary embolism over time

BACKGROUND

Although the current guidelines recommend anticoagulation up until 6 weeks after delivery in women at high risk of venous thromboembolism (VTE), the risk of VTE may extend beyond 6 weeks. Our objective was to estimate the risk of a pulmonary embolism in successive 2-week intervals during the postpartum period.

METHODS

In a population-based, case-crossover study, we analyzed the French national inpatient database from 2007 to 2013 (n = 5,517,680 singleton deliveries). Using ICD-10 codes, we identified women who were diagnosed with a postpartum pulmonary embolism between July 1^st, 2008, and December 31^st, 2013. Deliveries were identified during a case "period" immediately before the pulmonary embolism, and five different control periods one year before the pulmonary embolism. Using conditional logistic regression, Odds ratios (ORs) and 95% confidential intervals (CIs) were estimated for ten successive 2-week intervals that preceded the diagnosis of pulmonary embolism.

RESULTS

We identified 167,103 cases with a pulmonary embolism during the inclusion period. After delivery, the risk of pulmonary embolism declined progressively over time, with an OR [95%CI] of 17.2 [14.0-21.3] in postpartum weeks 1 to 2 and 1.9 [1.4-2.7] in postpartum weeks 11 to 12. The OR [95%CI] in postpartum weeks 13 to 14 was 1.4 [0.9-2.0], and the OR did not fall significantly after postpartum week 14.

CONCLUSIONS

Our findings indicate that women are at risk of a pulmonary embolism up to 12 weeks after delivery. The shape of the risk curve suggests that the risk decreases exponentially over time. Future research is needed to establish whether the duration of postpartum anticoagulation should be extended beyond 6 weeks.

Quantification of missing prescriptions in commercial claims databases: results of a cohort study

PURPOSE

This study aims to quantify the magnitude of missed dispensings in commercial claims databases.

METHODS

A retrospective cohort study has been used linking PharMetrics, a commercial claims database, to a prescription database (LRx) that captures pharmacy dispensings independently of payment method, including cash transactions. We included adults with dispensings for opioids, diuretics, antiplatelet medications, or anticoagulants. To determine the degree of capture of dispensings, we calculated the number of subjects with the following: (1) same number of dispensings in both databases; (2) at least one dispensing, but not all dispensings, missed in PharMetrics; and (3) all dispensings missing in PharMetrics. Similar analyses were conducted using dispensings as the unit of analysis. To assess whether a dispensing in LRx was in PharMetrics, the dispensing in PharMetrics had to be for the same medication class and within ±7 days in LRx.

RESULTS

A total of 1 426 498 subjects were included. Overall, 68% of subjects had the same number of dispensings in both databases. In 13% of subjects, PharMetrics identified ≥1 dispensing but also missed ≥1 dispensing. In 19% of the subjects, PharMetrics missed all the dispensings. Taking dispensings as the unit of analysis, 25% of the dispensings present in LRx were not captured in PharMetrics. These patterns were similar across all four classes of medications. Of the dispensings missing in PharMetrics, 48% involved a subject who had >1 health insurance plan.

CONCLUSIONS

Commercial claims databases provide an incomplete picture of all prescriptions dispensed to patients. The lack of capture goes beyond cash transactions and potentially introduces substantial misclassification bias. © 2017 The Authors. Pharmacoepidemiology & Drug Safety Published by John Wiley & Sons Ltd.

Risk of acute liver injury associated with use of antibiotics. Comparative cohort and nested case-control studies using two primary care databases in Europe

PURPOSE

To assess the impact of varying study designs, exposure and outcome definitions on the risk of acute liver injury (ALI) associated with antibiotic use.

METHODS

The source population comprised of patients registered in two primary care databases, in the UK and in Spain. We identified a cohort consisting of new users of antibiotics during the study period (2004-2009) and non-users during the study period or in the previous year. Cases with ALI were identified within this cohort and classified as definite or probable, based on recorded medical information. The relative risk (RR) of ALI associated with antibiotic use was computed using Poisson regression. For the nested case-control analyses, up to five controls were matched to each case by age, sex, date and practice (in CPRD) and odds ratios (OR) were computed with conditional logistic regression.

RESULTS

The age, sex and year adjusted RRs of definite ALI in the current antibiotic use periods was 10.04 (95% CI: 6.97-14.47) in CPRD and 5.76 (95% CI: 3.46-9.59) in BIFAP. In the case-control analyses adjusting for life-style, comorbidities and use of medications, the OR of ALI for current users of antibiotics was and 5.7 (95% CI: 3.46-9.36) in CPRD and 2.6 (95% CI: 1.26-5.37) in BIFAP.

CONCLUSION

Guided by a common protocol, both cohort and case-control study designs found an increased risk of ALI associated with the use of antibiotics in both databases, independent of the exposure and case definitions used. However, the magnitude of the risk was higher in CPRD compared to BIFAP.

Risk patterns in drug safety study using relative times by accelerated failure time models when proportional hazards assumption is questionable: an illustrative case study of cancer risk of patients on glucose-lowering therapies

Observational drug safety studies may be susceptible to confounding or protopathic bias. This bias may cause a spurious relationship between drug exposure and adverse side effect when none exists and may lead to unwarranted safety alerts. The spurious relationship may manifest itself through substantially different risk levels between exposure groups at the start of follow-up when exposure is deemed too short to have any plausible biological effect of the drug. The restrictive proportional hazards assumption with its arbitrary choice of baseline hazard function renders the commonly used Cox proportional hazards model of limited use for revealing such potential bias. We demonstrate a fully parametric approach using accelerated failure time models with an illustrative safety study of glucose-lowering therapies and show that its results are comparable against other methods that allow time-varying exposure effects. Our approach includes a wide variety of models that are based on the flexible generalized gamma distribution and allows direct comparisons of estimated hazard functions following different exposure-specific distributions of survival times. This approach lends itself to two alternative metrics, namely relative times and difference in times to event, allowing physicians more ways to communicate patient's prognosis without invoking the concept of risks, which some may find hard to grasp. In our illustrative case study, substantial differences in cancer risks at drug initiation followed by a gradual reduction towards null were found. This evidence is compatible with the presence of protopathic bias, in which undiagnosed symptoms of cancer lead to switches in diabetes medication.

Bridging islands of information to establish an integrated knowledge base of drugs and health outcomes of interest

The entire drug safety enterprise has a need to search, retrieve, evaluate, and synthesize scientific evidence more efficiently. This discovery and synthesis process would be greatly accelerated through access to a common framework that brings all relevant information sources together within a standardized structure. This presents an opportunity to establish an open-source community effort to develop a global knowledge base, one that brings together and standardizes all available information for all drugs and all health outcomes of interest (HOIs) from all electronic sources pertinent to drug safety. To make this vision a reality, we have established a workgroup within the Observational Health Data Sciences and Informatics (OHDSI, http://ohdsi.org) collaborative. The workgroup’s mission is to develop an open-source standardized knowledge base for the effects of medical products and an efficient procedure for maintaining and expanding it. The knowledge base will make it simpler for practitioners to access, retrieve, and synthesize evidence so that they can reach a rigorous and accurate assessment of causal relationships between a given drug and HOI. Development of the knowledge base will proceed with the measureable goal of supporting an efficient and thorough evidence-based assessment of the effects of 1,000 active ingredients across 100 HOIs. This non-trivial task will result in a high-quality and generally applicable drug safety knowledge base. It will also yield a reference standard of drug–HOI pairs that will enable more advanced methodological research that empirically evaluates the performance of drug safety analysis methods.

Computational approaches for pharmacovigilance signal detection: toward integrated and semantically-enriched frameworks

Computational signal detection constitutes a key element of postmarketing drug monitoring and surveillance. Diverse data sources are considered within the 'search space' of pharmacovigilance scientists, and respective data analysis methods are employed, all with their qualities and shortcomings, towards more timely and accurate signal detection. Recent systematic comparative studies highlighted not only event-based and data-source-based differential performance across methods but also their complementarity. These findings reinforce the arguments for exploiting all possible information sources for drug safety and the parallel use of multiple signal detection methods. Combinatorial signal detection has been pursued in few studies up to now, employing a rather limited number of methods and data sources but illustrating well-promising outcomes. However, the large-scale realization of this approach requires systematic frameworks to address the challenges of the concurrent analysis setting. In this paper, we argue that semantic technologies provide the means to address some of these challenges, and we particularly highlight their contribution in (a) annotating data sources and analysis methods with quality attributes to facilitate their selection given the analysis scope; (b) consistently defining study parameters such as health outcomes and drugs of interest, and providing guidance for study setup; (c) expressing analysis outcomes in a common format enabling data sharing and systematic comparisons; and (d) assessing/supporting the novelty of the aggregated outcomes through access to reference knowledge sources related to drug safety. A semantically-enriched framework can facilitate seamless access and use of different data sources and computational methods in an integrated fashion, bringing a new perspective for large-scale, knowledge-intensive signal detection.

Defining a reference set to support methodological research in drug safety

BACKGROUND

Methodological research to evaluate the performance of methods requires a benchmark to serve as a referent comparison. In drug safety, the performance of analyses of spontaneous adverse event reporting databases and observational healthcare data, such as administrative claims and electronic health records, has been limited by the lack of such standards.

OBJECTIVES

To establish a reference set of test cases that contain both positive and negative controls, which can serve the basis for methodological research in evaluating methods performance in identifying drug safety issues.

RESEARCH DESIGN

Systematic literature review and natural language processing of structured product labeling was performed to identify evidence to support the classification of drugs as either positive controls or negative controls for four outcomes: acute liver injury, acute kidney injury, acute myocardial infarction, and upper gastrointestinal bleeding.

RESULTS

Three-hundred and ninety-nine test cases comprised of 165 positive controls and 234 negative controls were identified across the four outcomes. The majority of positive controls for acute kidney injury and upper gastrointestinal bleeding were supported by randomized clinical trial evidence, while the majority of positive controls for acute liver injury and acute myocardial infarction were only supported based on published case reports. Literature estimates for the positive controls shows substantial variability that limits the ability to establish a reference set with known effect sizes.

CONCLUSIONS

A reference set of test cases can be established to facilitate methodological research in drug safety. Creating a sufficient sample of drug-outcome pairs with binary classification of having no effect (negative controls) or having an increased effect (positive controls) is possible and can enable estimation of predictive accuracy through discrimination. Since the magnitude of the positive effects cannot be reliably obtained and the quality of evidence may vary across outcomes, assumptions are required to use the test cases in real data for purposes of measuring bias, mean squared error, or coverage probability.