Design considerations in an active medical product safety monitoring system

Information-based group sequential design for post-market safety monitoring of medical products using real world data

Real world healthcare data are commonly used in post-market safety monitoring studies to address potential safety issues related to newly approved medical products. Such studies typically involve repeated evaluations of accumulating safety data with respect to pre-defined hypotheses, for which the group sequential design provides a rigorous and flexible statistical framework. A major challenge in designing a group sequential safety monitoring study is the uncertainty associated with product uptake, which makes it difficult to specify the final sample size or maximum duration of the study. To deal with this challenge, we propose an information-based group sequential design which specifies a target amount of information that would produce adequate power for detecting a clinically significant effect size. At each interim analysis, the variance estimate for the treatment effect of interest is used to compute the current information time, and a pre-specified alpha spending function is used to determine the stopping boundary. The proposed design can be applied to regression models that adjust for potential confounders and/or heterogeneous treatment exposure. Simulation results demonstrate that the proposed design performs reasonably well in realistic settings.

Research on product design of FAHP bone marrow aspiration needle

Bone marrow aspiration is a crucial medical procedure to obtain bone marrow samples for diagnosis and treatment. However, traditional bone marrow aspiration needles face several challenges such as operational difficulties, inadequate sample acquisition, and patient discomfort. To address these issues, we aimed to design a bone marrow aspiration needle product by using fuzzy analytic hierarchical process (FAHP). The FAHP method was used to identify key factors in the design of the bone marrow aspiration needle, including technicality, usage, and application characteristics. The importance weights and priorities of each factor were determined through questionnaires and interviews with experts. A new bone marrow aspiration needle product was developed based on the results of the FAHP. The new product design considers the weights and priorities assigned to key factors, resulting in improved convenience during operation and a higher success rate of sample acquisition. This was achieved by optimising the structure and material selection of the needle. This study presents a novel bone marrow puncture needle product that effectively integrates the importance and priority of the key factors. It successfully enhances operational performance and patient experience, thereby offering an innovative solution to improve the success rate and therapeutic effect of bone marrow punctures.

Large-scale evidence generation and evaluation across a network of databases for type 2 diabetes mellitus (LEGEND-T2DM): a protocol for a series of multinational, real-world comparative cardiovascular effectiveness and safety studies

INTRODUCTION

Therapeutic options for type 2 diabetes mellitus (T2DM) have expanded over the last decade with the emergence of cardioprotective novel agents, but without such data for older drugs, leaving a critical gap in our understanding of the relative effects of T2DM agents on cardiovascular risk.

METHODS AND ANALYSIS

The large-scale evidence generations across a network of databases for T2DM (LEGEND-T2DM) initiative is a series of systematic, large-scale, multinational, real-world comparative cardiovascular effectiveness and safety studies of all four major second-line anti-hyperglycaemic agents, including sodium-glucose co-transporter-2 inhibitor, glucagon-like peptide-1 receptor agonist, dipeptidyl peptidase-4 inhibitor and sulfonylureas. LEGEND-T2DM will leverage the Observational Health Data Sciences and Informatics (OHDSI) community that provides access to a global network of administrative claims and electronic health record data sources, representing 190 million patients in the USA and about 50 million internationally. LEGEND-T2DM will identify all adult, patients with T2DM who newly initiate a traditionally second-line T2DM agent. Using an active comparator, new-user cohort design, LEGEND-T2DM will execute all pairwise class-versus-class and drug-versus-drug comparisons in each data source, producing extensive study diagnostics that assess reliability and generalisability through cohort balance and equipoise to examine the relative risk of cardiovascular and safety outcomes. The primary cardiovascular outcomes include a composite of major adverse cardiovascular events and a series of safety outcomes. The study will pursue data-driven, large-scale propensity adjustment for measured confounding, a large set of negative control outcome experiments to address unmeasured and systematic bias.

ETHICS AND DISSEMINATION

The study ensures data safety through a federated analytic approach and follows research best practices, including prespecification and full disclosure of results. LEGEND-T2DM is dedicated to open science and transparency and will publicly share all analytic code from reproducible cohort definitions through turn-key software, enabling other research groups to leverage our methods, data and results to verify and extend our findings.

Differential anchoring effects of vaccination comparator selection: characterizing a potential bias due to healthcare utilization in COVID-19 versus influenza

Background

Observational data enables large-scale vaccine safety surveillance but requires careful evaluation of the potential sources of bias. One potential source of bias is the index date selection procedure for the unvaccinated cohort or unvaccinated comparison time (“anchoring”).

Objective

Here, we evaluated the different index date selection procedures for 2 vaccinations: COVID-19 and influenza.

Methods

For each vaccine, we extracted patient baseline characteristics on the index date and up to 450 days prior and then compared them to the characteristics of the unvaccinated patients indexed on (1) an arbitrary date or (2) a date of a visit. Additionally, we compared vaccinated patients indexed on the date of vaccination and the same patients indexed on a prior date or visit.

Results

COVID-19 vaccination and influenza vaccination differ drastically from each other in terms of the populations vaccinated and their status on the day of vaccination. When compared to indexing on a visit in the unvaccinated population, influenza vaccination had markedly higher covariate proportions, and COVID-19 vaccination had lower proportions of most covariates on the index date. In contrast, COVID-19 vaccination had similar covariate proportions when compared to an arbitrary date. These effects attenuated, but were still present, with a longer lookback period. The effect of day 0 was present even when the patients served as their own controls.

Conclusions

Patient baseline characteristics are sensitive to the choice of the index date. In vaccine safety studies, unexposed index event should represent vaccination settings. Study designs previously used to assess influenza vaccination must be reassessed for COVID-19 to account for a potentially healthier population and lack of medical activity on the day of vaccination.

Initiation of Opioid Prescription and Risk of Suicidal Behavior Among Youth and Young Adults

BACKGROUND AND OBJECTIVES

Opioids are involved in an increasing proportion of suicide deaths. This study examined the association between opioid analgesic prescription initiation and suicidal behavior among young people.

METHODS

We analyzed Swedish population-register data on 1 895 984 individuals ages 9 to 29 years without prior recorded opioid prescriptions. We identified prescriptions dispensed from January 2007 onward and diagnosed self-injurious behavior and death by suicide through December 2013. We first compared initiators with demographically matched noninitiators. To account for confounding, we applied an active comparator design, which examined suicidal behavior among opioid initiators relative to prescription nonsteroidal antiinflammatory drug (NSAID) initiators while inverse-probability-of-treatment weighting with individual and familial covariates.

RESULTS

Among the cohort, 201 433 individuals initiated opioid prescription. Relative to demographically matched noninitiators, initiators (N = 180 808) had more than doubled risk of incident suicidal behavior (hazard ratio = 2.64; 95% confidence interval [CI], 2.47-2.81). However, in the active comparator design, opioid initiators (N = 86 635) had only 19% relatively greater risk of suicidal behavior compared with NSAID initiators (N = 255 096; hazard ratio = 1.19; 95% CI,: 1.11-1.28), corresponding to a weighted 5-year cumulative incidence of 2.2% (95% CI, 2.1-2.4) for opioid and 1.9% (95% CI, 1.9-2.0) for NSAID initiators. Most sensitivity analyses produced comparable results.

CONCLUSIONS

Opioid initiation may make only a small contribution to the elevated risk of suicidal behavior among young people receiving pharmacologic pain management. In weighing benefits and harms of opioid initiation, our results suggest that increased risk of suicidal behavior may not be a major concern.

Medications for chronic obstructive pulmonary disease: a historical non-interventional cohort study with validation against RCT results

BACKGROUND

Chronic obstructive pulmonary disease treatment is informed by randomised controlled trial results, but it is unclear if these findings apply to people excluded from these trials. We used data from the TORCH (TOwards a Revolution in COPD Health) randomised controlled trial to validate non-interventional methods for assessing the clinical effectiveness of chronic obstructive pulmonary disease treatment in the UK Clinical Practice Research Datalink, before applying these methods to the analysis of people who would have been excluded from TORCH.

OBJECTIVES

To validate the use of non-interventional Clinical Practice Research Datalink data and methods for estimating chronic obstructive pulmonary disease treatment effects against trial results, and, using validated methods, to determine treatment effects in people who would have been excluded from the TORCH trial.

DESIGN

A historical non-interventional cohort design, including validation against randomised controlled trial results.

SETTING

The UK Clinical Practice Research Datalink.

PARTICIPANTS

People aged ≥ 18 years with chronic obstructive pulmonary disease registered in Clinical Practice Research Datalink GOLD between January 2000 and January 2017. For objective 1, we prepared a cohort that was analogous to the TORCH trial cohort by applying TORCH trial inclusion/exclusion criteria followed by individual matching to TORCH trial participants. For objectives 2 and 3, we prepared cohorts that were analogous to the TORCH trial that, nevertheless, would not have been eligible for the TORCH trial because of age, asthma, comorbidity or mild disease.

INTERVENTIONS

The long-acting beta-2 agonist and inhaled corticosteroid combination product Seretide (GlaxoSmithKline plc) [i.e. fluticasone propionate plus salmeterol (FP-SAL)] compared with (1) no FP-SAL exposure or (2) exposure to salmeterol (i.e. the long-acting beta-2 agonist) only.

MAIN OUTCOME MEASURES

Exacerbations, mortality, pneumonia and time to treatment change.

RESULTS

For objective 1, the exacerbation rate ratio was comparable to that in the TORCH trial for FP-SAL compared with salmeterol (0.85, 95% confidence interval 0.74 to 0.97, vs. TORCH trial 0.88, 95% confidence interval 0.81 to 0.95), but not for FP-SAL compared with no FP-SAL (1.30, 95% confidence interval 1.19 to 1.42, vs. TORCH trial 0.75, 95% confidence interval 0.69 to 0.81). Active comparator results were also consistent with the TORCH trial for mortality (hazard ratio 0.93, 95% confidence interval 0.65 to 1.32, vs. TORCH trial hazard ratio 0.93, 95% confidence interval 0.77 to 1.13) and pneumonia (risk ratio 1.39, 95% confidence interval 1.04 to 1.87, vs. TORCH trial risk ratio 1.47, 95% confidence interval 1.25 to 1.73). For objectives 2 and 3, active comparator results were consistent with the TORCH trial for exacerbations, with the exception of people with milder chronic obstructive pulmonary disease, in whom we observed a stronger protective association (risk ratio 0.56, 95% confidence interval 0.46 to 0.70, vs. TORCH trial risk ratio 0.85, 95% confidence interval 0.74 to 0.97). For the analysis of mortality, we saw a lack of association with being prescribed FP-SAL (vs. being prescribed salmeterol), with the exception of those with prior asthma, for whom we observed an increase in mortality (hazard ratio 1.49, 95% confidence interval 1.21 to 1.85, vs. TORCH trial-analogous HR 0.93, 95% confidence interval 0.64 to 1.32).

CONCLUSIONS

Routinely collected electronic health record data can be used to successfully measure chronic obstructive pulmonary disease treatment effects when comparing two treatments, but not for comparisons between active treatment and no treatment. Analyses involving patients who would have been excluded from trials mostly suggests that treatment effects for FP-SAL are similar to trial effects, although further work is needed to characterise a small increased risk of death in those with concomitant asthma.

LIMITATIONS

Some of our analyses had small numbers.

FUTURE WORK

The differences in treatment effects that we found should be investigated further in other data sets. Currently recommended chronic obstructive pulmonary disease inhaled combination therapy (other than FP-SAL) should also be investigated using these methods.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 51. See the NIHR Journals Library website for further project information.

Association of Opioid Prescription Initiation During Adolescence and Young Adulthood With Subsequent Substance-Related Morbidity

IMPORTANCE

Concerns about adverse outcomes associated with opioid analgesic prescription have led to major guideline and policy changes. Substantial uncertainty remains, however, regarding the association between opioid prescription initiation and increased risk of subsequent substance-related morbidity.

OBJECTIVE

To examine the association of opioid initiation among adolescents and young adults with subsequent broadly defined substance-related morbidity.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study analyzed population-register data from January 1, 2007, to December 31, 2013, on Swedish individuals aged 13 to 29 years by January 1, 2013, who were naive to opioid prescription. To account for confounding, the analysis compared opioid prescription recipients with recipients of nonsteroidal anti-inflammatory drugs as an active comparator, compared opioid-recipient twins and other multiple birth individuals with their nonrecipient co-multiple birth offspring (co-twin control), examined dental prescription as a specific indication, and included individual, parental, and socioeconomic covariates. Data were analyzed from March 30, 2019, to January 22, 2020.

EXPOSURES

Opioid prescription initiation, defined as first dispensed opioid analgesic prescription.

MAIN OUTCOMES AND MEASURES

Substance-related morbidity, assessed as clinically diagnosed substance use disorder or overdose identified from inpatient or outpatient specialist records, substance use disorder or overdose cause of death, dispensed pharmacotherapy for alcohol use disorder, or conviction for substance-related crime.

RESULTS

Among the included cohort (n = 1 541 862; 793 933 male [51.5%]), 193 922 individuals initiated opioid therapy by December 31, 2013 (median age at initiation, 20.9 years [interquartile range, 18.2-23.6 years]). The active comparator design included 77 143 opioid recipients without preexisting substance-related morbidity and 229 461 nonsteroidal anti-inflammatory drug recipients. The adjusted cumulative incidence of substance-related morbidity within 5 years was 6.2% (95% CI, 5.9%-6.5%) for opioid recipients and 4.9% (95% CI, 4.8%-5.1%) for nonsteroidal anti-inflammatory drug recipients (hazard ratio, 1.29; 95% CI, 1.23-1.35). The co-twin control design produced comparable results (3013 opioid recipients and 3107 nonrecipients; adjusted hazard ratio, 1.43; 95% CI, 1.02-2.01), as did restriction to analgesics prescribed for dental indications and additional sensitivity analyses.

CONCLUSIONS AND RELEVANCE

Among adolescents and young adults analyzed in this study, initial opioid prescription receipt was associated with an approximately 30% to 40% relative increase in risk of subsequent substance-related morbidity in multiple designs that adjusted for confounding. These findings suggest that this increase may be smaller than previously estimated in some other studies.

Analytic and Data Sharing Options in Real-World Multidatabase Studies of Comparative Effectiveness and Safety of Medical Products

A wide range of analytic and data sharing options are available in nonexperimental multidatabase studies designed to assess the real-world benefits and risks of medical products. Researchers often consider six scientific domains when choosing among these options-study design, exposure type, outcome type, covariate summarization technique, covariate adjustment method, and data sharing approach. This article reviews available analytic and data sharing options and discusses key scientific and practical considerations when choosing among these options in multidatabase studies of comparative effectiveness and safety of medical products. The scientific considerations must be balanced against what the data-contributing sites are able or willing to share. While pooling of person-level data sets remains the most familiar and analytically flexible approach, newer analytic and data sharing approaches that share less granular summary-level information may be equally valid and preferred in some multidatabase studies, especially when sharing of person-level data is challenging or infeasible.

Non-Vitamin K Antagonist Oral Anticoagulants and Angioedema: A Cohort and Case-Crossover Study

INTRODUCTION

Patients taking non-vitamin K antagonist oral anticoagulants (NOACs) such as dabigatran, rivaroxaban, and apixaban have reported experiencing angioedema in randomized trials and routine care.

OBJECTIVE

The aim of this study was to quantify the association between NOACs and angioedema relative to warfarin among routinely treated patients with atrial fibrillation in a cohort study. We also compared warfarin users with non-users in a case-crossover study.

METHODS

We utilized a cohort design that drew eligible patients from the Truven Health MarketScan Commercial database, the Optum^© Clinformatics^® Data Mart, and Medicare. Within each database, we compared the 6-month relative rate of angioedema among new users of NOACs (dabigatran, rivaroxaban, apixaban) and new users of warfarin. We estimated hazard ratios (HRs) and 95% confidence intervals (CIs) after adjusting for confounders using propensity score stratification, and meta-analyzed the database-specific HRs using a random-effects model. We also estimated an odds ratio (OR) for the association between warfarin and angioedema using a case-crossover design, a self-controlled design that controls for time-invariant confounders.

RESULTS

In the cohort study, we observed 249 incident angioedema events among 267,681 NOAC initiators and 281,143 warfarin initiators across all databases. The meta-analyzed HR for angioedema comparing any NOAC versus warfarin was 0.98 (95% CI 0.76-1.27). In the case-crossover design, the OR for the association between warfarin and angioedema was 0.91 (95% CI 0.68-1.21) based on 431 cases.

CONCLUSIONS

Our estimates were inconsistent with substantial short-term relative increases in the rate of angioedema associated with oral anticoagulant therapy.

Antidepressants and the risk of traumatic brain injury in the elderly: differences between individual agents

Objective

To determine the association of individual antidepressants (ADs) with the risk of traumatic brain injury (TBI) in the elderly.

Patients and methods

We conducted a case-control study nested in a cohort of new users of ADs aged ≥65 years, identified in the German Pharmacoepidemiological Research Database during 2005-2014. Cases were patients first hospitalized for TBI. Up to 100 controls per case were selected using incidence density sampling. AD use was ascertained at the index date based on the supply of last dispensing (adding 150% of the defined daily doses [DDDs]; in sensitivity analysis, no additional DDDs were considered). We estimated adjusted ORs (aORs) and 95% CIs using conditional logistic regression.

Results

Among 701,309 cohort members, 16,750 cases were identified and matched to 1,673,320 controls (in both groups: 70.4% women; median age 80 years). Compared with remote users of the same AD, current users had an aOR (95% CI) of 1.87 (1.56-2.24) for duloxetine, 1.74 (1.41-2.15) for escitalopram, 1.70 (1.58-1.83) for citalopram, 1.66 (1.40-1.97) for sertraline, 1.64 (1.24-2.15) for fluoxetine and 1.57 (1.20-2.06) for paroxetine. The aOR was lower for amitriptyline (1.45; 1.32-1.58), trimipramine (1.17; 0.99-1.38) and opipramol (1.11; 0.99-1.25). Mirtazapine had an aOR of 1.03 (0.94-1.12). Sensitivity analysis confirmed the findings.

Conclusion

The large variability between individual ADs shows the importance of considering the safety of individual agents rather than focusing on class alone.

Real-World Data Analytics Fit for Regulatory Decision-Making

Healthcare database analyses (claims, electronic health records) have been identified by various regulatory initiatives, including the 21^st Century Cures Act and Prescription Drug User Fee Act ("PDUFA"), as useful supplements to randomized clinical trials to generate evidence on the effectiveness, harm, and value of medical products in routine care. Specific applications include accelerated drug approval pathways and secondary indications for approved medical products. Such real-world data ("RWD") analyses reflect how medical products impact health outside a highly controlled research environment. A constant stream of data from the routine operation of modern healthcare systems that can be analyzed in rapid cycles enables incremental evidence development for regulatory decision-making. Key evidentiary needs by regulators include 1) monitoring of medication performance in routine care, including the effectiveness, safety and value; 2) identifying new patient strata in which a drug may have added value or unacceptable harms; and 3) monitoring targeted utilization. Four broad requirements have been proposed to enable successful regulatory decision-making based on healthcare database analyses (collectively, "MVET"): Meaningful evidence that provides relevant and context-informed evidence sufficient for interpretation, drawing conclusions, and making decisions; valid evidence that meets scientific and technical quality standards to allow causal interpretations; expedited evidence that provides incremental evidence that is synchronized with the decision-making process; and transparent evidence that is audible, reproducible, robust, and ultimately trusted by decision-makers. Evidence generation systems that satisfy MVET requirements to a high degree will contribute to effective regulatory decision-making. Rapid-cycle analytics of healthcare databases is maturing at a time when regulatory overhaul increasingly demands such evidence. Governance, regulations, and data quality are catching up as the utility of this resource is demonstrated in multiple contexts.

Short term use of oral corticosteroids and related harms among adults in the United States: population based cohort study

Objective To determine the frequency of prescriptions for short term use of oral corticosteroids, and adverse events (sepsis, venous thromboembolism, fractures) associated with their use.Design Retrospective cohort study and self controlled case series.Setting Nationwide dataset of private insurance claims.Participants Adults aged 18 to 64 years who were continuously enrolled from 2012 to 2014.Main outcome measures Rates of short term use of oral corticosteroids defined as less than 30 days duration. Incidence rates of adverse events in corticosteroid users and non-users. Incidence rate ratios for adverse events within 30 day and 31-90 day risk periods after drug initiation.Results Of 1 548 945 adults, 327 452 (21.1%) received at least one outpatient prescription for short term use of oral corticosteroids over the three year period. Use was more frequent among older patients, women, and white adults, with significant regional variation (all P<0.001). The most common indications for use were upper respiratory tract infections, spinal conditions, and allergies. Prescriptions were provided by a diverse range of specialties. Within 30 days of drug initiation, there was an increase in rates of sepsis (incidence rate ratio 5.30, 95% confidence interval 3.80 to 7.41), venous thromboembolism (3.33, 2.78 to 3.99), and fracture (1.87, 1.69 to 2.07), which diminished over the subsequent 31-90 days. The increased risk persisted at prednisone equivalent doses of less than 20 mg/day (incidence rate ratio 4.02 for sepsis, 3.61 for venous thromboembolism, and 1.83 for fracture; all P<0.001).Conclusion One in five American adults in a commercially insured plan were given prescriptions for short term use of oral corticosteroids during a three year period, with an associated increased risk of adverse events.

Self-controlled designs in pharmacoepidemiology involving electronic healthcare databases: a systematic review

Background

Observational studies are widely used in pharmacoepidemiology. Several designs can be used, in particular self-controlled designs (case-crossover and self-controlled case series). These designs offer the advantage of controlling for time-invariant confounders, which may not be collected in electronic healthcare databases. They are particularly useful in pharmacoepidemiology involving healthcare database. To be valid, they require the presence of some characteristics (key validity assumptions), and in such situations, these designs should be preferred. We aimed at describing the appropriate use and reporting of the key validity assumptions in self-controlled design studies.

Methods

Articles published between January 2011 and December 2014, and describing a self-controlled study design involving electronic healthcare databases were retrieved. The appropriate use (fulfilment of key assumptions) was studied in terms of major (abrupt onset event, rare or recurrent event, and intermittent exposure) and minor assumptions (those for which the design can be adapted).

Results

Among the 107 articles describing a self-controlled design, 35/53 (66%) case-crossover studies, and 48/55 (87%) self-controlled case series fulfilled the major validity assumptions for use of the design; 4/35 and 14/48 respectively did not fulfill the minor assumptions. Overall, 31/53 (58%) case-crossover studies and 34/55 (62%) self-controlled case series fulfilled both major and minor assumptions. The reporting of the methodology or the results was appropriate, except for power calculation.

Conclusions

Self-controlled designs were not appropriately used in34% and 13% of the articles we reviewed that described a case-crossover or a self-controlled case series design, respectively. We encourage better use of these designs in situations in which major validity assumptions are fulfilled (i.e., for which they are recommended), accounting for situations for which the design can be adapted.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0278-0) contains supplementary material, which is available to authorized users.

Successful Comparison of US Food and Drug Administration Sentinel Analysis Tools to Traditional Approaches in Quantifying a Known Drug-Adverse Event Association

The US Food and Drug Administration's Sentinel system has developed the capability to conduct active safety surveillance of marketed medical products in a large network of electronic healthcare databases. We assessed the extent to which the newly developed, semiautomated Sentinel Propensity Score Matching (PSM) tool could produce the same results as a customized protocol-driven assessment, which found an adjusted hazard ratio (HR) of 3.04 (95% confidence interval [CI], 2.81-3.27) comparing angioedema in patients initiating angiotensin-converting enzyme (ACE) inhibitors vs. beta-blockers. Using data from 13 Data Partners between 1 January 2008, and 30 September 2013, the PSM tool identified 2,211,215 eligible ACE inhibitor and 1,673,682 eligible beta-blocker initiators. The tool produced an HR of 3.14 (95% CI, 2.86-3.44). This comparison provides initial evidence that Sentinel analytic tools can produce findings similar to those produced by a highly customized protocol-driven assessment.

Comparison of Benefit-Risk Assessment Methods for Prospective Monitoring of Newly Marketed Drugs: A Simulation Study

OBJECTIVES

To compare benefit-risk assessment (BRA) methods for determining whether and when sufficient evidence exists to indicate that one drug is favorable over another in prospective monitoring.

METHODS

We simulated prospective monitoring of a new drug (A) versus an alternative drug (B) with respect to two beneficial and three harmful outcomes. We generated data for 1000 iterations of six scenarios and applied four BRA metrics: number needed to treat and number needed to harm (NNT|NNH), incremental net benefit (INB) with maximum acceptable risk, INB with relative-value-adjusted life-years, and INB with quality-adjusted life-years. We determined the proportion of iterations in which the 99% confidence interval for each metric included and excluded the null and we calculated mean time to alerting.

RESULTS

With no true difference in any outcome between drugs A and B, the proportion of iterations including the null was lowest for INB with relative-value-adjusted life-years (64%) and highest for INB with quality-adjusted life-years (76%). When drug A was more effective and the drugs were equally safe, all metrics indicated net favorability of A in more than 70% of the iterations. When drug A was safer than drug B, NNT|NNH had the highest proportion of iterations indicating net favorability of drug A (65%). Mean time to alerting was similar among methods across the six scenarios.

CONCLUSIONS

BRA metrics can be useful for identifying net favorability when applied to prospective monitoring of a new drug versus an alternative drug. INB-based approaches similarly outperform unweighted NNT|NNH approaches. Time to alerting was similar across approaches.

Prospective Benefit-Risk Monitoring of New Drugs for Rapid Assessment of Net Favorability in Electronic Health Care Data

BACKGROUND

Benefit-risk assessment (BRA) methods can combine measures of benefits and risks into a single value.

OBJECTIVES

To examine BRA metrics for prospective monitoring of new drugs in electronic health care data.

METHODS

Using two electronic health care databases, we emulated prospective monitoring of three drugs (rofecoxib vs. nonselective nonsteroidal anti-inflammatory drugs, prasugrel vs. clopidogrel, and denosumab vs. bisphosphonates) using a sequential propensity score-matched cohort design. We applied four BRA metrics: number needed to treat and number needed to harm; incremental net benefit (INB) with maximum acceptable risk; INB with relative-value-adjusted life-years; and INB with quality-adjusted life-years (QALYs). We determined whether and when the bootstrapped 99% confidence interval (CI) for each metric excluded zero, indicating net favorability of one drug over the other.

RESULTS

For rofecoxib, all four metrics yielded a negative value, suggesting net favorability of nonselective nonsteroidal anti-inflammatory drugs over rofecoxib, and the 99% CI for all but the number needed to treat and number needed to harm excluded the null during follow-up. For prasugrel, only the 99% CI for INB-QALY excluded the null, but trends in values over time were similar across the four metrics, suggesting overall net favorability of prasugrel versus clopidogrel. The 99% CI for INB-relative-value-adjusted life-years and INB-QALY excluded the null in the denosumab example, suggesting net favorability of denosumab over bisphosphonates.

CONCLUSIONS

Prospective benefit-risk monitoring can be used to determine net favorability of a new drug in electronic health care data. In three examples, existing BRA metrics produced qualitatively similar results but differed with respect to alert generation.

Structured assessment for prospective identification of safety signals in electronic medical records: evaluation in the health improvement network

Background

Pharmacovigilance signal detection largely relies on individual case reports, but longitudinal health data are being explored as complementary information sources. Research to date has focused on the ability of epidemiological methods to distinguish established adverse drug reactions (ADRs) from unrelated adverse events.

Objective

The aim of this study was to evaluate a process for structured clinical and epidemiological assessment of temporally associated drugs and medical events in electronic medical records.

Methods

Pairs of drugs and medical events were selected for review on the basis of their temporal association according to a calibrated self-controlled cohort analysis in The Health Improvement Network. Six assessors trained in pharmacovigilance and/or epidemiology evaluated seven drugs each, with up to 20 medical events per drug. A pre-specified questionnaire considered aspects related to the nature of the temporal pattern, demographic features of the cohort, concomitant medicines, earlier signs and symptoms, and possible confounding by underlying disease. This informed a classification of drug–event pairs as known ADRs, meriting further evaluation, or dismissed.

Results

The number of temporally associated medical events per drug ranged from 11 to 307 (median 50) for the 42 selected drugs. Out of the 509 relevant drug–event combinations subjected to the assessment, 127 (25 %) were classified as known ADRs. Ninety-one (24 %) of the remaining pairs were classified as potential signals meriting further evaluation and 291 (76 %) were dismissed. Suggestive temporal patterns and lack of clear alternative explanations were the most common reasons that drug–event pairs were classified as meriting further evaluation. Earlier signs and symptoms and confounding by the underlying disease were the most common reasons that drug–event pairs were dismissed.

Conclusions

Exploratory analysis of electronic medical records can detect important potential safety signals. However, effective signal detection requires that statistical signal detection be combined with clinical and epidemiological review to achieve an acceptable false positive rate.

A vaccine study design selection framework for the postlicensure rapid immunization safety monitoring program

The Postlicensure Rapid Immunization Safety Monitoring Program, the vaccination safety monitoring component of the US Food and Drug Administration's Mini-Sentinel project, is currently the largest cohort in the US general population for vaccine safety surveillance. We developed a study design selection framework to provide a roadmap and description of methods that may be utilized to evaluate potential associations between vaccines and health outcomes of interest in the Postlicensure Rapid Immunization Safety Monitoring Program and other systems using administrative data. The strengths and weaknesses of designs for vaccine safety monitoring, including the cohort design, the case-centered design, the risk interval design, the case-control design, the self-controlled risk interval design, the self-controlled case series method, and the case-crossover design, are described and summarized in tabular form. A structured decision table is provided to aid in planning of future vaccine safety monitoring activities, and the data components comprising the structured decision table are delineated. The study design selection framework provides a starting point for planning vaccine safety evaluations using claims-based data sources.

Empirical performance of a new user cohort method: lessons for developing a risk identification and analysis system

BACKGROUND

Observational healthcare data offer the potential to enable identification of risks of medical products, but appropriate methodology has not yet been defined. The new user cohort method, which compares the post-exposure rate among the target drug to a referent comparator group, is the prevailing approach for many pharmacoepidemiology evaluations and has been proposed as a promising approach for risk identification but its performance in this context has not been fully assessed.

OBJECTIVES

To evaluate the performance of the new user cohort method as a tool for risk identification in observational healthcare data.

RESEARCH DESIGN

The method was applied to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record) and in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively.

MEASURES

Method performance was evaluated through Area Under ROC Curve (AUC), bias, and coverage probability.

RESULTS

The new user cohort method achieved modest predictive accuracy across the outcomes and databases under study, with the top-performing analysis near AUC >0.70 in most scenarios. The performance of the method was particularly sensitive to the choice of comparator population. For almost all drug-outcome pairs there was a large difference, either positive or negative, between the true effect size and the estimate produced by the method, although this error was near zero on average. Simulation studies showed that in the majority of cases, the true effect estimate was not within the 95 % confidence interval produced by the method.

CONCLUSION

The new user cohort method can contribute useful information toward a risk identification system, but should not be considered definitive evidence given the degree of error observed within the effect estimates. Careful consideration of the comparator selection and appropriate calibration of the effect estimates is required in order to properly interpret study findings.

Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies

BACKGROUND

Clinical studies that use observational databases, such as administrative claims and electronic health records, to evaluate the effects of medical products have become commonplace. These studies begin by selecting a particular study design, such as a case control, cohort, or self-controlled design, and different authors can and do choose different designs for the same clinical question. Furthermore, published papers invariably report the study design but do not discuss the rationale for the specific choice. Studies of the same clinical question with different designs, however, can generate different results, sometimes with strikingly different implications. Even within a specific study design, authors make many different analytic choices and these too can profoundly impact results. In this paper, we systematically study heterogeneity due to the type of study design and due to analytic choices within study design.

METHODS AND FINDINGS

We conducted our analysis in 10 observational healthcare databases but mostly present our results in the context of the GE Centricity EMR database, an electronic health record database containing data for 11.2 million lives. We considered the impact of three different study design choices on estimates of associations between bisphosphonates and four particular health outcomes for which there is no evidence of an association. We show that applying alternative study designs can yield discrepant results, in terms of direction and significance of association. We also highlight that while traditional univariate sensitivity analysis may not show substantial variation, systematic assessment of all analytical choices within a study design can yield inconsistent results ranging from statistically significant decreased risk to statistically significant increased risk. Our findings show that clinical studies using observational databases can be sensitive both to study design choices and to specific analytic choices within study design.

CONCLUSION

More attention is needed to consider how design choices may be impacting results and, when possible, investigators should examine a wide array of possible choices to confirm that significant findings are consistently identified.

A modular, prospective, semi-automated drug safety monitoring system for use in a distributed data environment

PURPOSE

The aim of this study was to develop and test a semi-automated process for conducting routine active safety monitoring for new drugs in a network of electronic healthcare databases.

METHODS

We built a modular program that semi-automatically performs cohort identification, confounding adjustment, diagnostic checks, aggregation and effect estimation across multiple databases, and application of a sequential alerting algorithm. During beta-testing, we applied the system to five databases to evaluate nine examples emulating prospective monitoring with retrospective data (five pairs for which we expected signals, two negative controls, and two examples for which it was uncertain whether a signal would be expected): cerivastatin versus atorvastatin and rhabdomyolysis; paroxetine versus tricyclic antidepressants and gastrointestinal bleed; lisinopril versus angiotensin receptor blockers and angioedema; ciprofloxacin versus macrolide antibiotics and Achilles tendon rupture; rofecoxib versus non-selective non-steroidal anti-inflammatory drugs (ns-NSAIDs) and myocardial infarction; telithromycin versus azithromycin and hepatotoxicity; rosuvastatin versus atorvastatin and diabetes and rhabdomyolysis; and celecoxib versus ns-NSAIDs and myocardial infarction.

RESULTS

We describe the program, the necessary inputs, and the assumed data environment. In beta-testing, the system generated four alerts, all among positive control examples (i.e., lisinopril and angioedema; rofecoxib and myocardial infarction; ciprofloxacin and tendon rupture; and cerivastatin and rhabdomyolysis). Sequential effect estimates for each example were consistent in direction and magnitude with existing literature.

CONCLUSIONS

Beta-testing across nine drug-outcome examples demonstrated the feasibility of the proposed semi-automated prospective monitoring approach. In retrospective assessments, the system identified an increased risk of myocardial infarction with rofecoxib and an increased risk of rhabdomyolysis with cerivastatin years before these drugs were withdrawn from the market.

Statin use and infections in Veterans with cirrhosis

BACKGROUND

Evidence about the beneficial effects of statins on reducing infections is accumulating. Identifying ways to reduce infection risk in patients with cirrhosis is important because of increased mortality risk and costs associated with infections.

AIM

To estimate the extent to which statin use prolongs time to infection among patients with cirrhosis.

METHODS

We identified Veterans with cirrhosis, but without decompensation (n = 19 379) using US Veterans Health Administration data from 2001 to 2009. New users of statins were identified and propensity matched to non-users and users of other cholesterol-lowering medications (1:1 matching). The cohort was followed up for hospitalisations with infections. Cox regression models with time-varying exposures provided estimates of adjusted hazard ratios (HR) and 95% confidence intervals (CI).

RESULTS

New statin use was present among 13% of VA patients with cirrhosis without decompensation. Overall, 12.4% of patients developed a serious infection, and 0.1% of patients died. In the propensity-matched sample, statin users experienced hospitalisations with infections at a rate 0.67 less than non-users (95% Confidence Interval: 0.47-0.95).

CONCLUSIONS

Infections are a major concern among cirrhotic patients and have the potential to seriously impact both life expectancy and quality of life. Statin use may potentially reduce the risk of infections among patients with cirrhosis.

Treatment dynamics of newly marketed drugs and implications for comparative effectiveness research

OBJECTIVES

Clinicians and payers require rapid comparative effectiveness (CE) evidence generation to inform decisions for new drugs. We empirically assessed treatment dynamics of newly marked drugs and their implications for conducting CE research.

METHODS

We used claims data to evaluate five drug-outcome pairs: 1) raloxifene (vs. alendronate) and fracture; 2) risedronate (vs. alendronate) and fracture; 3) simvastatin plus ezetimibe fixed-dose combination (simvastatin + ezetimibe) (vs. simvastatin alone) and cardiovascular events; 4) rofecoxib (vs. nonselective nonsteroidal anti-inflammatory drugs [ns-NSAIDs]) and myocardial infarction; and 5) rofecoxib (vs. ns-NSAIDS) and gastrointestinal bleed. We examined utilization dynamics in the early marketing period, including evolving utilization patterns, outcome risk among those treated with new versus established drugs, and prior treatment patterns that may indicate treatment resistance or intolerance. We addressed these challenges by replicating active CE monitoring with sequential matched cohort analysis.

RESULTS

Patients initiating new drugs were more likely to have used other drugs for the same indication in the past, but the majority of patients in all new drug cohorts were treatment naive (82.0% overall). Patients initiating rofecoxib had higher predicted baseline risk of gastrointestinal bleed than did patients initiating ns-NSAIDs. Patients initiating risedronate and alendronate had similar predicted baseline risks of fracture, while those initiating raloxifene and simvastatin + ezetimibe had lower risks of outcomes of interest relative to their comparators. Prospective monitoring yielded results consistent with expectation for each example.

CONCLUSIONS

Many challenges to assessing the CE of new drugs are borne out in empirical data. Attention to these challenges can yield valid CE results.

Novel data-mining methodologies for adverse drug event discovery and analysis

An important goal of the health system is to identify new adverse drug events (ADEs) in the postapproval period. Datamining methods that can transform data into meaningful knowledge to inform patient safety have proven essential for this purpose. New opportunities have emerged to harness data sources that have not been used within the traditional framework. This article provides an overview of recent methodological innovations and data sources used to support ADE discovery and analysis.

Active safety monitoring of new medical products using electronic healthcare data: selecting alerting rules

BACKGROUND

Active medical-product-safety surveillance systems are being developed to monitor many products and outcomes simultaneously in routinely collected longitudinal electronic healthcare data. These systems will rely on algorithms to generate alerts about potential safety concerns.

METHODS

We compared the performance of 5 classes of algorithms in simulated data using a sequential matched-cohort framework, and applied the results to 2 electronic healthcare databases to replicate monitoring of cerivastatin-induced rhabdomyolysis. We generated 600,000 simulated scenarios with varying expected event frequency in the unexposed, alerting threshold, and outcome risk in the exposed, and compared the alerting algorithms in each scenario type using an event-based performance metric.

RESULTS

We observed substantial variation in algorithm performance across the groups of scenarios. Relative performance varied by the event frequency and by user-defined preferences for sensitivity versus specificity. Type I error-based statistical testing procedures achieved higher event-based performance than other approaches in scenarios with few events, whereas statistical process control and disproportionality measures performed relatively better with frequent events. In the empirical data, we observed 6 cases of rhabdomyolysis among 4294 person-years of follow-up, with all events occurring among cerivastatin-treated patients. All selected algorithms generated alerts before the drug was withdrawn from the market.

CONCLUSIONS

For active medical-product-safety monitoring in a sequential matched cohort framework, no single algorithm performed best in all scenarios. Alerting algorithm selection should be tailored to particular features of a product-outcome pair, including the expected event frequencies and trade-offs between false-positive and false-negative alerting.

Active safety monitoring of newly marketed medications in a distributed data network: application of a semi-automated monitoring system

We developed a semi-automated active monitoring system that uses sequential matched-cohort analyses to assess drug safety across a distributed network of longitudinal electronic health-care data. In a retrospective analysis, we show that the system would have identified cerivastatin-induced rhabdomyolysis. In this study, we evaluated whether the system would generate alerts for three drug-outcome pairs: rosuvastatin and rhabdomyolysis (known null association), rosuvastatin and diabetes mellitus, and telithromycin and hepatotoxicity (two examples for which alerting would be questionable). Over >5 years of monitoring, rate differences (RDs) in comparisons of rosuvastatin with atorvastatin were -0.1 cases of rhabdomyolysis per 1,000 person-years (95% confidence interval (CI): -0.4, 0.1) and -2.2 diabetes cases per 1,000 person-years (95% CI: -6.0, 1.6). The RD for hepatotoxicity comparing telithromycin with azithromycin was 0.3 cases per 1,000 person-years (95% CI: -0.5, 1.0). In a setting in which false positivity is a major concern, the system did not generate alerts for the three drug-outcome pairs.

The U.S. Food and Drug Administration's Mini-Sentinel program: status and direction

The Mini-Sentinel is a pilot program that is developing methods, tools, resources, policies, and procedures to facilitate the use of routinely collected electronic healthcare data to perform active surveillance of the safety of marketed medical products, including drugs, biologics, and medical devices. The U.S. Food and Drug Administration (FDA) initiated the program in 2009 as part of its Sentinel Initiative, in response to a Congressional mandate in the FDA Amendments Act of 2007. After two years, Mini-Sentinel includes 31 academic and private organizations. It has developed policies, procedures, and technical specifications for developing and operating a secure distributed data system comprised of separate data sets that conform to a common data model covering enrollment, demographics, encounters, diagnoses, procedures, and ambulatory dispensing of prescription drugs. The distributed data sets currently include administrative and claims data from 2000 to 2011 for over 300 million person-years, 2.4 billion encounters, 38 million inpatient hospitalizations, and 2.9 billion dispensings. Selected laboratory results and vital signs data recorded after 2005 are also available. There is an active data quality assessment and characterization program, and eligibility for medical care and pharmacy benefits is known. Systematic reviews of the literature have assessed the ability of administrative data to identify health outcomes of interest, and procedures have been developed and tested to obtain, abstract, and adjudicate full-text medical records to validate coded diagnoses. Mini-Sentinel has also created a taxonomy of study designs and analytical approaches for many commonly occurring situations, and it is developing new statistical and epidemiologic methods to address certain gaps in analytic capabilities. Assessments are performed by distributing computer programs that are executed locally by each data partner. The system is in active use by FDA, with the majority of assessments performed using customizable, reusable queries (programs). Prospective and retrospective assessments that use customized protocols are conducted as well. To date, several hundred unique programs have been distributed and executed. Current activities include active surveillance of several drugs and vaccines, expansion of the population, enhancement of the common data model to include additional types of data from electronic health records and registries, development of new methodologic capabilities, and assessment of methods to identify and validate additional health outcomes of interest.