Design of a national distributed health data network

A distributed health data network is a system that allows secure remote analysis of separate data sets, each comprising a different medical organization's or health plan's records. Distributed health data networks are currently being planned that could cover millions of people, permitting studies of comparative clinical effectiveness, best practices, diffusion of medical technologies, and quality of care. These networks could also support assessment of medical product safety and other public health needs. Distributed network technologies allow data holders to control all uses of their data, which overcomes many practical obstacles related to confidentiality, regulation, and proprietary interests. Some of the challenges and potential methods of operation of a multipurpose, multi-institutional distributed health data network are described.

Development and validation of an automated HIV prediction algorithm to identify candidates for pre-exposure prophylaxis: a modelling study

BACKGROUND

HIV pre-exposure prophylaxis (PrEP) is effective but underused, in part because clinicians do not have the tools to identify PrEP candidates. We developed and validated an automated prediction algorithm that uses electronic health record (EHR) data to identify individuals at increased risk for HIV acquisition.

METHODS

We used machine learning algorithms to predict incident HIV infections with 180 potential predictors of HIV risk drawn from EHR data from 2007-15 at Atrius Health, an ambulatory group practice in Massachusetts, USA. We included EHRs of all patients aged 15 years or older with at least one clinical encounter during 2007-15. We used ten-fold cross-validated area under the receiver operating characteristic curve (cv-AUC) with 95% CIs to assess the model's performance at identifying individuals with incident HIV and patients independently prescribed PrEP by clinicians. The best-performing model was validated prospectively with 2016 data from Atrius Health and externally with 2011-16 data from Fenway Health, a community health centre specialising in sexual health care in Boston (MA, USA). We calculated HIV risk scores (ie, probability of an incident HIV diagnosis) for every HIV-uninfected patient not on PrEP during 2007-15 at Atrius Health and assessed the distribution of scores for thresholds to determine possible candidates for PrEP in the three study cohorts.

FINDINGS

We included 1 155 966 Atrius Health patients from 2007-15 (150 [<0·1%] patients with incident HIV) in our development cohort, 537 257 Atrius Health patients in 2016 (16 [<0·1%] with incident HIV) in our prospective validation cohort, and 33 404 Fenway Health patients from 2011-16 (423 [1·3%] with incident HIV) in our external validation cohort. The best-performing algorithm was obtained with least absolute shrinkage and selection operator (LASSO) and had a cv-AUC of 0·86 (95% CI 0·82-0·90) for identification of incident HIV infections in the development cohort, 0·91 (0·81-1·00) on prospective validation, and 0·77 (0·74-0·79) on external validation. The LASSO model successfully identified patients independently prescribed PrEP by clinicians at Atrius Health in 2016 (cv-AUC 0·93, 95% CI 0·90-0·96) or Fenway Health (0·79, 0·78-0·80). HIV risk scores increased steeply at the 98th percentile. Using this score as a threshold, we prospectively identified 9515 (1·8%) of 536 384 patients at Atrius Health in 2016 and 4385 (15·3%) of 28 702 Fenway Health patients as potential PrEP candidates.

INTERPRETATION

Automated algorithms can efficiently identify patients at increased risk for HIV acquisition. Integrating these models into EHRs to alert providers about patients who might benefit from PrEP could improve prescribing and prevent new HIV infections.

FUNDING

Harvard University Center for AIDS Research, Providence/Boston Center for AIDS Research, Rhode Island IDeA-CTR, the National Institute of Mental Health, and the US Centers for Disease Control and Prevention.

Broadening the reach of the FDA Sentinel system: A roadmap for integrating electronic health record data in a causal analysis framework

The Sentinel System is a major component of the United States Food and Drug Administration's (FDA) approach to active medical product safety surveillance. While Sentinel has historically relied on large quantities of health insurance claims data, leveraging longitudinal electronic health records (EHRs) that contain more detailed clinical information, as structured and unstructured features, may address some of the current gaps in capabilities. We identify key challenges when using EHR data to investigate medical product safety in a scalable and accelerated way, outline potential solutions, and describe the Sentinel Innovation Center's initiatives to put solutions into practice by expanding and strengthening the existing system with a query-ready, large-scale data infrastructure of linked EHR and claims data. We describe our initiatives in four strategic priority areas: (1) data infrastructure, (2) feature engineering, (3) causal inference, and (4) detection analytics, with the goal of incorporating emerging data science innovations to maximize the utility of EHR data for medical product safety surveillance.

Assessment of Quadrivalent Human Papillomavirus Vaccine Safety Using the Self-Controlled Tree-Temporal Scan Statistic Signal-Detection Method in the Sentinel System

The self-controlled tree-temporal scan statistic-a new signal-detection method-can evaluate whether any of a wide variety of health outcomes are temporally associated with receipt of a specific vaccine, while adjusting for multiple testing. Neither health outcomes nor postvaccination potential periods of increased risk need be prespecified. Using US medical claims data in the Food and Drug Administration's Sentinel system, we employed the method to evaluate adverse events occurring after receipt of quadrivalent human papillomavirus vaccine (4vHPV). Incident outcomes recorded in emergency department or inpatient settings within 56 days after first doses of 4vHPV received by 9- through 26.9-year-olds in 2006-2014 were identified using International Classification of Diseases, Ninth Revision, diagnosis codes and analyzed by pairing the new method with a standard hierarchical classification of diagnoses. On scanning diagnoses of 1.9 million 4vHPV recipients, 2 statistically significant categories of adverse events were found: cellulitis on days 2-3 after vaccination and "other complications of surgical and medical procedures" on days 1-3 after vaccination. Cellulitis is a known adverse event. Clinically informed investigation of electronic claims records of the patients with "other complications" did not suggest any previously unknown vaccine safety problem. Considering that thousands of potential short-term adverse events and hundreds of potential risk intervals were evaluated, these findings add significantly to the growing safety record of 4vHPV.

Use of Sodium-Glucose Cotransporter 2 Inhibitors in Patients With Type 1 Diabetes and Rates of Diabetic Ketoacidosis

OBJECTIVE

To estimate real-world off-label use of sodium-glucose cotransporter 2 (SGLT2) inhibitors in patients with type 1 diabetes, estimate rates of diabetic ketoacidosis (DKA), and compare them with DKA rates observed in sotagliflozin clinical trials.

RESEARCH DESIGN AND METHODS

We identified initiators of SGLT2 inhibitors in the Sentinel System from March 2013 to June 2018, determined the prevalence of type 1 diabetes using a narrow and a broad definition, and measured rates of DKA using administrative claims data. Standardized incidence ratios (SIRs) were calculated using age- and sex-specific follow-up time in Sentinel and age- and sex-specific DKA rates from sotagliflozin trials 309, 310, and 312.

RESULTS

Among 475,527 initiators of SGLT2 inhibitors, 0.50% and 0.92% met narrow and broad criteria for type 1 diabetes, respectively. Rates of DKA in the narrow and broad groups were 7.3/100 person-years and 4.5/100 person-years, respectively. Among patients who met narrow criteria for type 1 diabetes, rates of DKA were highest for patients aged 25-44 years, especially females aged 25-44 years (19.7/100 person-years). More DKA events were observed during off-label use of SGLT2 inhibitors in Sentinel than would be expected based on sotagliflozin clinical trials (SIR = 1.83; 95% CI 1.45-2.28).

CONCLUSIONS

Real-world off-label use of SGLT2 inhibitors among patients with type 1 diabetes accounted for a small proportion of overall SGLT2 inhibitor use. However, the risk for DKA during off-label use was notable, especially among young, female patients. Although real-word rates of DKA exceeded the expectation based on clinical trials, results should be interpreted with caution due to differences in study methods, patient samples, and study drugs.

Prospective influenza vaccine safety surveillance using fresh data in the Sentinel System

Purpose

To develop the infrastructure to conduct timely active surveillance for safety of influenza vaccines and other medical countermeasures in the Sentinel System (formerly the Mini‐Sentinel Pilot), a Food and Drug Administration‐sponsored national surveillance system that typically relies on data that are mature, settled, and updated quarterly.

Methods

Three Data Partners provided their earliest available (“fresh”) cumulative claims data on influenza vaccination and health outcomes 3–4 times on a staggered basis during the 2013–2014 influenza season, collectively producing 10 data updates. We monitored anaphylaxis in the entire population using a cohort design and seizures in children ≤4 years of age using both a self‐controlled risk interval design (primary) and a cohort design (secondary). After each data update, we conducted sequential analysis for inactivated (IIV) and live (LAIV) influenza vaccines using the Maximized Sequential Probability Ratio Test, adjusting for data‐lag.

Results

Most of the 10 sequential analyses were conducted within 6 weeks of the last care‐date in the cumulative dataset. A total of 6 682 336 doses of IIV and 782 125 doses of LAIV were captured. The primary analyses did not identify any statistical signals following IIV or LAIV. In secondary analysis, the risk of seizures was higher following concomitant IIV and PCV13 than historically after IIV in 6‐ to 23‐month‐olds (relative risk = 2.7), which requires further investigation.

Conclusions

The Sentinel System can implement a sequential analysis system that uses fresh data for medical product safety surveillance. Active surveillance using sequential analysis of fresh data holds promise for detecting clinically significant health risks early. Limitations of employing fresh data for surveillance include cost and the need for careful scrutiny of signals. © 2015 The Authors. Pharmacoepidemiology and Drug Safety Published by John Wiley & Sons Ltd.

A COVID-19-ready public health surveillance system: The Food and Drug Administration's Sentinel System

The US Food and Drug Administration's Sentinel System was established in 2009 to use routinely collected electronic health data for improving the national capability to assess post‐market medical product safety. Over more than a decade, Sentinel has become an integral part of FDA's surveillance capabilities and has been used to conduct analyses that have contributed to regulatory decisions. FDA's role in the COVID‐19 pandemic response has necessitated an expansion and enhancement of Sentinel. Here we describe how the Sentinel System has supported FDA's response to the COVID‐19 pandemic. We highlight new capabilities developed, key data generated to date, and lessons learned, particularly with respect to working with inpatient electronic health record data.

Early in the pandemic, Sentinel developed a multi‐pronged approach to support FDA's anticipated data and analytic needs. It incorporated new data sources, created a rapidly refreshed database, developed protocols to assess the natural history of COVID‐19, validated a diagnosis‐code based algorithm for identifying patients with COVID‐19 in administrative claims data, and coordinated with other national and international initiatives.

Sentinel is poised to answer important questions about the natural history of COVID‐19 and is positioned to use this information to study the use, safety, and potentially the effectiveness of medical products used for COVID‐19 prevention and treatment.

Using and improving distributed data networks to generate actionable evidence: the case of real-world outcomes in the Food and Drug Administration's Sentinel system

The US Food and Drug Administration (FDA) Sentinel System uses a distributed data network, a common data model, curated real-world data, and distributed analytic tools to generate evidence for FDA decision-making. Sentinel system needs include analytic flexibility, transparency, and reproducibility while protecting patient privacy. Based on over a decade of experience, a critical system limitation is the inability to identify enough medical conditions of interest in observational data to a satisfactory level of accuracy. Improving the system's ability to use computable phenotypes will require an "all of the above" approach that improves use of electronic health data while incorporating the growing array of complementary electronic health record data sources. FDA recently funded a Sentinel System Innovation Center and a Community Building and Outreach Center that will provide a platform for collaboration across disciplines to promote better use of real-world data for decision-making.

The US Food and Drug Administration Sentinel System: a national resource for a learning health system

The US Food and Drug Administration (FDA) created the Sentinel System in response to a requirement in the FDA Amendments Act of 2007 that the agency establish a system for monitoring risks associated with drug and biologic products using data from disparate sources. The Sentinel System has completed hundreds of analyses, including many that have directly informed regulatory decisions. The Sentinel System also was designed to support a national infrastructure for a learning health system. Sentinel governance and guiding principles were designed to facilitate Sentinel's role as a national resource. The Sentinel System infrastructure now supports multiple non-FDA projects for stakeholders ranging from regulated industry to other federal agencies, international regulators, and academics. The Sentinel System is a working example of a learning health system that is expanding with the potential to create a global learning health system that can support medical product safety assessments and other research.

Using electronic health records to identify candidates for human immunodeficiency virus pre-exposure prophylaxis: An application of super learning to risk prediction when the outcome is rare

Human immunodeficiency virus (HIV) pre-exposure prophylaxis (PrEP) protects high risk patients from becoming infected with HIV. Clinicians need help to identify candidates for PrEP based on information routinely collected in electronic health records (EHRs). The greatest statistical challenge in developing a risk prediction model is that acquisition is extremely rare.

METHODS

Data consisted of 180 covariates (demographic, diagnoses, treatments, prescriptions) extracted from records on 399 385 patient (150 cases) seen at Atrius Health (2007-2015), a clinical network in Massachusetts. Super learner is an ensemble machine learning algorithm that uses k-fold cross validation to evaluate and combine predictions from a collection of algorithms. We trained 42 variants of sophisticated algorithms, using different sampling schemes that more evenly balanced the ratio of cases to controls. We compared super learner's cross validated area under the receiver operating curve (cv-AUC) with that of each individual algorithm.

RESULTS

The least absolute shrinkage and selection operator (lasso) using a 1:20 class ratio outperformed the super learner (cv-AUC = 0.86 vs 0.84). A traditional logistic regression model restricted to 23 clinician-selected main terms was slightly inferior (cv-AUC = 0.81).

CONCLUSION

Machine learning was successful at developing a model to predict 1-year risk of acquiring HIV based on a physician-curated set of predictors extracted from EHRs.

Validity of ICD-10-CM diagnoses to identify hospitalizations for serious infections among patients treated with biologic therapies

PURPOSE

Identifying hospitalizations for serious infections among patients dispensed biologic therapies within healthcare databases is important for post-marketing surveillance of these drugs. We determined the positive predictive value (PPV) of an ICD-10-CM-based diagnostic coding algorithm to identify hospitalization for serious infection among patients dispensed biologic therapy within the FDA's Sentinel Distributed Database.

METHODS

We identified health plan members who met the following algorithm criteria: (1) hospital ICD-10-CM discharge diagnosis of serious infection between July 1, 2016 and August 31, 2018; (2) either outpatient/emergency department infection diagnosis or outpatient antimicrobial treatment within 7 days prior to hospitalization; (3) inflammatory bowel disease, psoriasis, or rheumatological diagnosis within 1 year prior to hospitalization, and (4) were dispensed outpatient biologic therapy within 90 days prior to admission. Medical records were reviewed by infectious disease clinicians to adjudicate hospitalizations for serious infection. The PPV (95% confidence interval [CI]) for confirmed events was determined after further weighting by the prevalence of the type of serious infection in the database.

RESULTS

Among 223 selected health plan members who met the algorithm, 209 (93.7% [95% CI, 90.1%-96.9%]) were confirmed to have a hospitalization for serious infection. After weighting by the prevalence of the type of serious infection, the PPV of the ICD-10-CM algorithm identifying a hospitalization for serious infection was 80.2% (95% CI, 75.3%-84.7%).

CONCLUSIONS

The ICD-10-CM-based algorithm for hospitalization for serious infection among patients dispensed biologic therapies within the Sentinel Distributed Database had 80% PPV for confirmed events and could be considered for use within pharmacoepidemiologic studies.

Using the Self-Controlled Tree-Temporal Scan Statistic to Assess the Safety of Live Attenuated Herpes Zoster Vaccine

The self-controlled tree-temporal scan statistic allows detection of potential vaccine- or drug-associated adverse events without prespecifying the specific events or postexposure risk intervals of concern. It thus opens a promising new avenue for safety studies. The method has been successfully used to evaluate the safety of 2 vaccines for adolescents and young adults, but its suitability to study vaccines for older adults had not been established. The present study applied the method to assess the safety of live attenuated herpes zoster vaccination during 2011-2017 in US adults aged ≥60 years, using claims data from Truven Health MarketScan Research Databases. Counts of International Classification of Diseases diagnosis codes recorded in emergency department or hospital settings were scanned for any statistically unusual clustering within a hierarchical tree structure of diagnoses and within 42 days after vaccination. Among 1.24 million vaccinations, 4 clusters were found: cellulitis on days 1-3, nonspecific erythematous condition on days 2-4, "other complications . . ." on days 1-3, and nonspecific allergy on days 1-6. These results are consistent with local injection-site reactions and other known, generally mild, vaccine-associated adverse events and a favorable safety profile. This method might be useful for assessing the safety of other vaccines for older adults.

Development and application of two semi-automated tools for targeted medical product surveillance in a distributed data network

PURPOSE OF REVIEW

An important component of the Food and Drug Administration's Sentinel Initiative is the active post-market risk identification and analysis (ARIA) system, which utilizes semi-automated, parameterized computer programs to implement propensity-score adjusted and self-controlled risk interval designs to conduct targeted surveillance of medical products in the Sentinel Distributed Database. In this manuscript, we review literature relevant to the development of these programs and describe their application within the Sentinel Initiative.

RECENT FINDINGS

These quality-checked and publicly available tools have been successfully used to conduct rapid, replicable, and targeted safety analyses of several medical products. In addition to speed and reproducibility, use of semi-automated tools allows investigators to focus on decisions regarding key methodological parameters. We also identified challenges associated with the use of these methods in distributed and prospective datasets like the Sentinel Distributed Database, namely uncertainty regarding the optimal approach to estimating propensity scores in dynamic data among data partners of heterogeneous size.

SUMMARY

Future research should focus on the methodological challenges raised by these applications as well as developing new modular programs for targeted surveillance of medical products.

Minimizing signal detection time in postmarket sequential analysis: balancing positive predictive value and sensitivity

PURPOSE

Outcome misclassification in retrospective epidemiologic analyses has been well-studied, but little is known about such misclassification with respect to sequential statistical analysis during surveillance of medical product-associated risks, a planned capability of the US Food and Drug Administration's Sentinel System.

METHODS

Using a vaccine example, we model and simulate sequential database surveillance in an observational data network using a variety of outcome detection algorithms. We consider how these algorithms, as characterized by sensitivity and positive predictive value, impact the length of surveillance and timeliness of safety signal detection. We show investigators/users of these networks how they can perform preparatory study design calculations that consider outcome misclassification in sequential database surveillance.

RESULTS

Non-differential outcome misclassification generates longer surveillance times and less timely safety signal detection as compared with the case of no misclassification. Inclusive algorithms characterized by high sensitivity but low positive predictive value outperform more narrow algorithms when detecting rare outcomes. This decision calculus may change considerably if medical chart validation procedures were required.

CONCLUSIONS

These findings raise important questions regarding the design of observational data networks used for pharmacovigilance. Specifically, there are tradeoffs involved when choosing to populate such networks with component databases that are large as compared with smaller integrated delivery system databases that can more easily access laboratory or clinical data and perform medical chart validation.

A Broad Safety Assessment of the Recombinant Herpes Zoster Vaccine

The recombinant herpes zoster vaccine (RZV), approved as a 2-dose series in the United States in October 2017, has proven highly effective and generally safe. However, a small risk of Guillain-Barré syndrome after vaccination was identified after approval, and questions remain about other possible adverse events. This data-mining study assessed RZV safety in the United States using the self-controlled tree-temporal scan statistic, scanning data on thousands of diagnoses recorded during follow-up to detect any statistically unusual temporal clustering of cases within a large hierarchy of diagnoses. IBM MarketScan data on commercially insured persons at least 50 years of age receiving RZV between January 1, 2018, and May 5, 2020, were used, including 56 days of follow-up; 1,014,329 doses were included. Statistically significant clustering was found within a few days of vaccination for unspecified adverse effects, complications, or reactions to immunization or other medical substances/care; fever; unspecified allergy; syncope/collapse; cellulitis; myalgia; and dizziness/giddiness. These findings are consistent with the known safety profile of this and other injected vaccines. No cluster of Guillain-Barré syndrome was detected, possibly due to insufficient sample size. This signal-detection method has now been applied to 5 vaccines, with consistently plausible results, and seems a promising addition to vaccine-safety evaluation methods.

A General Propensity Score for Signal Identification Using Tree-Based Scan Statistics

The tree-based scan statistic (TreeScan; Martin Kulldorff, Harvard Medical School, Boston, Massachusetts) is a data-mining method that adjusts for multiple testing of correlated hypotheses when screening thousands of potential adverse events for signal identification. Simulation has demonstrated the promise of TreeScan with a propensity score (PS)-matched cohort design. However, it is unclear which variables to include in a PS for applied signal identification studies to simultaneously adjust for confounding across potential outcomes. We selected 4 pairs of medications with well-understood safety profiles. For each pair, we evaluated 5 candidate PSs with different combinations of 1) predefined general covariates (comorbidity, frailty, utilization), 2) empirically selected (data-driven) covariates, and 3) covariates tailored to the drug pair. For each pair, statistical alerting patterns were similar with alternative PSs (≤11 alerts in 7,996 outcomes scanned). Inclusion of covariates tailored to exposure did not appreciably affect screening results. Inclusion of empirically selected covariates can provide better proxy coverage for confounders but can also decrease statistical power. Unlike tailored covariates, empirical and predefined general covariates can be applied "out of the box" for signal identification. The choice of PS depends on the level of concern about residual confounding versus loss of power. Potential signals should be followed by pharmacoepidemiologic assessment where confounding control is tailored to the specific outcome(s) under investigation.

Association of Risk for Venous Thromboembolism With Use of Low-Dose Extended- and Continuous-Cycle Combined Oral Contraceptives: A Safety Study Using the Sentinel Distributed Database

IMPORTANCE

Continuous/extended cyclic estrogen use (84/7 or 365/0 days cycles) in combined oral contraceptives (COCs) could potentially expose women to an increased cumulative dose of estrogen, compared with traditional cyclic regimens (21/7 days cycle), and may increase the risk for venous thromboembolism (VTE).

OBJECTIVE

To determine, while holding the progestogen type constant, whether the risk for VTE is higher with use of continuous/extended COCs than with cyclic COCs among women who initiated a COC containing ethinyl estradiol and levonorgestrel.

DESIGN, SETTING, AND PARTICIPANTS

Incident user retrospective cohort study of primarily commercially insured US population identified from the Sentinel Distributed Database. Participants were women aged 18 to 50 years at the time of initiating a study COC between May 2007 and September 2015. Using a propensity score approach and Cox proportional hazards regression models, we estimated the hazard ratios of VTE overall and separately by ethinyl estradiol dose and age groups.

EXPOSURES

Initiation of continuous/extended or traditional cyclic COCs containing ethinyl estradiol or levonorgestrel of any dose.

MAIN OUTCOMES AND MEASURES

First VTE hospitalization that occurred during the study follow-up, identified by an inpatient International Classification of Diseases, Ninth Revision, Clinical Modification diagnosis code of 415.1, 415.1x, 453, 453.x, or 453.xx.

RESULTS

We identified 210 691 initiators of continuous/extended COCs (mean [SD] age, 30.4 [8.6] years) and 522 316 initiators of cyclic COCs (mean [SD] age, 28.8 [8.3] years), with a mean of 0.7 person-years at risk among continuous/extended and cyclic users. Baseline cardiovascular and metabolic conditions (7.2% vs 4.7%), gynecological conditions (39.7% vs 32.3%), and health services utilization were slightly higher among continuous/extended cyclic than cyclic COC users. Propensity score matching decreased the hazard ratio estimates from 1.84 (95% CI, 1.53-2.21) to 1.32 (95% CI, 1.07-1.64) for continuous/extended use compared with cyclic COC use. The absolute risk difference (0.27 per 1000 persons) and the incidence rate difference (0.35 cases per 1000 person-years [1.44 vs 1.09 cases per 1000 person-years]) between the 2 propensity score-matched cohorts remained low, which may not translate into a clinically significant risk differences between cyclic and noncyclic estrogen use.

CONCLUSIONS AND RELEVANCE

Holding the progestogen type constant (levonorgestrel), we observed a slightly elevated VTE risk in association with continuous/extended COC use when compared with cyclic COC use. However, due to the small absolute risk difference and potential residual confounding, our findings did not show strong evidence supporting a VTE risk difference between continuous/extended and cyclic COC use.

A broad assessment of covid-19 vaccine safety using tree-based data-mining in the vaccine safety datalink

BACKGROUND

Except for spontaneous reporting systems, vaccine safety monitoring generally involves pre-specifying health outcomes and post-vaccination risk windows of concern. Instead, we used tree-based data-mining to look more broadly for possible adverse events after Pfizer-BioNTech, Moderna, and Janssen COVID-19 vaccination.

METHODS

Vaccine Safety Datalink enrollees receiving ≥1 dose of COVID-19 vaccine in 2020-2021 were followed for 70 days after Pfizer-BioNTech or Moderna and 56 days after Janssen vaccination. Incident diagnoses in inpatient or emergency department settings were analyzed for clustering within both the hierarchical ICD-10-CM code structure and the post-vaccination follow-up period. We used the self-controlled tree-temporal scan statistic and TreeScan software. Monte Carlo simulation was used to estimate p-values; p = 0.01 was the pre-specified cut-off for statistical significance of a cluster.

RESULTS

There were 4.1, 2.6, and 0.4 million Pfizer-BioNTech, Moderna, and Janssen vaccinees, respectively. Clusters after Pfizer-BioNTech vaccination included: (1) unspecified adverse effects, (2) common vaccine reactions, such as fever, myalgia, and headache, (3) myocarditis/pericarditis, and (4) less specific cardiac or respiratory symptoms, all with the strongest clusters generally after Dose 2; and (5) COVID-19/viral pneumonia/sepsis/respiratory failure in the first 3 weeks after Dose 1. Moderna results were similar but without a significant myocarditis/pericarditis cluster. Further investigation suggested the fifth signal group was a manifestation of mRNA vaccine effectiveness after the first 3 weeks. Janssen vaccinees had clusters of unspecified or common vaccine reactions, gait/mobility abnormalities, and muscle weakness. The latter two were deemed to have arisen from confounding related to practices at one site.

CONCLUSIONS

We detected post-vaccination clusters of unspecified adverse effects, common vaccine reactions, and, for the mRNA vaccines, chest pain and palpitations, as well as myocarditis/pericarditis after Pfizer-BioNTech Dose 2. Unique advantages of this data mining are its untargeted nature and its inherent adjustment for the multiplicity of diagnoses and risk intervals scanned.

Leveraging the Capabilities of the FDA's Sentinel System To Improve Kidney Care

The Sentinel System is a national electronic postmarketing resource established by the US Food and Drug Administration to support assessment of the safety and effectiveness of marketed medical products. It has built a large, multi-institutional, distributed data network that contains comprehensive electronic health data, covering about 700 million person-years of longitudinal observation time nationwide. With its sophisticated infrastructure and a large selection of flexible analytic tools, the Sentinel System permits rapid and secure analyses, while preserving patient privacy and health-system autonomy. The Sentinel System also offers enhanced capabilities, including accessing full-text medical records, supporting randomized clinical trials embedded in healthcare delivery systems, and facilitating effective collection of patient-reported data using mobile devices, among many other research programs. The nephrology research community can use the infrastructure, tools, and data that this national resource offers for evidence generation. This review summarizes the Sentinel System and its ability to rapidly generate high-quality, real-world evidence; discusses the program's experience in, and potential for, addressing gaps in kidney care; and outlines avenues for conducting research, leveraging this national resource in collaboration with Sentinel investigators.

A Synthesis of Current Surveillance Planning Methods for the Sequential Monitoring of Drug and Vaccine Adverse Effects Using Electronic Health Care Data

INTRODUCTION

The large-scale assembly of electronic health care data combined with the use of sequential monitoring has made proactive postmarket drug- and vaccine-safety surveillance possible. Although sequential designs have been used extensively in randomized trials, less attention has been given to methods for applying them in observational electronic health care database settings.

EXISTING METHODS

We review current sequential-surveillance planning methods from randomized trials, and the Vaccine Safety Datalink (VSD) and Mini-Sentinel Pilot projects-two national observational electronic health care database safety monitoring programs.

FUTURE SURVEILLANCE PLANNING

Based on this examination, we suggest three steps for future surveillance planning in health care databases: (1) prespecify the sequential design and analysis plan, using available feasibility data to reduce assumptions and minimize later changes to initial plans; (2) assess existing drug or vaccine uptake, to determine if there is adequate information to proceed with surveillance, before conducting more resource-intensive planning; and (3) statistically evaluate and clearly communicate the sequential design with all those designing and interpreting the safety-surveillance results prior to implementation. Plans should also be flexible enough to accommodate dynamic and often unpredictable changes to the database information made by the health plans for administrative purposes.

CONCLUSIONS

This paper is intended to encourage dialogue about establishing a more systematic, scalable, and transparent sequential design-planning process for medical-product safety-surveillance systems utilizing observational electronic health care databases. Creating such a framework could yield improvements over existing practices, such as designs with increased power to assess serious adverse events.

The Devil's in the details: Reports on reproducibility in pharmacoepidemiologic studies

PURPOSE

The U.S. Food and Drug Administration's Sentinel Initiative "modular programs" have been shown to replicate findings from conventional protocol-driven, custom-programmed studies. One such parallel assessment-dabigatran and warfarin and selected outcomes-produced concordant findings for three of four study outcomes. The effect estimates and confidence intervals for the fourth-acute myocardial infarction-had more variability as compared with other outcomes. This paper evaluates the potential sources of that variability that led to unexpected divergence in findings.

METHODS

We systematically compared the two studies and evaluated programming differences and their potential impact using a different dataset that allowed more granular data access for investigation. We reviewed the output at each of five main processing steps common in both study programs: cohort identification, propensity score estimation, propensity score matching, patient follow-up, and risk estimation.

RESULTS

Our findings point to several design features that warrant greater investigator attention when performing observational database studies: (a) treatment of recorded events (eg, diagnoses, procedures, and dispensings) co-occurring on the index date of study drug dispensing in cohort eligibility criteria and propensity score estimation and (b) construction of treatment episodes for study drugs of interest that have more complex dispensing patterns.

CONCLUSIONS

More precise and unambiguous operational definitions of all study parameters will increase transparency and reproducibility in observational database studies.

Statistical Power for Postlicensure Medical Product Safety Data Mining

OBJECTIVE

To perform sample size calculations when using tree-based scan statistics in longitudinal observational databases.

METHODS

Tree-based scan statistics enable data mining on epidemiologic datasets where thousands of disease outcomes are organized into hierarchical tree structures with automatic adjustment for multiple testing. We show how to evaluate the statistical power of the unconditional and conditional Poisson versions. The null hypothesis is that there is no increase in the risk for any of the outcomes. The alternative is that one or more outcomes have an excess risk. We varied the excess risk, total sample size, frequency of the underlying event rate, and the level of across-the-board health care utilization. We also quantified the reduction in statistical power resulting from specifying a risk window that was too long or too short.

RESULTS

For 500,000 exposed people, we had at least 98 percent power to detect an excess risk of 1 event per 10,000 exposed for all outcomes. In the presence of potential temporal confounding due to across-the-board elevations of health care utilization in the risk window, the conditional tree-based scan statistic controlled type I error well, while the unconditional version did not.

DISCUSSION

Data mining analyses using tree-based scan statistics expand the pharmacovigilance toolbox, ensuring adequate monitoring of thousands of outcomes of interest while controlling for multiple hypothesis testing. These power evaluations enable investigators to design and optimize implementation of retrospective data mining analyses.

Health outcomes coding trends in the US Food and Drug Administration's Sentinel System during transition to International Classification of Diseases-10 coding system: A brief review

Background and purpose

The transition from International Classification of Diseases, 9th revision, clinical modification (ICD‐9‐CM) to ICD‐10‐CM poses a challenge to epidemiologic studies that use diagnostic codes to identify health outcomes and covariates. We evaluated coding trends in health outcomes in the US Food and Drug Administration's Sentinel System during the transition.

Methods

We reviewed all health outcomes coding trends reports on the Sentinel website through November 30, 2019 and analyzed trends in incidence and prevalence across the ICD‐9‐CM and ICD‐10‐CM eras by visual inspection.

Results

We identified 78 unique health outcomes (22 acute, 32 chronic, and 24 acute or chronic) and 140 time‐series graphs of incidence and prevalence. The reports also included code lists and code mapping methods used. Of the 140 graphs reviewed, 81 (57.9%) showed consistent trends across the ICD‐9‐CM and ICD‐10‐CM eras, while 51 (36.4%) and 8 (5.7%) graphs showed inconsistent and uncertain trends, respectively. Chronic HOIs and acute/chronic HOIs had higher proportions of consistent trends in prevalence definitions (83.9% and 78.3%, respectively) than acute HOIs (28.6%). For incidence, 55.6% of acute HOIs showed consistent trends, while 41.2% of chronic HOIs and 39.3% of acute/chronic HOIs showed consistency.

Conclusions

Researchers using ICD‐10‐CM algorithms obtained by standardized mappings from ICD‐9‐CM algorithms should assess the mapping performance before use. The Sentinel reports provide a valuable resource for researchers who need to develop and assess mapping strategies. The reports could benefit from additional information about the algorithm selection process and additional details on monthly incidence and prevalence rates.

Key points

We reviewed health outcomes coding trends reports on the US FDA Sentinel website through November 30, 2019 and analyzed trends in incidence and prevalence across the International Classification of Diseases, 9th revision, Clinical Modification (ICD‐9‐CM) and ICD‐10‐CM eras by code mapping method and the type of health outcomes of interest (acute, chronic, acute or chronic).

More than a third of the 140 time‐series graphs of incidence and prevalence of health outcomes showed inconsistent or uncertain trends. Consistency in trends varied by code mapping method, type of health outcomes of interest, and whether the measurement was incidence or prevalence.

Studies using ICD‐9‐CM‐based algorithms mapped to ICD‐10‐CM codes need to assess the performance of the mappings and conduct manual refinement of the algorithms as needed before using them.

Orphan therapies: making best use of postmarket data

BACKGROUND

Postmarket surveillance of the comparative safety and efficacy of orphan therapeutics is challenging, particularly when multiple therapeutics are licensed for the same orphan indication. To make best use of product-specific registry data collected to fulfill regulatory requirements, we propose the creation of a distributed electronic health data network among registries. Such a network could support sequential statistical analyses designed to detect early warnings of excess risks. We use a simulated example to explore the circumstances under which a distributed network may prove advantageous.

METHODS

We perform sample size calculations for sequential and non-sequential statistical studies aimed at comparing the incidence of hepatotoxicity following initiation of two newly licensed therapies for homozygous familial hypercholesterolemia. We calculate the sample size savings ratio, or the proportion of sample size saved if one conducted a sequential study as compared to a non-sequential study. Then, using models to describe the adoption and utilization of these therapies, we simulate when these sample sizes are attainable in calendar years. We then calculate the analytic calendar time savings ratio, analogous to the sample size savings ratio. We repeat these analyses for numerous scenarios.

KEY RESULTS

Sequential analyses detect effect sizes earlier or at the same time as non-sequential analyses. The most substantial potential savings occur when the market share is more imbalanced (i.e., 90% for therapy A) and the effect size is closest to the null hypothesis. However, due to low exposure prevalence, these savings are difficult to realize within the 30-year time frame of this simulation for scenarios in which the outcome of interest occurs at or more frequently than one event/100 person-years.

CONCLUSIONS

We illustrate a process to assess whether sequential statistical analyses of registry data performed via distributed networks may prove a worthwhile infrastructure investment for pharmacovigilance.