The US Food and Drug Administration Sentinel System: a national resource for a learning health system

Real-World Evidence to Support EU Regulatory Decision Making-Results From a Pilot of Regulatory Use Cases

Studies using real-world data (RWD) can complement evidence from clinical trials and fill evidence gaps during different stages of a medicine's lifecycle. This review presents the experience resulting from the European Medicines Agency (EMA) pilot to generate RWE to support evaluations by EU regulators and down-stream decision makers from September 2021 to February 2023. A total of 61 research topics were identified for RWE generation during this period, covering a wide range of research questions, primarily generating evidence on medicines safety (22, 36%), followed by questions on the design and feasibility of clinical trials (11, 18%), drug utilization (10, 16%), clinical management (10, 16%), and disease epidemiology. A significant number of questions were related to the pediatric population and/or rare diseases. A total of 27 regulatory-led RWD studies have been conducted. Most studies were descriptive and aimed at estimating incidence and prevalence rates of clinical outcomes including adverse events or to evaluate medicines utilization. The review highlights key learnings to guide further efforts to enable the use and establish the value of real-world evidence (RWE) for regulatory decisions. For instance, there is a need to access additional fit-for-purpose and representative data, and to explore further means to provide timely evidence that meets regulatory timelines. The need for early interactions and close collaboration with study requesters, e.g., from the Agency's scientific Committees, to better understand the research question is equally important. Finally, the review provides our perspective on the way forward to maximize the potential of regulatory-led RWE generation.

Use of the Sentinel System to Examine Medical Product Use and Outcomes During Pregnancy

While many pregnant individuals use prescription medications, evidence supporting product safety during pregnancy is often inadequate. Existing electronic healthcare data sources provide large, diverse samples of health plan members to allow for the study of medical product utilization during pregnancy, as well as pregnancy, maternal, and infant outcomes. The Sentinel System is a national medical product surveillance system that includes administrative claims and electronic health record databases from large national and regional health insurers. In addition to these data sources, Sentinel develops and maintains a sizeable selection of analytic tools to facilitate epidemiologic analyses in a way that protects patient privacy and health system autonomy. In this article, we provide an overview of Sentinel System infrastructure, including the Mother-Infant Linkage Table, parameterizable analytic tools, and algorithms to estimate gestational age and identify pregnancy outcomes. We also describe past and future Sentinel work that contributes to our understanding of the way medical products are used and the safety of these products during pregnancy.

Trust but Verify: Lessons Learned for the Application of AI to Case-Based Clinical Decision-Making From Postmarketing Drug Safety Assessment at the US Food and Drug Administration

Adverse drug reactions are a common cause of morbidity in health care. The US Food and Drug Administration (FDA) evaluates individual case safety reports of adverse events (AEs) after submission to the FDA Adverse Event Reporting System as part of its surveillance activities. Over the past decade, the FDA has explored the application of artificial intelligence (AI) to evaluate these reports to improve the efficiency and scientific rigor of the process. However, a gap remains between AI algorithm development and deployment. This viewpoint aims to describe the lessons learned from our experience and research needed to address both general issues in case-based reasoning using AI and specific needs for individual case safety report assessment. Beginning with the recognition that the trustworthiness of the AI algorithm is the main determinant of its acceptance by human experts, we apply the Diffusion of Innovations theory to help explain why certain algorithms for evaluating AEs at the FDA were accepted by safety reviewers and others were not. This analysis reveals that the process by which clinicians decide from case reports whether a drug is likely to cause an AE is not well defined beyond general principles. This makes the development of high performing, transparent, and explainable AI algorithms challenging, leading to a lack of trust by the safety reviewers. Even accounting for the introduction of large language models, the pharmacovigilance community needs an improved understanding of causal inference and of the cognitive framework for determining the causal relationship between a drug and an AE. We describe specific future research directions that underpin facilitating implementation and trust in AI for drug safety applications, including improved methods for measuring and controlling of algorithmic uncertainty, computational reproducibility, and clear articulation of a cognitive framework for causal inference in case-based reasoning.

When Will Real-World Data Fulfill Its Promise to Provide Timely Insights in Oncology?

Randomized trials provide high-quality, internally consistent data on selected clinical questions, but lack generalizability for the aging population who are most often diagnosed with cancer and have comorbid conditions that may affect the interpretation of treatment benefit. The need for high-quality, relevant, and timely data is greater than ever. Promising solutions lie in the collection and analysis of real-world data (RWD), which can potentially provide timely insights about the patient's course during and after initial treatment and the outcomes of important subgroups such as the elderly, rural populations, children, and patients with greater social health needs. However, to inform practice and policy, real-world evidence must be created from trustworthy and comprehensive sources of RWD; these may include pragmatic clinical trials, registries, prospective observational studies, electronic health records (EHRs), administrative claims, and digital technologies. There are unique challenges in oncology since key parameters (eg, cancer stage, biomarker status, genomic assays, imaging response, side effects, quality of life) are not recorded, siloed in inaccessible documents, or available only as free text or unstructured reports in the EHR. Advances in analytics, such as artificial intelligence, may greatly enhance the ability to obtain more granular information from EHRs and support integrated diagnostics; however, they will need to be validated purpose by purpose. We recommend a commitment to standardizing data across sources and building infrastructures that can produce fit-for-purpose RWD that will provide timely understanding of the effectiveness of individual interventions.

Transparency, reproducibility, and replicability of pharmacoepidemiology studies in a distributed network environment

PURPOSE

Our objective is to describe how the U.S. Food and Drug Administration (FDA)'s Sentinel System implements best practices to ensure trust in drug safety studies using real-world data from disparate sources.

METHODS

We present a stepwise schematic for Sentinel's data harmonization, data quality check, query design and implementation, and reporting practices, and describe approaches to enhancing the transparency, reproducibility, and replicability of studies at each step.

CONCLUSIONS

Each Sentinel data partner converts its source data into the Sentinel Common Data Model. The transformed data undergoes rigorous quality checks before it can be used for Sentinel queries. The Sentinel Common Data Model framework, data transformation codes for several data sources, and data quality assurance packages are publicly available. Designed to run against the Sentinel Common Data Model, Sentinel's querying system comprises a suite of pre-tested, parametrizable computer programs that allow users to perform sophisticated descriptive and inferential analysis without having to exchange individual-level data across sites. Detailed documentation of capabilities of the programs as well as the codes and information required to execute them are publicly available on the Sentinel website. Sentinel also provides public trainings and online resources to facilitate use of its data model and querying system. Its study specifications conform to established reporting frameworks aimed at facilitating reproducibility and replicability of real-world data studies. Reports from Sentinel queries and associated design and analytic specifications are available for download on the Sentinel website. Sentinel is an example of how real-world data can be used to generate regulatory-grade evidence at scale using a transparent, reproducible, and replicable process.

Generating and using real-world data: A worthwhile uphill battle

The precision oncology paradigm challenges the feasibility and data generalizability of traditional clinical trials. Consequently, an unmet need exists for practical approaches to test many subgroups, evaluate real-world drug value, and gather comprehensive, accessible datasets to validate novel biomarkers. Real-world data (RWD) are increasingly recognized to have the potential to fill this gap in research methodology. Established applications of RWD include informing disease epidemiology, pharmacovigilance, and healthcare quality assessment. Currently, concerns regarding RWD quality and comprehensiveness, privacy, and biases hamper their broader application. Nonetheless, RWD may play a pivotal role in supplementing clinical trials, enabling conditional reimbursement and accelerated drug access, and innovating trial conduct. Moreover, purpose-built RWD repositories may support the extension or refinement of drug indications and facilitate the discovery and validation of new biomarkers. This perspective explores the potential of leveraging RWD to advance oncology, highlights its benefits and challenges, and suggests a path forward in this evolving field.

Validation of an ICD-10 case-finding algorithm for endometrial cancer in US insurance claims

PURPOSE

To evaluate the positive predictive value (PPV) of an endometrial cancer case finding algorithm using International Classification of Disease 10th revision Clinical Modification (ICD-10-CM) diagnosis codes from US insurance claims for implementation in a planned post-marketing safety study. Two algorithm variants were evaluated.

METHODS

Provisional incident endometrial cancer cases were identified from 2016 through 2020 among women aged ≥50 years. One algorithm variant used diagnosis codes for malignant neoplasms of uterine sites (C54.x), excluding C54.2 (malignant neoplasm of myometrium); the other used only C54.1 (malignant neoplasm of endometrium). A random sample of medical records of recent incident provisional cases (2018-2020) was requested for adjudication. Confirmed cases showed biopsy evidence of endometrial cancer, documentation of cancer staging, or hysterectomy following diagnosis. We estimated the PPV of the variants with 95% confidence intervals (CI) excluding cases that had insufficient information.

RESULTS

Of 294 provisional cases adjudicated, 85% were from outpatient settings (n = 249). Mean age at diagnosis was 69.3 years. Among the 294 adjudicated cases (identified with the broader algorithm variant), the same 223 were confirmed endometrial cancer cases by both algorithm variants. The PPV (95% CI) for the broader algorithm variant was 84.2% (79.2% and 88.3%), and for the variant using only C54.1 was 85.8% (80.9% and 89.8%).

CONCLUSION

We developed and validated an algorithm using ICD-10-CM diagnosis codes to identify endometrial cancer cases in health insurance claims with a sufficiently high PPV to use in a planned post-marketing safety study.

Clinical pharmacology facing the real-world setting: Pharmacovigilance, pharmacoepidemiology and the economic evaluation of drugs

Traditionally, clinical pharmacology has focused its activities on drug-organism interaction, from an individual or collective perspective. Drug efficacy assessment by performing randomized clinical trials and analysis of drug use in clinical practice by carrying out drug utilization studies have also been other areas of interest. From now on, Clinical pharmacology should move from the analysis of the drug-individual interaction to the analysis of the drug-individual-society interaction. It should also analyze the clinical and economic consequences of the use of drugs in the conditions of normal clinical practice, beyond clinical trials. The current exponential technological development that facilitates the analysis of real-life data offers us a golden opportunity to move to all these other areas of interest. This review describes the role that clinical pharmacology has played at the beginning and during the evolution of pharmacovigilance, pharmacoepidemiology and economic drug evaluations in Spain. In addition, the challenges that clinical pharmacology is going to face in the following years in these three areas are going to be outlined too.

Switching patterns of immediate-release forms of generic mixed amphetamine salts products among privately and publicly insured individuals aged 15-64 years in the United States, 2013-2019

PURPOSE

Immediate-release forms of generic mixed amphetamine salts (MAS) have been the subject of passive surveillance reports signaling lack of effectiveness. We examined switching patterns that might suggest whether long-term users of specific MAS are more likely to switch away or switch back after use of the MAS of interest in the FDA's Sentinel Distributed Database.

METHODS

We required at least 60-day continuous supply of selected MAS grouped by Abbreviated New Drug Application (ANDA) to describe patterns of switching away from and to generics approved under the ANDAs of interest among individuals ages 15-64 years with attention deficit hyperactivity disorder or narcolepsy during 2013-2019.

RESULTS

We observed the greatest number of treatment episodes for ANDA 040422 (n = 525 771), followed by ANDA 202424 (n = 181 693), ANDA 040439 (n = 62 363), ANDA 040440 (n = 21 143), and ANDA 040480 (n = 8792). Of those with switches away from their original ANDA, episodes initiated on generic products under ANDA 040422 (48.6%) and ANDA 202424 (43.0%) were most likely to switch back, while those initiated on generic product under ANDA 040480 were least likely (24.1%). Of those episodes with switches to a generic under an ANDA of interest, about one-third (range 27.1% to 37.0%) switched back to the same product. These switches back had a median time to switch of about 30 days.

CONCLUSIONS

These descriptive analyses, although subject to limitations, did not suggest increased switching away or switching back after use of the generics of interest. Continued post-marketing surveillance is warranted.

Improving data capture of race and ethnicity for the Food and Drug Administration Sentinel database: a narrative review

PURPOSE

The U.S. Food and Drug Administration's Sentinel System is a national medical product safety surveillance system consisting of a large multisite distributed database of administrative claims supplemented by electronic health-care record data. The program seeks to improve data capture of race and ethnicity for pharmacoepidemiology studies.

METHODS

We conducted a narrative literature review of published research on data augmentation and imputation methods to improve race and ethnicity capture in U.S. health-care systems databases. We focused on methods with limited (five-digit ZIP codes only) or full patient identifiers available to link to external sources of self-reported data. We organized the literature by themes: (1) variation in data capture of self-reported data, (2) data augmentation from external sources of self-reported data, and (3) imputation methods, including Bayesian analysis and multiple regression.

RESULTS

Researchers reduced data missingness with high validity for Asian, Black, White, and Pacific Islander racial groups and Hispanic ethnicity. Native American and multiracial groups were difficult to validate due to relatively small sample sizes.

CONCLUSIONS

Limitations on accessible self-reported data for validation will dictate methods to improve race and ethnicity data capture. We recommend methods leveraging multiple sources that account for variations in geography, age, and sex.

Six Years of the US Food and Drug Administration's Postmarket Active Risk Identification and Analysis System in the Sentinel Initiative: Implications for Real World Evidence Generation

Congress mandated the creation of a postmarket Active Risk Identification and Analysis (ARIA) system containing data on 100 million individuals for monitoring risks associated with drug and biologic products using data from disparate sources to complement the US Food and Drug Administration's (FDA's) existing postmarket capabilities. We report on the first 6 years of ARIA utilization in the Sentinel System (2016-2021). The FDA has used the ARIA system to evaluate 133 safety concerns; 54 of these evaluations have closed with regulatory determinations, whereas the rest remain in progress. If the ARIA system and the FDA's Adverse Event Reporting System are deemed insufficient to address a safety concern, then the FDA may issue a postmarket requirement to a product's manufacturer. One hundred ninety-seven ARIA insufficiency determinations have been made. The most common situation for which ARIA was found to be insufficient is the evaluation of adverse pregnancy and fetal outcomes following in utero drug exposure, followed by neoplasms and death. ARIA was most likely to be sufficient for thromboembolic events, which have high positive predictive value in claims data alone and do not require supplemental clinical data. The lessons learned from this experience illustrate the continued challenges using administrative claims data, especially to define novel clinical outcomes. This analysis can help to identify where more granular clinical data are needed to fill gaps to improve the use of real-world data for drug safety analyses and provide insights into what is needed to efficiently generate high-quality real-world evidence for efficacy.

The Evolve to Next-Gen ACT Network: An evolving open-access, real-world data resource primed for real-world evidence research across the Clinical and Translational Science Award Consortium

The ACT Network was funded by NIH to provide investigators from across the Clinical and Translational Science Award (CTSA) Consortium the ability to directly query national federated electronic health record (EHR) data for cohort discovery and feasibility assessment of multi-site studies. NIH refunded the program for expanded research application to become "Evolve to Next-Gen ACT" (ENACT). In parallel, the US Food and Drug Administration has been evaluating the use of real-world data (RWD), including EHR data, as sources of real-world evidence (RWE) for its regulatory decisions involving drug and biological products. Using insights from implementation science, six lessons learned from ACT for developing and sustaining RWD/RWE infrastructures and networks across the CTSA Consortium are presented in order to inform ENACT's development from the outset. Lessons include intentional institutional relationship management, end-user engagement, beta-testing, and customer-driven adaptation. The ENACT team is also conducting customer discovery interviews with CTSA hub and investigators using Innovation-Corps@NCATS (I-Corps™) methodology for biomedical entrepreneurs to uncover unmet RWD needs. Possible ENACT value proposition hypotheses are presented by stage of research. Developing evidence about methods for sustaining academically derived data infrastructures and support can advance the science of translation and support our nation's RWD/RWE research capacity.

Common data model for sickle cell disease surveillance: considerations and implications

Objective

Population-level data on sickle cell disease (SCD) are sparse in the United States. The Centers for Disease Control and Prevention (CDC) is addressing the need for SCD surveillance through state-level Sickle Cell Data Collection Programs (SCDC). The SCDC developed a pilot common informatics infrastructure to standardize processes across states.

Materials and Methods

We describe the process for establishing and maintaining the proposed common informatics infrastructure for a rare disease, starting with a common data model and identify key data elements for public health SCD reporting.

Results

The proposed model is constructed to allow pooling of table shells across states for comparison. Core Surveillance Data reports are compiled based on aggregate data provided by states to CDC annually.

Discussion and Conclusion

We successfully implemented a pilot SCDC common informatics infrastructure to strengthen our distributed data network and provide a blueprint for similar initiatives in other rare diseases.