The FDA's sentinel initiative-A comprehensive approach to medical product surveillance

Assessing Real-World Data From Electronic Health Records for Health Technology Assessment: The SUITABILITY Checklist: A Good Practices Report of an ISPOR Task Force

This ISPOR Good Practices report provides a framework for assessing the suitability of electronic health records data for use in health technology assessments (HTAs). Although electronic health record (EHR) data can fill evidence gaps and improve decisions, several important limitations can affect its validity and relevance. The ISPOR framework includes 2 components: data delineation and data fitness for purpose. Data delineation provides a complete understanding of the data and an assessment of its trustworthiness by describing (1) data characteristics; (2) data provenance; and (3) data governance. Fitness for purpose comprises (1) data reliability items, ie, how accurate and complete the estimates are for answering the question at hand and (2) data relevance items, which assess how well the data are suited to answer the particular question from a decision-making perspective. The report includes a checklist specific to EHR data reporting: the ISPOR SUITABILITY Checklist. It also provides recommendations for HTA agencies and policy makers to improve the use of EHR-derived data over time. The report concludes with a discussion of limitations and future directions in the field, including the potential impact from the substantial and rapid advances in the diffusion and capabilities of large language models and generative artificial intelligence. The report's immediate audiences are HTA evidence developers and users. We anticipate that it will also be useful to other stakeholders, particularly regulators and manufacturers, in the future.

Transparency, reproducibility, and replicability of pharmacoepidemiology studies in a distributed network environment

PURPOSE

Our objective is to describe how the U.S. Food and Drug Administration (FDA)'s Sentinel System implements best practices to ensure trust in drug safety studies using real-world data from disparate sources.

METHODS

We present a stepwise schematic for Sentinel's data harmonization, data quality check, query design and implementation, and reporting practices, and describe approaches to enhancing the transparency, reproducibility, and replicability of studies at each step.

CONCLUSIONS

Each Sentinel data partner converts its source data into the Sentinel Common Data Model. The transformed data undergoes rigorous quality checks before it can be used for Sentinel queries. The Sentinel Common Data Model framework, data transformation codes for several data sources, and data quality assurance packages are publicly available. Designed to run against the Sentinel Common Data Model, Sentinel's querying system comprises a suite of pre-tested, parametrizable computer programs that allow users to perform sophisticated descriptive and inferential analysis without having to exchange individual-level data across sites. Detailed documentation of capabilities of the programs as well as the codes and information required to execute them are publicly available on the Sentinel website. Sentinel also provides public trainings and online resources to facilitate use of its data model and querying system. Its study specifications conform to established reporting frameworks aimed at facilitating reproducibility and replicability of real-world data studies. Reports from Sentinel queries and associated design and analytic specifications are available for download on the Sentinel website. Sentinel is an example of how real-world data can be used to generate regulatory-grade evidence at scale using a transparent, reproducible, and replicable process.

Neurocircuitry underlying the actions of glucagon-like peptide 1 and peptide YY3-36 in the suppression of food, drug-seeking, and anxiogenesis

Obesity is a critical health condition worldwide that increases the risks of comorbid chronic diseases, but it can be managed with weight loss. However, conventional interventions relying on diet and exercise are inadequate for achieving and maintaining weight loss, thus there is significant market interest for pharmaceutical anti-obesity agents. For decades, receptor agonists for the gut peptide glucagon-like peptide 1 (GLP-1) featured prominently in anti-obesity medications by suppressing appetite and food reward to elicit rapid weight loss. As the neurocircuitry underlying food motivation overlaps with that for drugs of abuse, GLP-1 receptor agonism has also been shown to decrease substance use and relapse, thus its therapeutic potential may extend beyond weight management to treat addictions. However, as prolonged use of anti-obesity drugs may increase the risk of mood-related disorders like anxiety and depression, and individuals taking GLP-1-based medication commonly report feeling demotivated, the long-term safety of such drugs is an ongoing concern. Interestingly, current research now focuses on dual agonist approaches that include GLP-1 receptor agonism to enable synergistic effects on weight loss or associated functions. GLP-1 is secreted from the same intestinal cells as the anorectic gut peptide, Peptide YY3-36 (PYY3-36), thus this review assessed the therapeutic potential and underlying neural circuits targeted by PYY3-36 when administered independently or in combination with GLP-1 to curb the appetite for food or drugs of abuse like opiates, alcohol, and nicotine. Additionally, we also reviewed animal and human studies to assess the impact, if any, for GLP-1 and/or PYY3-36 on mood-related behaviors in relation to anxiety and depression. As dual agonists targeting GLP-1 and PYY3-36 may produce synergistic effects, they can be effective at lower doses and offer an alternative approach for therapeutic benefits while mitigating undesirable side effects.

Generating and using real-world data: A worthwhile uphill battle

The precision oncology paradigm challenges the feasibility and data generalizability of traditional clinical trials. Consequently, an unmet need exists for practical approaches to test many subgroups, evaluate real-world drug value, and gather comprehensive, accessible datasets to validate novel biomarkers. Real-world data (RWD) are increasingly recognized to have the potential to fill this gap in research methodology. Established applications of RWD include informing disease epidemiology, pharmacovigilance, and healthcare quality assessment. Currently, concerns regarding RWD quality and comprehensiveness, privacy, and biases hamper their broader application. Nonetheless, RWD may play a pivotal role in supplementing clinical trials, enabling conditional reimbursement and accelerated drug access, and innovating trial conduct. Moreover, purpose-built RWD repositories may support the extension or refinement of drug indications and facilitate the discovery and validation of new biomarkers. This perspective explores the potential of leveraging RWD to advance oncology, highlights its benefits and challenges, and suggests a path forward in this evolving field.

Staging and clean room: Constructs designed to facilitate transparency and reduce bias in comparative analyses of real-world data

PURPOSE

We describe constructs designed to protect the integrity of the results from comparative analyses using real-world data (RWD): staging and clean room.

METHODS

Staging involves performing sequential preliminary analyses and evaluating the population size available and potential bias before conducting comparative analyses. A clean room involves restricted access to data and preliminary results, policies governing exploratory analyses and protocol deviations, and audit trail. These constructs are intended to allow decisions about protocol deviations, such as changes to design or model specification, to be made without knowledge of how they might affect subsequent analyses. We describe an example for implementing staging with a clean room.

RESULTS

Stage 1 may involve selecting a data source, developing and registering a protocol, establishing a clean room, and applying inclusion/exclusion criteria. Stage 2 may involve attempting to achieve covariate balance, often through propensity score models. Stage 3 may involve evaluating the presence of residual confounding using negative control outcomes. After each stage, check points may be implemented when a team of statisticians, epidemiologists and clinicians masked to how their decisions may affect study outcomes, reviews the results. This review team may be tasked with making recommendations for protocol deviations to address study precision or bias. They may recommend proceeding to the next stage, conducting additional analyses to address bias, or terminating the study. Stage 4 may involve conducting the comparative analyses.

CONCLUSIONS

The staging and clean room constructs are intended to protect the integrity and enhance confidence in the results of analyses of RWD.

Data-driven automated classification algorithms for acute health conditions: applying PheNorm to COVID-19 disease

OBJECTIVES

Automated phenotyping algorithms can reduce development time and operator dependence compared to manually developed algorithms. One such approach, PheNorm, has performed well for identifying chronic health conditions, but its performance for acute conditions is largely unknown. Herein, we implement and evaluate PheNorm applied to symptomatic COVID-19 disease to investigate its potential feasibility for rapid phenotyping of acute health conditions.

MATERIALS AND METHODS

PheNorm is a general-purpose automated approach to creating computable phenotype algorithms based on natural language processing, machine learning, and (low cost) silver-standard training labels. We applied PheNorm to cohorts of potential COVID-19 patients from 2 institutions and used gold-standard manual chart review data to investigate the impact on performance of alternative feature engineering options and implementing externally trained models without local retraining.

RESULTS

Models at each institution achieved AUC, sensitivity, and positive predictive value of 0.853, 0.879, 0.851 and 0.804, 0.976, and 0.885, respectively, at quantiles of model-predicted risk that maximize F1. We report performance metrics for all combinations of silver labels, feature engineering options, and models trained internally versus externally.

DISCUSSION

Phenotyping algorithms developed using PheNorm performed well at both institutions. Performance varied with different silver-standard labels and feature engineering options. Models developed locally at one site also worked well when implemented externally at the other site.

CONCLUSION

PheNorm models successfully identified an acute health condition, symptomatic COVID-19. The simplicity of the PheNorm approach allows it to be applied at multiple study sites with substantially reduced overhead compared to traditional approaches.

Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data

Objective

To explore how design emulation and population differences relate to variation in results between randomised controlled trials (RCT) and non-randomised real world evidence (RWE) studies, based on the RCT-DUPLICATE initiative (Randomised, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology).

Design

Meta-analysis of RCT-DUPLICATE data.

Data sources

Trials included in RCT-DUPLICATE, a demonstration project that emulated 32 randomised controlled trials using three real world data sources: Optum Clinformatics Data Mart, 2004-19; IBM MarketScan, 2003-17; and subsets of Medicare parts A, B, and D, 2009-17.

Eligibility criteria for selecting studies

Trials where the primary analysis resulted in a hazard ratio; 29 RCT-RWE study pairs from RCT-DUPLICATE.

Results

Differences and variation in effect sizes between the results from randomised controlled trials and real world evidence studies were investigated. Most of the heterogeneity in effect estimates between the RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: treatment started in hospital (which does not appear in health insurance claims data), discontinuation of some baseline treatments at randomisation (which would have been an unusual care decision in clinical practice), and delayed onset of drug effects (which would be under-reported in real world clinical practice because of the relatively short persistence of the treatment). Adding the three emulation differences to the meta-regression reduced heterogeneity from 1.9 to almost 1 (absence of heterogeneity).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from randomised controlled trials and real world evidence studies can be attributed to differences in design emulation.

Real-world data analysis on effectiveness of integrative therapies: A practical guide to study design and data analysis using healthcare databases

Real world data (RWD) is increasingly used to investigate health outcomes and treatment efficacy in the field of integrative medicine. Due to the fact that the majority of RWDs are not intended for research, their secondary use in research necessitates complex study designs to account for bias and confounding. To conduct a robust analysis of RWD in integrative medicine, a comprehensive study design process that reflects the characteristics of integrative therapies is necessary. In this paper, we present a guide for designing comparative effectiveness RWE research in integrative medicine. We discuss key factors to consider when selecting RWDs for research on integrative medicine. We provide practical steps for developing a research question, formulating the PICOT objectives (population, intervention, comparator, outcome, and time horizon), and selecting and defining covariates with a summary table. Specific study designs are depicted with corresponding diagrams. Finally, data analysis procedures are introduced. We hope this article clarifies the importance of RWE research design and related processes in order to improve the rigor of RWD studies in the field of integrative medicine research.

Adult Moyamoya Disease and Syndrome: Current Perspectives and Future Directions: A Scientific Statement From the American Heart Association/American Stroke Association

Adult moyamoya disease and syndrome are rare disorders with significant morbidity and mortality. A writing group of experts was selected to conduct a literature search, summarize the current knowledge on the topic, and provide a road map for future investigation. The document presents an update in the definitions of moyamoya disease and syndrome, modern methods for diagnosis, and updated information on pathophysiology, epidemiology, and both medical and surgical treatment. Despite recent advancements, there are still many unresolved questions about moyamoya disease and syndrome, including lack of unified diagnostic criteria, reliable biomarkers, better understanding of the underlying pathophysiology, and stronger evidence for treatment guidelines. To advance progress in this area, it is crucial to acknowledge the limitations and weaknesses of current studies and explore new approaches, which are outlined in this scientific statement for future research strategies.

Six Years of the US Food and Drug Administration's Postmarket Active Risk Identification and Analysis System in the Sentinel Initiative: Implications for Real World Evidence Generation

Congress mandated the creation of a postmarket Active Risk Identification and Analysis (ARIA) system containing data on 100 million individuals for monitoring risks associated with drug and biologic products using data from disparate sources to complement the US Food and Drug Administration's (FDA's) existing postmarket capabilities. We report on the first 6 years of ARIA utilization in the Sentinel System (2016-2021). The FDA has used the ARIA system to evaluate 133 safety concerns; 54 of these evaluations have closed with regulatory determinations, whereas the rest remain in progress. If the ARIA system and the FDA's Adverse Event Reporting System are deemed insufficient to address a safety concern, then the FDA may issue a postmarket requirement to a product's manufacturer. One hundred ninety-seven ARIA insufficiency determinations have been made. The most common situation for which ARIA was found to be insufficient is the evaluation of adverse pregnancy and fetal outcomes following in utero drug exposure, followed by neoplasms and death. ARIA was most likely to be sufficient for thromboembolic events, which have high positive predictive value in claims data alone and do not require supplemental clinical data. The lessons learned from this experience illustrate the continued challenges using administrative claims data, especially to define novel clinical outcomes. This analysis can help to identify where more granular clinical data are needed to fill gaps to improve the use of real-world data for drug safety analyses and provide insights into what is needed to efficiently generate high-quality real-world evidence for efficacy.

Cardiovascular Care Innovation through Data-Driven Discoveries in the Electronic Health Record

The electronic health record (EHR) represents a rich source of patient information, increasingly being leveraged for cardiovascular research. Although its primary use remains the seamless delivery of health care, the various longitudinally aggregated structured and unstructured data elements for each patient within the EHR can define the computational phenotypes of disease and care signatures and their association with outcomes. Although structured data elements, such as demographic characteristics, laboratory measurements, problem lists, and medications, are easily extracted, unstructured data are underused. The latter include free text in clinical narratives, documentation of procedures, and reports of imaging and pathology. Rapid scaling up of data storage and rapid innovation in natural language processing and computer vision can power insights from unstructured data streams. However, despite an array of opportunities for research using the EHR, specific expertise is necessary to adequately address confidentiality, accuracy, completeness, and heterogeneity challenges in EHR-based research. These often require methodological innovation and best practices to design and conduct successful research studies. Our review discusses these challenges and their proposed solutions. In addition, we highlight the ongoing innovations in federated learning in the EHR through a greater focus on common data models and discuss ongoing work that defines such an approach to large-scale, multicenter, federated studies. Such parallel improvements in technology and research methods enable innovative care and optimization of patient outcomes.

Combining meta-analysis with multiple imputation for one-step, privacy-protecting estimation of causal treatment effects in multi-site studies

Missing data complicates statistical analyses in multi-site studies, especially when it is not feasible to centrally pool individual-level data across sites. We combined meta-analysis with within-site multiple imputation for one-step estimation of the average causal effect (ACE) of a target population comprised of all individuals from all data-contributing sites within a multi-site distributed data network, without the need for sharing individual-level data to handle missing data. We considered two orders of combination and three choices of weights for meta-analysis, resulting in six approaches. The first three approaches, denoted as RR + metaF, RR + metaR and RR + std, first combined results from imputed data sets within each site using Rubin's rules and then meta-analyzed the combined results across sites using fixed-effect, random-effects and sample-standardization weights, respectively. The last three approaches, denoted as metaF + RR, metaR + RR and std + RR, first meta-analyzed results across sites separately for each imputation and then combined the meta-analysis results using Rubin's rules. Simulation results confirmed very good performance of RR + std and std + RR under various missing completely at random and missing at random settings. A direct application of the inverse-variance weighted meta-analysis based on site-specific ACEs can lead to biased results for the targeted network-wide ACE in the presence of treatment effect heterogeneity by site, demonstrating the need to clearly specify the target population and estimand and properly account for potential site heterogeneity in meta-analyses seeking to draw causal interpretations. An illustration using a large administrative claims database is presented.

DESIGN DIFFERENCES EXPLAIN VARIATION IN RESULTS BETWEEN RANDOMIZED TRIALS AND THEIR NON-RANDOMIZED EMULATIONS

Objectives

While randomized controlled trials (RCTs) are considered a standard for evidence on the efficacy of medical treatments, non-randomized real-world evidence (RWE) studies using data from health insurance claims or electronic health records can provide important complementary evidence. The use of RWE to inform decision-making has been questioned because of concerns regarding confounding in non-randomized studies and the use of secondary data. RCT-DUPLICATE was a demonstration project that emulated the design of 32 RCTs with non-randomized RWE studies. We sought to explore how emulation differences relate to variation in results between the RCT-RWE study pairs.

Methods

We include all RCT-RWE study pairs from RCT-DUPLICATE where the measure of effect was a hazard ratio and use exploratory meta-regression methods to explain differences and variation in the effect sizes between the results from the RCT and the RWE study. The considered explanatory variables are related to design and population differences.

Results

Most of the observed variation in effect estimates between RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: (i) in-hospital start of treatment (not observed in claims data), (ii) discontinuation of certain baseline therapies at randomization (not part of clinical practice), (iii) delayed onset of drug effects (missed by short medication persistence in clinical practice).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from RCTs and RWE studies can be attributed to design emulation differences. (238 words).

Treatment and care received by children hospitalized with COVID-19 in a large hospital network in the United States, February 2020 to September 2021

We described care received by hospitalized children with COVID-19 or multi-system inflammatory syndrome (MIS-C) prior to the 2021 COVID-19 Omicron variant surge in the US. We identified hospitalized children <18 years of age with a COVID-19 or MIS-C diagnosis (COVID-19 not required), separately, from February 2020-September 2021 (n = 126 hospitals). We described high-risk conditions, inpatient treatments, and complications among these groups. Among 383,083 pediatric hospitalizations, 2,186 had COVID-19 and 395 had MIS-C diagnosis. Less than 1% had both COVID-19 and MIS-C diagnosis (n = 154). Over half were >6 years old (54% COVID-19, 70% MIS-C). High-risk conditions included asthma (14% COVID-19, 11% MIS-C), and obesity (9% COVID-19, 10% MIS-C). Pulmonary complications in children with COVID-19 included viral pneumonia (24%) and acute respiratory failure (11%). In reference to children with COVID-19, those with MIS-C had more hematological disorders (62% vs 34%), sepsis (16% vs 6%), pericarditis (13% vs 2%), myocarditis (8% vs 1%). Few were ventilated or died, but some required oxygen support (38% COVID-19, 45% MIS-C) or intensive care (42% COVID-19, 69% MIS-C). Treatments included: methylprednisolone (34% COVID-19, 75% MIS-C), dexamethasone (25% COVID-19, 15% MIS-C), remdesivir (13% COVID-19, 5% MIS-C). Antibiotics (50% COVID-19, 68% MIS-C) and low-molecular weight heparin (17% COVID-19, 34% MIS-C) were frequently administered. Markers of illness severity among hospitalized children with COVID-19 prior to the 2021 Omicron surge are consistent with previous studies. We report important trends on treatments in hospitalized children with COVID-19 to improve the understanding of real-world treatment patterns in this population.

Use of the Electronic Health Record for Monitoring Adverse Drug Reactions

PURPOSE OF REVIEW

Adverse drug reactions (ADRs) are a significant cause of morbidity and mortality. The electronic health record (EHR) provides an opportunity to monitor ADRs, mainly through the utilization of drug allergy data and pharmacogenomics. This review article explores the current use of the EHR for ADR monitoring and highlights areas that require improvement.

RECENT FINDINGS

Recent research has identified several issues with using EHR for ADR monitoring. These include the lack of standardization between EHR systems, specificity in data entry options, incomplete and inaccurate documentation, and alert fatigue. These issues can limit the effectiveness of ADR monitoring and compromise patient safety. The EHR has great potential for monitoring ADR but needs significant updates to improve patient safety and optimize care. Future research should concentrate on developing standardized documentation and clinical decision support systems within EHRs. Healthcare professionals should also be educated on the significance of accurate and complete ADR monitoring.

Development and Verification of Time-Series Deep Learning for Drug-Induced Liver Injury Detection in Patients Taking Angiotensin II Receptor Blockers: A Multicenter Distributed Research Network Approach

OBJECTIVES

The objective of this study was to develop and validate a multicenter-based, multi-model, time-series deep learning model for predicting drug-induced liver injury (DILI) in patients taking angiotensin receptor blockers (ARBs). The study leveraged a national-level multicenter approach, utilizing electronic health records (EHRs) from six hospitals in Korea.

METHODS

A retrospective cohort analysis was conducted using EHRs from six hospitals in Korea, comprising a total of 10,852 patients whose data were converted to the Common Data Model. The study assessed the incidence rate of DILI among patients taking ARBs and compared it to a control group. Temporal patterns of important variables were analyzed using an interpretable timeseries model.

RESULTS

The overall incidence rate of DILI among patients taking ARBs was found to be 1.09%. The incidence rates varied for each specific ARB drug and institution, with valsartan having the highest rate (1.24%) and olmesartan having the lowest rate (0.83%). The DILI prediction models showed varying performance, measured by the average area under the receiver operating characteristic curve, with telmisartan (0.93), losartan (0.92), and irbesartan (0.90) exhibiting higher classification performance. The aggregated attention scores from the models highlighted the importance of variables such as hematocrit, albumin, prothrombin time, and lymphocytes in predicting DILI.

CONCLUSIONS

Implementing a multicenter-based timeseries classification model provided evidence that could be valuable to clinicians regarding temporal patterns associated with DILI in ARB users. This information supports informed decisions regarding appropriate drug use and treatment strategies.

Rewiring Drug Research and Development through Human Data-Driven Discovery (HD3)

In an era of unparalleled technical advancement, the pharmaceutical industry is struggling to transform data into increased research and development efficiency, and, as a corollary, new drugs for patients. Here, we briefly review some of the commonly discussed issues around this counterintuitive innovation crisis. Looking at both industry- and science-related factors, we posit that traditional preclinical research is front-loading the development pipeline with data and drug candidates that are unlikely to succeed in patients. Applying a first principles analysis, we highlight the critical culprits and provide suggestions as to how these issues can be rectified through the pursuit of a Human Data-driven Discovery (HD³) paradigm. Consistent with other examples of disruptive innovation, we propose that new levels of success are not dependent on new inventions, but rather on the strategic integration of existing data and technology assets. In support of these suggestions, we highlight the power of HD³, through recently published proof-of-concept applications in the areas of drug safety analysis and prediction, drug repositioning, the rational design of combination therapies and the global response to the COVID-19 pandemic. We conclude that innovators must play a key role in expediting the path to a largely human-focused, systems-based approach to drug discovery and research.

Assessment and Improvement of Drug Data Structuredness From Electronic Health Records: Algorithm Development and Validation

BACKGROUND

Digitization offers a multitude of opportunities to gain insights into current diagnostics and therapies from retrospective data. In this context, real-world data and their accessibility are of increasing importance to support unbiased and reliable research on big data. However, routinely collected data are not readily usable for research owing to the unstructured nature of health care systems and a lack of interoperability between these systems. This challenge is evident in drug data.

OBJECTIVE

This study aimed to present an approach that identifies and increases the structuredness of drug data while ensuring standardization according to Anatomical Therapeutic Chemical (ATC) classification.

METHODS

Our approach was based on available drug prescriptions and a drug catalog and consisted of 4 steps. First, we performed an initial analysis of the structuredness of local drug data to define a point of comparison for the effectiveness of the overall approach. Second, we applied 3 algorithms to unstructured data that translated text into ATC codes based on string comparisons in terms of ingredients and product names and performed similarity comparisons based on Levenshtein distance. Third, we validated the results of the 3 algorithms with expert knowledge based on the 1000 most frequently used prescription texts. Fourth, we performed a final validation to determine the increased degree of structuredness.

RESULTS

Initially, 47.73% (n=843,980) of 1,768,153 drug prescriptions were classified as structured. With the application of the 3 algorithms, we were able to increase the degree of structuredness to 85.18% (n=1,506,059) based on the 1000 most frequent medication prescriptions. In this regard, the combination of algorithms 1, 2, and 3 resulted in a correctness level of 100% (with 57,264 ATC codes identified), algorithms 1 and 3 resulted in 99.6% (with 152,404 codes identified), and algorithms 1 and 2 resulted in 95.9% (with 39,472 codes identified).

CONCLUSIONS

As shown in the first analysis steps of our approach, the availability of a product catalog to select during the documentation process is not sufficient to generate structured data. Our 4-step approach reduces the problems and reliably increases the structuredness automatically. Similarity matching shows promising results, particularly for entries with no connection to a product catalog. However, further enhancement of the correctness of such a similarity matching algorithm needs to be investigated in future work.

HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force

PROBLEM

Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias.

WHAT WE DID

The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The overarching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility.

STRATEGIES TO DISSEMINATE AND FACILITATE USE

Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years.

CONCLUSION

The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

A novel drug selection decision support model based on real-world medical data by the hybrid entropic weight TOPSIS method

BACKGROUND

The medicine selection method is a critical and challenging issue in medical insurance decision-making.

OBJECTIVES

This study proposed a real-world data-based multi-criteria decision analysis (MCDA) model with a hybrid entropic weight Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) algorithms to select satisfactory drugs.

METHODS

The evaluation index includes two levels: primary criteria and sub-criteria. Firstly, we proposed six primary criteria to form the value health framework. The primary criteria's weights were derived from the policymakers' questionnaire. Meanwhile, clinically relevant sub-criteria were derived from high-quality (screened by GRADE scores) clinical-research literature. Their weights are determined by the entropy weight (EW) algorithm. Secondly, we split the primary criteria into six mini-EW-TOPSIS models. Then, we obtained six ideal closeness degree scores (ICDS) for each candidate drug. Thirdly, we get the total utility score by linear weighting the ICDS. The higher the utility score, the higher the ranking.

RESULTS

A national multicenter real-world case study of the ranking of four generic antibiotics validated the proposed model. This model is verified by comparative experiments and sensitivity analysis. The whole ranking model was consistent and reliable. Based on these results, medical policymakers can intuitively and easily understand the characteristics of each drug to facilitate follow-up drug policy-making.

CONCLUSION

The ranking algorithm combines the objective characteristics of medicine and policy makers' opinions, which can improve the applicability of the results. This model can help decision-makers, clinicians, and related researchers better understand the drug assessment process.

A comparative study of antihypertensive drugs prediction models for the elderly based on machine learning algorithms

Background

Globally, blood pressure management strategies were ineffective, and a low percentage of patients receiving hypertension treatment had their blood pressure controlled. In this study, we aimed to build a medication prediction model by correlating patient attributes with medications to help physicians quickly and rationally match appropriate medications.

Methods

We collected clinical data from elderly hypertensive patients during hospitalization and combined statistical methods and machine learning (ML) algorithms to filter out typical indicators. We constructed five ML models to evaluate all datasets using 5-fold cross-validation. Include random forest (RF), support vector machine (SVM), light gradient boosting machine (LightGBM), artificial neural network (ANN), and naive Bayes (NB) models. And the performance of the models was evaluated using the micro-F1 score.

Results

Our experiments showed that by statistical methods and ML algorithms for feature selection, we finally selected Age, SBP, DBP, Lymph, RBC, HCT, MCHC, PLT, AST, TBIL, Cr, UA, Urea, K, Na, Ga, TP, GLU, TC, TG, γ-GT, Gender, HTN CAD, and RI as feature metrics of the models. LightGBM had the best prediction performance with the micro-F1 of 78.45%, which was higher than the other four models.

Conclusion

LightGBM model has good results in predicting antihypertensive medication regimens, and the model can be beneficial in improving the personalization of hypertension treatment.

The US Food and Drug Administration Sentinel System: a national resource for a learning health system

The US Food and Drug Administration (FDA) created the Sentinel System in response to a requirement in the FDA Amendments Act of 2007 that the agency establish a system for monitoring risks associated with drug and biologic products using data from disparate sources. The Sentinel System has completed hundreds of analyses, including many that have directly informed regulatory decisions. The Sentinel System also was designed to support a national infrastructure for a learning health system. Sentinel governance and guiding principles were designed to facilitate Sentinel's role as a national resource. The Sentinel System infrastructure now supports multiple non-FDA projects for stakeholders ranging from regulated industry to other federal agencies, international regulators, and academics. The Sentinel System is a working example of a learning health system that is expanding with the potential to create a global learning health system that can support medical product safety assessments and other research.

Understanding variation in the results of real-world evidence studies that seem to address the same question

OBJECTIVES

Multiple database studies on the same question, conducted by different investigators using different approaches or different data sources, can be considered sensitivity analyses for the same causal treatment effect question. We evaluated the contribution of alternative study design parameters and analysis choices to variation in estimates of the risk of major bleeding with dabigatran compared with warfarin.

STUDY DESIGN AND SETTING

We followed a 7-step process: (1) identify published studies asking the same question, (2) independently reproduce selected studies in the same data sources as the original authors, (3) contact original authors, (4) evaluate validity, (5) document critical study parameter specifications, (6) implement a designed matrix of variations in study parameters based on the original studies, and (7) evaluate contributors to variation in results.

RESULTS

Most variation remained unexplained (60-88%). Of the explained variation, two-thirds were related to data and population differences, and one-third were related to the use of alternative study design and analysis parameters. Among these, the most prominent were differences in outcome algorithms and criteria used to define follow-up.

CONCLUSION

When making policy decisions based on database study findings, it is important to evaluate the validity, consistency, and robustness of results to alternative design and analysis decisions.

HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force

OBJECTIVES

Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias.

METHODS

The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The over-arching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility.

STRATEGIES TO DISSEMINATE AND FACILITATE USE

Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years.

CONCLUSION

The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Developing a common data model approach for DISCOVER CKD: A retrospective, global cohort of real-world patients with chronic kidney disease

Objectives

To describe a flexible common data model (CDM) approach that can be efficiently tailored to study-specific needs to facilitate pooled patient-level analysis and aggregated/meta-analysis of routinely collected retrospective patient data from disparate data sources; and to detail the application of this CDM approach to the DISCOVER CKD retrospective cohort, a longitudinal database of routinely collected (secondary) patient data of individuals with chronic kidney disease (CKD).

Methods

The flexible CDM approach incorporated three independent, exchangeable components that preceded data mapping and data model implementation: (1) standardized code lists (unifying medical events from different coding systems); (2) laboratory unit harmonization tables; and (3) base cohort definitions. Events between different coding vocabularies were not mapped code-to-code; for each data source, code lists of labels were curated at the entity/event level. A study team of epidemiologists, clinicians, informaticists, and data scientists were included within the validation of each component.

Results

Applying the CDM to the DISCOVER CKD retrospective cohort, secondary data from 1,857,593 patients with CKD were harmonized from five data sources, across three countries, into a discrete database for rapid real-world evidence generation.

Conclusions

This flexible CDM approach facilitates evidence generation from real-world data within the DISCOVER CKD retrospective cohort, providing novel insights into the epidemiology of CKD that may expedite improvements in diagnosis, prognosis, early intervention, and disease management. The adaptable architecture of this CDM approach ensures scalable, fast, and efficient application within other therapy areas to facilitate the combined analysis of different types of secondary data from multiple, heterogeneous sources.

Patient characteristics, pain treatment patterns, and incidence of total joint replacement in a US population with osteoarthritis

BACKGROUND

Currently available medications for chronic osteoarthritis pain are only moderately effective, and their use is limited in many patients because of serious adverse effects and contraindications. The primary surgical option for osteoarthritis is total joint replacement (TJR). The objectives of this study were to describe the treatment history of patients with osteoarthritis receiving prescription pain medications and/or intra-articular corticosteroid injections, and to estimate the incidence of TJR in these patients.

METHODS

This retrospective, multicenter, cohort study utilized health plan administrative claims data (January 1, 2013, through December 31, 2019) of adult patients with osteoarthritis in the Innovation in Medical Evidence Development and Surveillance Distributed Database, a subset of the US FDA Sentinel Distributed Database. Patients were analyzed in two cohorts: those with prevalent use of "any pain medication" (prescription non-steroidal anti-inflammatory drugs [NSAIDs], opioids, and/or intra-articular corticosteroid injections) using only the first qualifying dispensing (index date); and those with prevalent use of "each specific pain medication class" with all qualifying treatment episodes identified.

RESULTS

Among 1 992 670 prevalent users of "any pain medication", pain medications prescribed on the index date were NSAIDs (596 624 [29.9%] patients), opioids (1 161 806 [58.3%]), and intra-articular corticosteroids (323 459 [16.2%]). Further, 92 026 patients received multiple pain medications on the index date, including 71 632 (3.6%) receiving both NSAIDs and opioids. Altogether, 20.6% of patients used an NSAID at any time following an opioid index dispensing and 17.2% used an opioid following an NSAID index dispensing. The TJR incidence rates per 100 person-years (95% confidence interval [CI]) were 3.21 (95% CI: 3.20-3.23) in the "any pain medication" user cohort, and among those receiving "each specific pain medication class" were NSAIDs, 4.63 (95% CI: 4.58-4.67); opioids, 7.45 (95% CI: 7.40-7.49); and intra-articular corticosteroids, 8.05 (95% CI: 7.97-8.13).

CONCLUSIONS

In patients treated with prescription medications for osteoarthritis pain, opioids were more commonly prescribed at index than NSAIDs and intra-articular corticosteroid injections. Of the pain medication classes examined, the incidence of TJR was highest in patients receiving intra-articular corticosteroids and lowest in patients receiving NSAIDs.

Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions

Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson’s correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratio_original/hazard ratio_reproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.

Analyses of real-world evidence from digital clinical practice data provide important insights for healthcare decision makers. Here, authors test reproducibility of 150 peer-reviewed studies, reporting strong reproducibility, which could be further improved through more complete reporting in future original studies

The IeDEA harmonist data toolkit: A data quality and data sharing solution for a global HIV research consortium

We describe the design, implementation, and impact of a data harmonization, data quality checking, and dynamic report generation application in an international observational HIV research network. The IeDEA Harmonist Data Toolkit is a web-based application written in the open source programming language R, employs the R/Shiny and RMarkdown packages, and leverages the REDCap data collection platform for data model definition and user authentication. The Toolkit performs data quality checks on uploaded datasets, checks for conformance with the network's common data model, displays the results both interactively and in downloadable reports, and stores approved datasets in secure cloud storage for retrieval by the requesting investigator. Including stakeholders and users in the design process was key to the successful adoption of the application. A survey of regional data managers as well as initial usage metrics indicate that the Toolkit saves time and results in improved data quality, with a 61% mean reduction in the number of error records in a dataset. The generalized application design allows the Toolkit to be easily adapted to other research networks.

Distributed Quasi-Poisson regression algorithm for modeling multi-site count outcomes in distributed data networks

BACKGROUND

Observational studies incorporating real-world data from multiple institutions facilitate study of rare outcomes or exposures and improve generalizability of results. Due to privacy concerns surrounding patient-level data sharing across institutions, methods for performing regression analyses distributively are desirable. Meta-analysis of institution-specific estimates is commonly used, but has been shown to produce biased estimates in certain settings. While distributed regression methods are increasingly available, methods for analyzing count outcomes are currently limited. Count data in practice are commonly subject to overdispersion, exhibiting greater variability than expected under a given statistical model.

OBJECTIVE

We propose a novel computational method, a one-shot distributed algorithm for quasi-Poisson regression (ODAP), to distributively model count outcomes while accounting for overdispersion.

METHODS

ODAP incorporates a surrogate likelihood approach to perform distributed quasi-Poisson regression without requiring patient-level data sharing, only requiring sharing of aggregate data from each participating institution. ODAP requires at most three rounds of non-iterative communication among institutions to generate coefficient estimates and corresponding standard errors. In simulations, we evaluate ODAP under several data scenarios possible in multi-site analyses, comparing ODAP and meta-analysis estimates in terms of error relative to pooled regression estimates, considered the gold standard. In a proof-of-concept real-world data analysis, we similarly compare ODAP and meta-analysis in terms of relative error to pooled estimatation using data from the OneFlorida Clinical Research Consortium, modeling length of stay in COVID-19 patients as a function of various patient characteristics. In a second proof-of-concept analysis, using the same outcome and covariates, we incorporate data from the UnitedHealth Group Clinical Discovery Database together with the OneFlorida data in a distributed analysis to compare estimates produced by ODAP and meta-analysis.

RESULTS

In simulations, ODAP exhibited negligible error relative to pooled regression estimates across all settings explored. Meta-analysis estimates, while largely unbiased, were increasingly variable as heterogeneity in the outcome increased across institutions. When baseline expected count was 0.2, relative error for meta-analysis was above 5% in 25% of iterations (250/1000), while the largest relative error for ODAP in any iteration was 3.59%. In our proof-of-concept analysis using only OneFlorida data, ODAP estimates were closer to pooled regression estimates than those produced by meta-analysis for all 15 covariates. In our distributed analysis incorporating data from both OneFlorida and the UnitedHealth Group Clinical Discovery Database, ODAP and meta-analysis estimates were largely similar, while some differences in estimates (as large as 13.8%) could be indicative of bias in meta-analytic estimates.

CONCLUSIONS

ODAP performs privacy-preserving, communication-efficient distributed quasi-Poisson regression to analyze count outcomes using data stored within multiple institutions. Our method produces estimates nearly matching pooled regression estimates and sometimes more accurate than meta-analysis estimates, most notably in settings with relatively low counts and high outcome heterogeneity across institutions.

Advancing drug safety science by integrating molecular knowledge with post-marketing adverse event reports

Promising drug development efforts may frequently fail due to unintended adverse reactions. Several methods have been developed to analyze such data, aiming to improve pharmacovigilance and drug safety. In this work, we provide a brief review of key directions to quantitatively analyzing adverse events and explore the potential of augmenting these methods using additional molecular data descriptors. We argue that molecular expansion of adverse event data may provide a path to improving the insights gained through more traditional pharmacovigilance approaches. Examples include the ability to assess statistical relevance with respect to underlying biomolecular mechanisms, the ability to generate plausible causative hypotheses and/or confirmation where possible, the ability to computationally study potential clinical trial designs and/or results, as well as the further provision of advanced features incorporated in innovative methods, such as machine learning. In summary, molecular data expansion provides an elegant way to extend mechanistic modeling, systems pharmacology, and patient‐centered approaches for the assessment of drug safety. We anticipate that such advances in real‐world data informatics and outcome analytics will help to better inform public health, via the improved ability to prospectively understand and predict various types of drug‐induced molecular perturbations and adverse events.

A Framework for Visualizing Study Designs and Data Observability in Electronic Health Record Data

Background

There is growing interest in using evidence generated from clinical practice data to support regulatory, coverage and other healthcare decision-making. A graphical framework for depicting longitudinal study designs to mitigate this barrier was introduced and has found wide acceptance. We sought to enhance the framework to contain information that helps readers assess the appropriateness of the source data in which the study design was applied.

Methods

For the enhanced graphical framework, we added a simple visualization of data type and observability to capture differences between electronic health record (EHR) and other registry data that may have limited data continuity and insurance claims data that have enrollment files.

Results

We illustrate the revised graphical framework with 2 example studies conducted using different data sources, including administrative claims only, EHR only, linked claims and EHR, as well as specialty community based EHRs with and without external linkages.

Conclusion

The enhanced visualization framework is important because evaluation of study validity needs to consider the triad of study question, design, and data together. Any given data source or study design may be appropriate for some questions but not others.

eTRANSAFE: Building a sustainable framework to share reproducible drug safety knowledge with the public domain

Integrative drug safety research in translational health informatics has rapidly evolved and included data that are drawn in from many resources, combining diverse data that are either reused from (curated) repositories, or newly generated at source. Each resource is mandated by different sets of metadata rules that are imposed on the incoming data. Combination of the data cannot be readily achieved without interference of data stewardship and the top-down policy guidelines that supervise and inform the process for data combination to aid meaningful interpretation and analysis of such data.

The eTRANSAFE Consortium's effort to drive integrative drug safety research at a large scale hereby present the lessons learnt and the proposal of solution at the guidelines in practice at this Innovative Medicines Initiative (IMI) project. Recommendations in these guidelines were compiled from feedback received from key stakeholders in regulatory agencies, EFPIA companies, and academic partners. The research reproducibility guidelines presented in this study lay the foundation for a comprehensive data sharing and knowledge management plans accounting for research data management in the drug safety space - FAIR data sharing guidelines, and the model verification guidelines as generic deliverables that best practices that can be reused by other scientific community members at large.

FAIR data sharing is a dynamic landscape that rapidly evolves with fast-paced technology advancements. The research reproducibility in drug safety guidelines introduced in this study provides a reusable framework that can be adopted by other research communities that aim to integrate public and private data in biomedical research space.

Using inpatient electronic medical records to study influenza for pandemic preparedness

BACKGROUND

We assessed the ability to identify key data relevant to influenza and other respiratory virus surveillance in a large-scale US-based hospital electronic medical record (EMR) dataset using seasonal influenza as a use case. We describe characteristics and outcomes of hospitalized influenza cases across three seasons.

METHODS

We identified patients with an influenza diagnosis between March 2017 and March 2020 in 140 US hospitals as part of the US FDA's Sentinel System. We calculated descriptive statistics on the presence of high-risk conditions, influenza antiviral administrations, and severity endpoints.

RESULTS

Among 5.1 million hospitalizations, we identified 29,520 hospitalizations with an influenza diagnosis; 64% were treated with an influenza antiviral within 2 days of admission, and 25% were treated >2 days after admission. Patients treated >2 days after admission had more comorbidities than patients treated within 2 days of admission. Patients never treated during hospitalization had more documentation of cardiovascular and other diseases than treated patients. We observed more severe endpoints in patients never treated (death = 3%, mechanical ventilation [MV] = 9%, intensive care unit [ICU] = 26%) or patients treated >2 days after admission (death = 2%, MV = 14%, ICU = 32%) than in patients treated earlier (treated on admission: death = 1%, MV = 5%, ICU = 23%, treated within 2 days of admission: death = 1%, MV = 7%, ICU = 27%).

CONCLUSIONS

We identified important trends in influenza severity related to treatment timing in a large inpatient dataset, laying the groundwork for the use of this and other inpatient EMR data for influenza and other respiratory virus surveillance.

Harms in Systematic Reviews Paper 1: An introduction to research on harms

OBJECTIVE

Most systematic reviews of interventions focus on potential benefits. Common methods and assumptions that are appropriate for assessing benefits can be inappropriate for harms. This paper provides a primer on researching harms, particularly in systematic reviews.

STUDY DESIGN AND SETTING

Commentary describing challenges with assessing harm.

RESULTS

Investigators should be familiar with various terminologies used to describe, classify, and group harms. Published reports of clinical trials include limited information about harms, so systematic reviewers should not depend on these studies and journal articles to reach conclusions about harms. Visualizations might improve communication of multiple dimensions of harms such as severity, relatedness, and timing.

CONCLUSION

The terminology, classification, detection, collection, and reporting of harms create unique challenges that take time, expertise, and resources to navigate in both primary studies and evidence syntheses. Systematic reviewers might reach incorrect conclusions if they focus on evidence about harms found in published reports of randomized trials of a particular health problem. Systematic reviews could be improved through better identification and reporting of harms in primary studies and through better training and uptake of appropriate methods for synthesizing evidence about harms.

W. Emerging technologies and their impact on regulatory science

There is an evolution and increasing need for the utilization of emerging cellular, molecular and in silico technologies and novel approaches for safety assessment of food, drugs, and personal care products. Convergence of these emerging technologies is also enabling rapid advances and approaches that may impact regulatory decisions and approvals. Although the development of emerging technologies may allow rapid advances in regulatory decision making, there is concern that these new technologies have not been thoroughly evaluated to determine if they are ready for regulatory application, singularly or in combinations. The magnitude of these combined technical advances may outpace the ability to assess fit for purpose and to allow routine application of these new methods for regulatory purposes. There is a need to develop strategies to evaluate the new technologies to determine which ones are ready for regulatory use. The opportunity to apply these potentially faster, more accurate, and cost-effective approaches remains an important goal to facilitate their incorporation into regulatory use. However, without a clear strategy to evaluate emerging technologies rapidly and appropriately, the value of these efforts may go unrecognized or may take longer. It is important for the regulatory science field to keep up with the research in these technically advanced areas and to understand the science behind these new approaches. The regulatory field must understand the critical quality attributes of these novel approaches and learn from each other's experience so that workforces can be trained to prepare for emerging global regulatory challenges. Moreover, it is essential that the regulatory community must work with the technology developers to harness collective capabilities towards developing a strategy for evaluation of these new and novel assessment tools.

Applying Machine Learning in Distributed Data Networks for Pharmacoepidemiologic and Pharmacovigilance Studies: Opportunities, Challenges, and Considerations

Increasing availability of electronic health databases capturing real-world experiences with medical products has garnered much interest in their use for pharmacoepidemiologic and pharmacovigilance studies. The traditional practice of having numerous groups use single databases to accomplish similar tasks and address common questions about medical products can be made more efficient through well-coordinated multi-database studies, greatly facilitated through distributed data network (DDN) architectures. Access to larger amounts of electronic health data within DDNs has created a growing interest in using data-adaptive machine learning (ML) techniques that can automatically model complex associations in high-dimensional data with minimal human guidance. However, the siloed storage and diverse nature of the databases in DDNs create unique challenges for using ML. In this paper, we discuss opportunities, challenges, and considerations for applying ML in DDNs for pharmacoepidemiologic and pharmacovigilance studies. We first discuss major types of activities performed by DDNs and how ML may be used. Next, we discuss practical data-related factors influencing how DDNs work in practice. We then combine these discussions and jointly consider how opportunities for ML are affected by practical data-related factors for DDNs, leading to several challenges. We present different approaches for addressing these challenges and highlight efforts that real-world DDNs have taken or are currently taking to help mitigate them. Despite these challenges, the time is ripe for the emerging interest to use ML in DDNs, and the utility of these data-adaptive modeling techniques in pharmacoepidemiologic and pharmacovigilance studies will likely continue to increase in the coming years.

Broadening the reach of the FDA Sentinel system: A roadmap for integrating electronic health record data in a causal analysis framework

The Sentinel System is a major component of the United States Food and Drug Administration's (FDA) approach to active medical product safety surveillance. While Sentinel has historically relied on large quantities of health insurance claims data, leveraging longitudinal electronic health records (EHRs) that contain more detailed clinical information, as structured and unstructured features, may address some of the current gaps in capabilities. We identify key challenges when using EHR data to investigate medical product safety in a scalable and accelerated way, outline potential solutions, and describe the Sentinel Innovation Center's initiatives to put solutions into practice by expanding and strengthening the existing system with a query-ready, large-scale data infrastructure of linked EHR and claims data. We describe our initiatives in four strategic priority areas: (1) data infrastructure, (2) feature engineering, (3) causal inference, and (4) detection analytics, with the goal of incorporating emerging data science innovations to maximize the utility of EHR data for medical product safety surveillance.

Validation of diagnosis codes to identify hospitalized COVID-19 patients in health care claims data

PURPOSE

Health plan claims may provide complete longitudinal data for timely, real-world population-level COVID-19 assessment. However, these data often lack laboratory results, the standard for COVID-19 diagnosis.

METHODS

We assessed the validity of ICD-10-CM diagnosis codes for identifying patients hospitalized with COVID-19 in U.S. claims databases, compared to linked laboratory results, among six Food and Drug Administration Sentinel System data partners (two large national insurers, four integrated delivery systems) from February 20-October 17, 2020. We identified patients hospitalized with COVID-19 according to five ICD-10-CM diagnosis code-based algorithms, which included combinations of codes U07.1, B97.29, general coronavirus codes, and diagnosis codes for severe symptoms. We calculated the positive predictive value (PPV) and sensitivity of each algorithm relative to laboratory test results. We stratified results by data source type and across three time periods: February 20-March 31 (Time A), April 1-30 (Time B), May 1-October 17 (Time C).

RESULTS

The five algorithms identified between 34 806 and 47 293 patients across the study periods; 23% with known laboratory results contributed to PPV calculations. PPVs were high and similar across algorithms. PPV of U07.1 alone was stable around 93% for integrated delivery systems, but declined over time from 93% to 70% among national insurers. Overall PPV of U07.1 across all data partners was 94.1% (95% CI, 92.3%-95.5%) in Time A and 81.2% (95% CI, 80.1%-82.2%) in Time C. Sensitivity was consistent across algorithms and over time, at 94.9% (95% CI, 94.2%-95.5%).

CONCLUSION

Our results support the use of code U07.1 to identify hospitalized COVID-19 patients in U.S. claims data.

Development and evaluation of standardized pregnancy identification and trimester distribution algorithms in U.S. IBM MarketScan® Commercial and Medicaid data

OBJECTIVES

Creation of new algorithms to identify pregnancies in automated health care claims databases is of public health importance, as it allows us to learn more about medication use and safety in a vulnerable population. Previous algorithms were largely created using international classification of disease codes, but despite the U.S. code transition in 2015, few algorithms are available with the latest ICD-10-CM codes.

METHODS

Using U.S. IBM MarketScan® Commercial Claims and Encounters and Multi-State Medicaid databases for women aged 10-64 years during 2014 and 2016, two pregnancy algorithms (ICD-9-CM and ICD-10-CM) were created using expert clinical review. The algorithms were evaluated by assessing the distribution of pregnancy outcomes (live birth and pregnancy losses) within each time-based cohort and the ability of the algorithms to identify select medication use during pregnancy. Medication exposure, demographics, comorbidities, and pregnancy outcomes were compared to published literature estimates for the two time periods.

RESULTS

For the IBM MarketScan® Commercial database, the algorithms identified 687,228 pregnancies in 2014 and 444,293 in 2016. In the IBM MarketScan® Medicaid database, 389,132 pregnancies in 2014 and 406,418 in 2016 were identified. Percentages of most pregnancy outcomes identified using the algorithms were similar to national data sources; however, percentages of preterm births and pregnancy losses were not comparable. Most medication use estimates from the algorithms were similar to or higher than published estimates.

CONCLUSIONS

By incorporating the latest ICD-10-CM codes, the new algorithms provide more complete estimates of medication use during pregnancy than algorithms using the outdated codes.

Advancing the Regulation of Traditional and Complementary Medicine Products: A Comparison of Five Regulatory Systems on Traditional Medicines with a Long History of Use

Background

An appropriate regulatory system to ensure and promote the quality, safety, and efficacy of the products of traditional medicine (TM) and complementary medicine (CM) is critical to not only public health but also economic growth. The regulatory approach and evaluation standards for TM/CM products featured with a long history of use are yet to be developed. This study aims to investigate and compare the existing regulatory approaches for TM/CM products with a long history of use.

Method

A mixed approach of documentary analysis involving official and legal documents from official websites, as well as a scoping review of scholarly work in scientific databases about regulatory systems of TM/CM products in China, Hong Kong, Taiwan, Japan, and Korea, was employed in this study and used for comparison.

Results

For registration purposes, all five regulatory systems recognized the history of use as part of the totality of evidence when evaluating the safety and efficacy of TM/CM products with a long history of use. Generally, the list of classic formulas is predefined and bound to the formulas recommended in the prescribed list of ancient medical textbooks. Expedited pathways are usually in place and scientific data of nonclinical and clinical studies may be exempted. At the same time, additional restrictions with the scope of products constitute a comprehensive approach in the regulation. Quality assurance and postmarketing safety surveillance were found to be the major focus across the regulatory schemes investigated in this study.

Conclusion

The regulatory systems investigated in this study allow less stringent registration requirements for TM/CM products featured with a long history of use, assuming safety and efficacy to be plausible based on historic use. Considering the safety and efficacy of these products, regulatory standards should emphasize the technical requirements for quality control and postmarket surveillance.

The Role of Real-World Evidence in FDA-Approved New Drug and Biologics License Applications

The US Food and Drug Administration (FDA) is open to accepting real‐world evidence (RWE) to support its assessment of medical products. However, RWE stakeholders lack a shared understanding of FDA’s evidentiary expectations for the use of RWE in applications for new drugs and biologics. We conducted a systematic review of publicly available FDA approval documents from January 2019 to June 2021. We sought to quantify, by year, how many approvals incorporated RWE in any form, and the intended use of RWE in those applications. Among approvals with RWE intended to support safety and/or effectiveness, we classified whether and how those studies impacted FDA’s benefit‐risk considerations, whether those studies were incorporated into the product label, and the therapeutic area of the medical product. Finally, we qualified FDA’s documented feedback where available. We found that 116 approvals incorporated RWE in any form, with the proportion of approvals incorporating RWE increasing each year. Of these approvals, 88 included an RWE study intended to provide evidence of safety or effectiveness. Among these 88 approvals, 65 of the studies influenced FDA’s final decision and 38 were included in product labels. The 88 approvals spanned 18 therapeutic areas. FDA’s feedback on RWE study quality included methodological issues, sample size concerns, omission of patient level data, and other limitations. Based on these findings, we would anticipate that future guidance on FDA’s evidentiary expectations of RWE use will incorporate fit‐for‐purpose real‐world data selection and careful attention to study design and analysis.

Pharmacosurveillance in Juvenile Idiopathic Arthritis

The advent of biologic disease-modifying antirheumatic drugs targeting specific cytokines or cell-cell interactions has dramatically changed the outlook of patients with juvenile idiopathic arthritis. However, safety concerns remain around the use of therapeutic agents for children with juvenile idiopathic arthritis. Foremost among these are the risks of serious infections and malignancy. This article provides an overview of methodologies for pharmacosurveillance in juvenile idiopathic arthritis, including spontaneous reporting systems and the use of diverse data sources, such as electronic health records, administrative claims, and clinical registries. The risks of infections and malignancies are then briefly reviewed.

Who gets treated for influenza: A surveillance study from the US Food and Drug Administration's Sentinel System

OBJECTIVE

We describe the baseline characteristics and complications of individuals with influenza in the US FDA's Sentinel System by antiviral treatment timing.

DESIGN

Retrospective cohort design.

PATIENTS

Individuals aged ≥6 months with outpatient diagnoses of influenza in June 2014-July 2017, 3 influenza seasons.

METHODS

We identified the comorbidities, vaccination history, influenza testing, and outpatient antiviral dispensings of individuals with influenza using administrative claims data from 13 data partners including the Centers for Medicare and Medicaid Services, integrated delivery systems, and commercial health plans. We assessed complications within 30 days: hospitalization, oxygen use, mechanical ventilation, critical care, ECMO, and death.

RESULTS

There were 1,090,333 influenza diagnoses in 2014-2015; 1,005,240 in 2016-2017; and 578,548 in 2017-2018. Between 49% and 55% of patients were dispensed outpatient treatment within 5 days. In all periods >80% of treated individuals received treatment on the day of diagnosis. Those treated on days 1-5 after diagnosis had higher prevalences of diabetes, chronic obstructive pulmonary disease, asthma, and obesity compared to those treated on the day of diagnosis or not treated at all. They also had higher rates of hospitalization, oxygen use, and critical care. In 2014-2015, among those aged ≥65 years, the rates of hospitalization were 45 per 1,000 diagnoses among those treated on day 0; 74 per 1,000 among those treated on days 1-5; and 50 per 1,000 among those who were untreated.

CONCLUSIONS

In a large, national analysis, approximately half of people diagnosed with influenza in the outpatient setting were treated with antiviral medications. Delays in outpatient dispensed treatment were associated with higher prevalence of comorbidities and higher rates of complication.

Random control selection for conducting high-throughput adverse drug events screening using large-scale longitudinal health data

Case-control design based high-throughput pharmacoinformatics study using large-scale longitudinal health data is able to detect new adverse drug event (ADEs) signals. Existing control selection approaches for case-control design included the dynamic/super control selection approach. The dynamic/super control selection approach requires all individuals to be evaluated at all ADE case index dates, as the individuals' eligibilities as control depend on ADE/enrollment history. Thus, using large-scale longitudinal health data, the dynamic/super control selection approach requires extraordinarily high computational time. We proposed a random control selection approach in which ADE case index dates were matched by randomly generated control index dates. The random control selection approach does not depend on ADE/enrollment history. It is able to significantly reduce computational time to prepare case-control data sets, as it requires all individuals to be evaluated only once. We compared the performance metrics of all control selection approaches using two large-scale longitudinal health data and a drug-ADE gold standard including 399 drug-ADE pairs. The F-scores for the random control selection approach were between 0.586 and 0.600 compared to between 0.545 and 0.562 for dynamic/super control selection approaches. The random control selection approach was ~ 1000 times faster than dynamic/super control selection approach on preparing case-control data sets. With large-scale longitudinal health data, a case-control design-based pharmacoinformatics study using random control selection is able to generate comparable ADE signals than the existing control selection approaches. The random control selection approach also significantly reduces computational time to prepare the case-control data sets.