Surveillance systems and methods for monitoring the post-marketing safety of influenza vaccines at the Centers for Disease Control and Prevention

Enhancing Relation Extraction for COVID-19 Vaccine Shot-Adverse Event Associations with Large Language Models

Objective The rapid evolution of the COVID-19 virus has led to the development of different vaccine shots, each designed to combat specific variants and enhance overall efficacy. While vaccines have been crucial in controlling the spread of the virus, they can also cause adverse events (AEs). Understanding these relationships is vital for vaccine safety monitoring and surveillance. Methods In our study, we collected data from the Vaccine Adverse Event Reporting System (VAERS) and social media platforms (Twitter and Reddit) to extract relationships between COVID-19 vaccine shots and adverse events. The dataset comprised 771 relation pairs, enabling a comprehensive analysis of adverse event patterns. We employed state-of-the-art GPT models, including GPT-3.5 and GPT-4, alongside traditional models such as Recurrent Neural Networks (RNNs) and BioBERT, to extract these relationships. Additionally, we used two sets of post-processing rules to further refine the extracted relations. Evaluation metrics including precision, recall, and F1-score were used to assess the performance of our models in extracting these relationships accurately. Results The most commonly reported AEs following the primary series of COVID-19 vaccines include arm soreness, fatigue, and headache, while the spectrum of AEs following boosters is more diverse. In relation extraction, fine-tuned GPT-3.5 with Sentence-based Relation Identification achieved the highest precision of 0.94 and a perfect recall of 1, resulting in an impressive F1 score of 0.97. Conclusion This study advances biomedical informatics by showing how large language models and deep learning models can extract relationships between vaccine shots and adverse events from VAERS and social media. These findings improve vaccine safety monitoring and clinical practice by enhancing our understanding of post-vaccination symptoms. The study sets a precedent for future research in natural language processing and biomedical informatics, with potential applications in pharmacovigilance and clinical decision-making.

Improving entity recognition using ensembles of deep learning and fine-tuned large language models: A case study on adverse event extraction from VAERS and social media

OBJECTIVE

Adverse event (AE) extraction following COVID-19 vaccines from text data is crucial for monitoring and analyzing the safety profiles of immunizations, identifying potential risks and ensuring the safe use of these products. Traditional deep learning models are adept at learning intricate feature representations and dependencies in sequential data, but often require extensive labeled data. In contrast, large language models (LLMs) excel in understanding contextual information, but exhibit unstable performance on named entity recognition (NER) tasks, possibly due to their broad but unspecific training. This study aims to evaluate the effectiveness of LLMs and traditional deep learning models in AE extraction, and to assess the impact of ensembling these models on performance.

METHODS

In this study, we utilized reports and posts from the Vaccine Adverse Event Reporting System (VAERS) (n = 230), Twitter (n = 3,383), and Reddit (n = 49) as our corpora. Our goal was to extract three types of entities: vaccine, shot, and adverse event (ae). We explored and fine-tuned (except GPT-4) multiple LLMs, including GPT-2, GPT-3.5, GPT-4, Llama-2 7b, and Llama-2 13b, as well as traditional deep learning models like Recurrent neural network (RNN) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT). To enhance performance, we created ensembles of the three models with the best performance. For evaluation, we used strict and relaxed F1 scores to evaluate the performance for each entity type, and micro-average F1 was used to assess the overall performance.

RESULTS

The ensemble demonstrated the best performance in identifying the entities "vaccine," "shot," and "ae," achieving strict F1-scores of 0.878, 0.930, and 0.925, respectively, and a micro-average score of 0.903. These results underscore the significance of fine-tuning models for specific tasks and demonstrate the effectiveness of ensemble methods in enhancing performance.

CONCLUSION

In conclusion, this study demonstrates the effectiveness and robustness of ensembling fine-tuned traditional deep learning models and LLMs, for extracting AE-related information following COVID-19 vaccination. This study contributes to the advancement of natural language processing in the biomedical domain, providing valuable insights into improving AE extraction from text data for pharmacovigilance and public health surveillance.

New exploration of signal detection of Regional Risks from the perspective of data mining: a pharmacovigilance analysis based on spontaneous reporting data in Zhenjiang, China

BACKGROUND

This study aimed to adopt the conventional signal detection methods to explore a new way of risk identification and to mine important drug risks from the perspective of big data based on Zhenjiang Adverse Event Reporting System (ZAERS).

RESEARCH DESIGN AND METHODS

Data were extracted from ZAERS database between 2012 and 2022. The risks of all the reported drug event combinations were identified at the preferred term level and the standardized MedDRA query level using disproportionality analysis. Then, we conducted signal assessment according to the descriptions of drug labels.

RESULTS

In total 41,473 ADE were reported and there were 12 risky signals. Signal assessment indicates the suspected causal associations in clindamycin-taste and smell disorders, valsartan-hepatic enzyme increased and valsartan-edema peripheral; the specific manifestations of allergic reactions triggered by clindamycin, cefotaxime, cefazodime, ShexiangZhuanggu plaster, ShexiangZhuifeng plaster, and Yanhuning need to be refined in drug labels. In addition, the drug labels of NiuHuangShangQing tablet/capsule, Fuyanxiao capsule, and BiYanLing tablet should be improved.

CONCLUSIONS

In this study, we attempted a new way to find potential drug risks using small spontaneous reporting data. Our findings also suggested the need for more precise identification of allergic risks and the improvement of traditional Chinese medicine labels.

Bayesian safety surveillance with adaptive bias correction

Postmarket safety surveillance is an integral part of mass vaccination programs. Typically relying on sequential analysis of real-world health data as they accrue, safety surveillance is challenged by sequential multiple testing and by biases induced by residual confounding in observational data. The current standard approach based on the maximized sequential probability ratio test (MaxSPRT) fails to satisfactorily address these practical challenges and it remains a rigid framework that requires prespecification of the surveillance schedule. We develop an alternative Bayesian surveillance procedure that addresses both aforementioned challenges using a more flexible framework. To mitigate bias, we jointly analyze a large set of negative control outcomes that are adverse events with no known association with the vaccines in order to inform an empirical bias distribution, which we then incorporate into estimating the effect of vaccine exposure on the adverse event of interest through a Bayesian hierarchical model. To address multiple testing and improve on flexibility, at each analysis timepoint, we update a posterior probability in favor of the alternative hypothesis that vaccination induces higher risks of adverse events, and then use it for sequential detection of safety signals. Through an empirical evaluation using six US observational healthcare databases covering more than 360 million patients, we benchmark the proposed procedure against MaxSPRT on testing errors and estimation accuracy, under two epidemiological designs, the historical comparator and the self-controlled case series. We demonstrate that our procedure substantially reduces Type 1 error rates, maintains high statistical power and fast signal detection, and provides considerably more accurate estimation than MaxSPRT. Given the extensiveness of the empirical study which yields more than 7 million sets of results, we present all results in a public R ShinyApp. As an effort to promote open science, we provide full implementation of our method in the open-source R package EvidenceSynthesis.

Current Approaches to Vaccine Safety Using Observational Data: A Rationale for the EUMAEUS (Evaluating Use of Methods for Adverse Events Under Surveillance-for Vaccines) Study Design

Post-marketing vaccine safety surveillance aims to detect adverse events following immunization in a population. Whether certain methods of surveillance are more precise and unbiased in generating safety signals is unclear. Here, we synthesized information from existing literature to provide an overview of the strengths, weaknesses, and clinical applications of epidemiologic and analytical methods used in vaccine monitoring, focusing on cohort, case-control and self-controlled designs. These designs are proposed to be evaluated in the EUMAEUS (Evaluating Use of Methods for Adverse Event Under Surveillance-for vaccines) study because of their widespread use and potential utility. Over the past decades, there have been an increasing number of epidemiological study designs used for vaccine safety surveillance. While traditional cohort and case-control study designs remain widely used, newer, novel designs such as the self-controlled case series and self-controlled risk intervals have been developed. Each study design comes with its strengths and limitations, and the most appropriate study design will depend on availability of resources, access to records, number and distribution of cases, and availability of population coverage data. Several assumptions have to be made while using the various study designs, and while the goal is to mitigate any biases, violations of these assumptions are often still present to varying degrees. In our review, we discussed some of the potential biases (i.e., selection bias, misclassification bias and confounding bias), and ways to mitigate them. While the types of epidemiological study designs are well established, a comprehensive comparison of the analytical aspects (including method evaluation and performance metrics) of these study designs are relatively less well studied. We summarized the literature, reporting on two simulation studies, which compared the detection time, empirical power, error rate and risk estimate bias across the above-mentioned study designs. While these simulation studies provided insights on the analytic performance of each of the study designs, its applicability to real-world data remains unclear. To bridge that gap, we provided the rationale of the EUMAEUS study, with a brief description of the study design; and how the use of real-world multi-database networks can provide insights into better methods evaluation and vaccine safety surveillance.

Guillain-Barré Syndrome After High-Dose Influenza Vaccine Administration in the United States, 2018-2019 Season

BACKGROUND

The Vaccine Safety Datalink (VSD) identified a statistical signal for an increased risk of Guillain-Barré syndrome (GBS) in days 1-42 after 2018-2019 high-dose influenza vaccine (IIV3-HD) administration. We evaluated the signal using Medicare.

METHODS

We conducted early- and end-of-season claims-based self-controlled risk interval analyses among Medicare beneficiaries ages ≥65 years, using days 8-21 and 1-42 postvaccination as risk windows and days 43-84 as control window. The VSD conducted chart-confirmed analyses.

RESULTS

Among 7 453 690 IIV3-HD vaccinations, we did not detect a statistically significant increased GBS risk for either the 8- to 21-day (odds ratio [OR], 1.85; 95% confidence interval [CI], 0.99-3.44) or 1- to 42-day (OR, 1.31; 95% CI, 0.78-2.18) risk windows. The findings from the end-of-season analyses were fully consistent with the early-season analyses for both the 8- to 21-day (OR, 1.64; 95% CI, 0.92-2.91) and 1- to 42-day (OR, 1.12; 95% CI, 0.70-1.79) risk windows. The VSD's chart-confirmed analysis, involving 646 996 IIV3-HD vaccinations, with 1 case each in the risk and control windows, yielded a relative risk of 1.00 (95% CI, 0.06-15.99).

CONCLUSIONS

The Medicare analyses did not exclude an association between IIV3-HD and GBS, but it determined that, if such a risk existed, it was similar in magnitude to prior seasons. Chart-confirmed VSD results did not confirm an increased risk of GBS.

Enhanced Passive Safety Surveillance (EPSS) confirms an optimal safety profile of the use of MF59® -adjuvanted influenza vaccine in older adults: Results from three consecutive seasons

BACKGROUND

In Europe, the enhanced safety surveillance (ESS) of seasonal influenza vaccines is mandatory, in order to detect any potential increase in reactogenicity when the vaccine composition is updated. The MF59® -adjuvanted influenza vaccine (Fluad™) is the first and the only licensed adjuvanted seasonal influenza vaccine in Europe.

OBJECTIVE

Our objective was to summarize the safety data of Fluad™ over three consecutive seasons.

METHODS

A passive approach to ESS (EPSS) was adopted, in which reporting of spontaneous adverse events (AEs) by vaccinees and vaccine exposure was estimated, in order to generate a near real-time reporting rate. EPSS was conducted in Italy during the 2015, 2016, and 2017 influenza seasons in the primary care setting. All AEs reported within 7 days following immunization were analyzed by season, type and seriousness. Fisher's exact test was used to compare frequencies between seasons.

RESULTS

Total exposure accounted for approximately 1,000 doses of Fluad™ for each season. A total of 0.5% (2015), 0.7% (2016), and 0.5% (2017) individual case safety reports (ICSRs) were received, corresponding to a total of 9 (2015), 18 (2016), and 12 (2017) spontaneous AEs. The frequencies of AEs of interest were below those expected on the basis of the known safety profile of the vaccine. Most AEs were mild-to-moderate in severity. No between-season difference was found.

CONCLUSIONS

Our analyses confirmed that the safety data observed were consistent with the known safety profile of Fluad™, which has been amply established over the last 20 years. No significant changes in the safety profile were observed.

Challenges in implementing yearly enhanced safety surveillance of influenza vaccination in Europe: lessons learned and future perspectives

Seasonal influenza vaccines are frequently reformulated, leading to specific challenges for continuous benefit/risk monitoring. In 2014, the European Medicines Agency started requiring annual enhanced safety surveillance (ESS). This article provides a perspective on ESS studies conducted ever since and aims to map existing initiatives used to monitor adverse events following influenza immunization. Of 11 ESS studies, reporting surveillance data of at least five different vaccine brands during four seasons, all were able to rapidly capture vaccine-specific adverse events of interest reports. However, challenges have been identified during study implementation, including recruitment of sufficient participants, enrolling younger age groups, collecting data of vaccine batch numbers, comparing observed with expected rates and achieving adequate return of reported events. Harmonizing safety monitoring standards across countries, and bridging between routine pharmacovigilance and ESS, is likely to allow more comprehensive assessments of influenza vaccine safety, requiring close collaboration between regulators, public health, and manufacturers.

The use of technology to promote vaccination: A social ecological model based framework

Vaccinations are an important and effective cornerstone of preventive medical care. Growing technologic capabilities and use by both patients and providers present critical opportunities to leverage these tools to improve vaccination rates and public health. We propose the Social Ecological Model as a useful theoretical framework to identify areas in which technology has been or may be leveraged to target undervaccination across the individual, interpersonal, organizational, community, and society levels and the ways in which these levels interact.

Influenza vaccines: Evaluation of the safety profile

The safety of vaccines is a critical factor in maintaining public trust in national vaccination programs. Vaccines are recommended for children, adults and elderly subjects and have to meet higher safety standards, since they are administered to healthy subjects, mainly healthy children. Although vaccines are strictly monitored before authorization, the possibility of adverse events and/or rare adverse events cannot be totally eliminated. Two main types of influenza vaccines are currently available: parenteral inactivated influenza vaccines and intranasal live attenuated vaccines. Both display a good safety profile in adults and children. However, they can cause adverse events and/or rare adverse events, some of which are more prevalent in children, while others with a higher prevalence in adults. The aim of this review is to provide an overview of influenza vaccine safety according to target groups, vaccine types and production methods.

A multi-site feasibility study to assess fever and wheezing in children after influenza vaccines using text messaging

BACKGROUND

Using text messaging for vaccine safety monitoring, particularly for non-medically attended events, would be valuable for pandemic influenza and emergency vaccination program preparedness. We assessed the feasibility and acceptability of text messaging to evaluate fever and wheezing post-influenza vaccination in a prospective, observational, multi-site pediatric study.

METHODS

Children aged 2-11 years old, with an emphasis on children with asthma, were recruited during the 2014-2015 influenza season from three community-based clinics in New York City, and during the 2014-2015 and 2015-2016 seasons from a private practice in Fall River, Massachusetts. Parents of enrolled children receiving quadrivalent live attenuated (LAIV4) or inactivated influenza vaccine (IIV4) replied to text messages assessing respiratory symptoms (day 3 and 7, then weekly through day 42), and temperature on the night of vaccination and the next seven nights (day 0-7). Missing data were collected via diary (day 0-7 only) and phone. Phone confirmation was obtained for both presence and absence of respiratory symptoms. Reporting rates, fever (T≥100.4 °F) frequency, proportion of wheezing and/or chest tightness reports captured via text message versus all sources (text, phone, diary, electronic health record) and parental satisfaction were assessed.

RESULTS

Across both seasons, 266 children were analyzed; 49.2% with asthma. Parental text message response rates were high (>70%) across sites. Overall, fever frequency was low (day 0-2: 4.1% [95% confidence interval (CI) 2.3-7.4%]; d3-7: 6.7% [95% CI 4.1-10.8%]). A third (39.2%) of parents reported a respiratory problem in their child, primarily cough. Most (88.2%) of the 52 wheezing and/or chest tightness reports were by text message. Most (88.1%) participants preferred text messaging over paper reporting.

CONCLUSIONS

Text messaging can provide information about pediatric post-vaccination fever and wheezing and was viewed positively by parents. It could be a helpful tool for rapid vaccine safety monitoring during a pandemic or other emergency vaccination program.

TRIAL REGISTRATION

clinicaltrials.gov Identifier: NCT02295007.

Value of an in-depth analysis of unpublished data on the safety of influenza vaccines in pregnant women

Background

Unpublished data can sometimes provide valuable information on the safety of biologic products.

Methods

We assessed information potentially available from regulatory authorities, manufacturers, and public health agencies. We explored 4 recently established vaccine registries, reviewed package inserts from 99 influenza vaccines, and contacted vaccine manufacturers and regulatory agencies for data on influenza vaccine safety in pregnant women.

Results

The vaccine registries did not have sufficient data to analyze and there are problems with the quality of the information. The majority of package inserts provided no product-specific safety information for pregnant women, especially in less developed countries. The majority of available data come from reports gathered from passive adverse event reporting systems in the general population and reports of women enrolled in clinical trials of influenza vaccines who became pregnant at various times before or after receiving influenza vaccine. The information was not collected in a systematic manner, there are inconsistencies in the follow up of pregnant women and the available information about pregnancy outcomes. Considerable resources would be needed to systematically identify all of the information, try to obtain missing follow up information, and conduct analyses. There would be substantial limitations to any attempt to conduct a systematic analysis.

Conclusions

The value of trying to analyze unpublished data on the safety of influenza vaccine in pregnancy is limited and would require considerable resources to thoroughly investigate. Expanding efforts to identify and review unpublished data regarding the safety of influenza vaccines in pregnancy is not likely to produce information of high scientific value or information that could not be identified from publications and other publically available data.