Early signal detection of adverse events following influenza vaccination using proportional reporting ratio, Victoria, Australia

Near Real-Time Syndromic Surveillance of Emergency Department Triage Texts Using Natural Language Processing: Case Study in Febrile Convulsion Detection

BACKGROUND

Collecting information on adverse events following immunization from as many sources as possible is critical for promptly identifying potential safety concerns and taking appropriate actions. Febrile convulsions are recognized as an important potential reaction to vaccination in children aged <6 years.

OBJECTIVE

The primary aim of this study was to evaluate the performance of natural language processing techniques and machine learning (ML) models for the rapid detection of febrile convulsion presentations in emergency departments (EDs), especially with respect to the minimum training data requirements to obtain optimum model performance. In addition, we examined the deployment requirements for a ML model to perform real-time monitoring of ED triage notes.

METHODS

We developed a pattern matching approach as a baseline and evaluated ML models for the classification of febrile convulsions in ED triage notes to determine both their training requirements and their effectiveness in detecting febrile convulsions. We measured their performance during training and then compared the deployed models' result on new incoming ED data.

RESULTS

Although the best standard neural networks had acceptable performance and were low-resource models, transformer-based models outperformed them substantially, justifying their ongoing deployment.

CONCLUSIONS

Using natural language processing, particularly with the use of large language models, offers significant advantages in syndromic surveillance. Large language models make highly effective classifiers, and their text generation capacity can be used to enhance the quality and diversity of training data.

Myocarditis and Pericarditis Post-mRNA COVID-19 Vaccination: Insights from a Pharmacovigilance Perspective

Concerns remain regarding the rare cardiovascular adverse events, myocarditis and pericarditis (myo/pericarditis), particularly in younger individuals following mRNA COVID-19 vaccination. Our study aimed to comprehensively assess potential safety signals related to these cardiac events following the primary and booster doses, with a specific focus on younger populations, including children as young as 6 months of age. Using the Vaccine Adverse Events Reporting System (VAERS), the United States national passive surveillance system, we conducted a retrospective pharmacovigilance study analyzing spontaneous reports of myo/pericarditis. We employed both frequentist and Bayesian methods and conducted subgroup analyses by age, sex, and vaccine dose. We observed a higher reporting rate of myo/pericarditis following the primary vaccine series, particularly in males and mainly after the second dose. However, booster doses demonstrated a lower number of reported cases, with no significant signals detected after the fourth or fifth doses. In children and young adults, we observed notable age and sex differences in the reporting of myo/pericarditis cases. Males in the 12-17 and 18-24-year-old age groups had the highest number of cases, with significant signals for both males and females after the second dose. We also identified an increased reporting for a spectrum of cardiovascular symptoms such as chest pain and dyspnea, which increased with age, and were reported more frequently than myo/pericarditis. The present study identified signals of myo/pericarditis and related cardiovascular symptoms after mRNA COVID-19 vaccination, especially among children and adolescents. These findings underline the importance for continued vaccine surveillance and the need for further studies to confirm these results and to determine their clinical implications in public health decision-making, especially for younger populations.

Stacking Ensemble of Disproportionality Indicators for Adverse Vaccine Reactions Detection-An Empirical Study on Predicting Adverse Reactions of COVID-19 Vaccines

Vaccine safety is a critical issue for public health, which has recently become more crucial than ever since COVID-19 started to spread worldwide in 2020. Many COVID-19 vaccines have been developed and used without following the traditional three clinical trial stages. Instead, most COVID-19 vaccines were approved through emergency use approval (EUA) within one year, significantly raising the risk of rare and severe adverse events. Reporting systems like the Vaccine Adverse Event Reporting System (VAERS) have been established worldwide to detect unknown and severe adverse reactions as early as possible. Although experts and researchers have been working hard to find ways to detect adverse vaccine event (AVE) signals from VAERS data, most of the contemporary methods are statistical methods based on measuring the disproportionality between vaccine-induced events and non-vaccine-induced events. This paper proposes a novel ensemble AVE detection method, which adopts a stacking ensemble of various disproportionality indicators, fusing dual-scale contingency values measured in single and cumulative yearly duration, and embraces the concept of feature concatenation. Experiments conducted on US VAERS data to predict AVE caused by COVID-19 vaccines show that our proposed method is effective. We observed that: (1) Stacking ensemble of various disproportionality indicators is superior to any single disproportionality indicator and voting ensemble method; (2) Fusing dual-scale contingency values and feature concatenation brings synergy to our proposed stacking ensemble AVE detection. Compared to the best disproportionality metric in this study, our top-performing ensemble version exhibited a 34% improvement in accuracy, 71% in precision, 29% in recall, and 77% in F-measure, with a slight decrease (8%) in specificity.

Performance of subgrouped proportional reporting ratios in the US Food and Drug Administration (FDA) adverse event reporting system

BACKGROUND

Many signal detection algorithms give the same weight to information from all products and patients, which may result in signals being masked or false positives being flagged as potential signals. Subgrouped analysis can be used to help correct for this.

RESEARCH DESIGN AND METHODS

The publicly available US Food and Drug Administration Adverse Event Reporting System quarterly data extract files from 1 January 2015 through 30 September 2017 were utilized. A proportional reporting ratio (PRR) analysis subgrouped by either age, sex, ADE report type, seriousness of ADE, or reporter was compared to the crude PRR analysis using sensitivity, specificity, precision, and c-statistic.

RESULTS

Subgrouping by age (n = 78, 34.5% increase), sex (n = 67, 15.5% increase), and reporter (n = 64, 10.3% increase) identified more signals than the crude analysis. Subgrouping by either age or sex increased both the sensitivity and precision. Subgrouping by report type or seriousness resulted in fewer signals (n = 50, -13.8% for both). Subgrouped analyses had higher c-statistic values, with age having the highest (0.468).

CONCLUSIONS

Subgrouping by either age or sex produced more signals with higher sensitivity and precision than the crude PRR analysis. Subgrouping by these variables can unmask potentially important associations.

Vaccine adverse event mentions in social media: Mining the language of Twitter conversations (Preprint)

Background

Traditional monitoring for adverse events following immunization (AEFI) relies on various established reporting systems, where there is inevitable lag between an AEFI occurring and its potential reporting and subsequent processing of reports. AEFI safety signal detection strives to detect AEFI as early as possible, ideally close to real time. Monitoring social media data holds promise as a resource for this.

Objective

The primary aim of this study is to investigate the utility of monitoring social media for gaining early insights into vaccine safety issues, by extracting vaccine adverse event mentions (VAEMs) from Twitter, using natural language processing techniques. The secondary aims are to document the natural language processing techniques used and identify the most effective of them for identifying tweets that contain VAEM, with a view to define an approach that might be applicable to other similar social media surveillance tasks.

Methods

A VAEM-Mine method was developed that combines topic modeling with classification techniques to extract maximal VAEM posts from a vaccine-related Twitter stream, with high degree of confidence. The approach does not require a targeted search for specific vaccine reaction–indicative words, but instead, identifies VAEM posts according to their language structure.

Results

The VAEM-Mine method isolated 8992 VAEMs from 811,010 vaccine-related Twitter posts and achieved an F₁ score of 0.91 in the classification phase.

Conclusions

Social media can assist with the detection of vaccine safety signals as a valuable complementary source for monitoring mentions of vaccine adverse events. A social media–based VAEM data stream can be assessed for changes to detect possible emerging vaccine safety signals, helping to address the well-recognized limitations of passive reporting systems, including lack of timeliness and underreporting.

Post-vaccination healthcare attendance rate as a proxy measure for syndromic surveillance of adverse events following immunisation

OBJECTIVE

This study explored whether all-cause healthcare attendance rate post-vaccination could detect the two historical influenza safety episodes occurring in 2010 and 2015 using a large de-identified general practitioner (GP) consultations dataset.

METHODS

A retrospective observational cohort study was conducted using GP consultation data routinely collected from 2008 to 2017 in Victoria, Australia. Post-vaccination GP consultation rates were monitored, over a 22-week surveillance period each year that aligned with each year's influenza vaccination season, using the Observed minus Expected (O-E) and the Log-Likelihood Ratio (LLR) CUSUM charts. Days 1-7 post-vaccination were considered as the risk period. The LLR CUSUM was designed to detect both a 50% and two-fold rise in the odds of the baseline post-vaccination GP consultation rates.

RESULTS

Over the 10-year study period, more than 1.5 million seasonal influenza vaccines doses were administered to 295,091 persons. Overall, 1.29% had a GP consultation within one week of vaccination, but 98.53% of the consultations occurred in days 1-3 post-vaccination. The LLR CUSUM chart detected significant increases in the weekly rates of post-vaccination GP consultation in 2010 in children aged under ten years and in 2015 in adults aged 19-64 years. These increases were aligned by week, but one week earlier and by age category, with the historical adverse events following immunisation (AEFI) signals occurring in 2010 and 2015. However, in the absence of historical AEFI signals, increased rates of post-vaccination GP consultations were identified in three of the eight influenza vaccination years.

CONCLUSION

The crude post-vaccination healthcare attendance rate has the potential to offer a sensitive proxy to monitor vaccine safety signal. Implications for public health: Vaccine safety monitoring using syndromic indicator has the potential to augment the existing surveillance systems as part of an integrated vaccine safety monitoring approach.

Efficacy of m-Health for the detection of adverse events following immunization - The stimulated telephone assisted rapid safety surveillance (STARSS) randomised control trial

INTRODUCTION

Passive surveillance is recommended globally for the detection of adverse events following immunisation (AEFI) but this has significant challenges. Use of Mobile health for vaccine safety surveillance enables a consumer-centred approach to reporting. The Stimulated Telephone Assisted Rapid Safety Surveillance (STARSS) a randomised control trial (RCT) sought to evaluate the efficacy and acceptability of SMS for AEFI surveillance.

METHODS

Multi-centre RCT, participants were adult vaccinees or parents of children receiving any vaccine at a trial site. At enrolment randomisation occurred to one of two SMS groups or a control group. Prompts on days 2, 7 and 14 post-immunisation, were sent to the SMS group, to ascertain if a medical event following immunisation (MEFI) had occurred. No SMS's were sent to the control participants. Those in the SMS who notified an MEFI were pre-randomised to complete a computer assisted telephone interview or a web based report to determine if an AEFI had occurred whilst an AEFI in the controls was determined by a search for passive reports. The primary outcome was the AEFI detection rate in the SMS group compared to controls.

RESULTS

We enrolled 6,338 participants, who were equally distributed across groups and who received 11,675 vaccines. The SMS group (4,225) received 12,675 surveillance prompts with 9.8% being non-compliant and not responding. In those that responded 90% indicated that no MEFI had been experienced and 184 had a verified AEFI. 6 control subjects had a reported AEFI. The AEFI detection rate was 13 fold greater in the SMS group when compared with controls (4.3 vs 0.3%).

CONCLUSION

We have demonstrated that the STARSS methodology improves AEFI detection. Our findings should inform the wider use of SMS-based surveillance which is particularly relevant since establishing robust and novel pharmacovigilance systems is critical to monitoring novel vaccines which includes potential COVID vaccines.