1
|
Khademi S, Palmer C, Javed M, Dimaguila GL, Clothier H, Buttery J, Black J. Near Real-Time Syndromic Surveillance of Emergency Department Triage Texts Using Natural Language Processing: Case Study in Febrile Convulsion Detection. JMIR AI 2024; 3:e54449. [PMID: 39213519 PMCID: PMC11399745 DOI: 10.2196/54449] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Revised: 03/09/2024] [Accepted: 03/30/2024] [Indexed: 09/04/2024]
Abstract
BACKGROUND Collecting information on adverse events following immunization from as many sources as possible is critical for promptly identifying potential safety concerns and taking appropriate actions. Febrile convulsions are recognized as an important potential reaction to vaccination in children aged <6 years. OBJECTIVE The primary aim of this study was to evaluate the performance of natural language processing techniques and machine learning (ML) models for the rapid detection of febrile convulsion presentations in emergency departments (EDs), especially with respect to the minimum training data requirements to obtain optimum model performance. In addition, we examined the deployment requirements for a ML model to perform real-time monitoring of ED triage notes. METHODS We developed a pattern matching approach as a baseline and evaluated ML models for the classification of febrile convulsions in ED triage notes to determine both their training requirements and their effectiveness in detecting febrile convulsions. We measured their performance during training and then compared the deployed models' result on new incoming ED data. RESULTS Although the best standard neural networks had acceptable performance and were low-resource models, transformer-based models outperformed them substantially, justifying their ongoing deployment. CONCLUSIONS Using natural language processing, particularly with the use of large language models, offers significant advantages in syndromic surveillance. Large language models make highly effective classifiers, and their text generation capacity can be used to enhance the quality and diversity of training data.
Collapse
Affiliation(s)
- Sedigh Khademi
- Department of Paediatrics, University of Melbourne, Melbourne, Australia
- Health Informatics Group, Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
| | - Christopher Palmer
- Health Informatics Group, Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
| | - Muhammad Javed
- Health Informatics Group, Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
| | - Gerardo Luis Dimaguila
- Department of Paediatrics, University of Melbourne, Melbourne, Australia
- Health Informatics Group, Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
- SAEFVIC, Murdoch Children's Research Institute, Melbourne, Australia
| | - Hazel Clothier
- Department of Paediatrics, University of Melbourne, Melbourne, Australia
- Health Informatics Group, Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
- SAEFVIC, Murdoch Children's Research Institute, Melbourne, Australia
- Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Australia
| | - Jim Buttery
- Department of Paediatrics, University of Melbourne, Melbourne, Australia
- Health Informatics Group, Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
- SAEFVIC, Murdoch Children's Research Institute, Melbourne, Australia
- Infectious Diseases, Royal Children's Hospital, Melbourne, Australia
| | - Jim Black
- Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Australia
- Department of Health, State Government of Victoria, Melbourne, Australia
| |
Collapse
|
2
|
Alami A, Villeneuve PJ, Farrell PJ, Mattison D, Farhat N, Haddad N, Wilson K, Gravel CA, Crispo JAG, Perez-Lloret S, Krewski D. Myocarditis and Pericarditis Post-mRNA COVID-19 Vaccination: Insights from a Pharmacovigilance Perspective. J Clin Med 2023; 12:4971. [PMID: 37568373 PMCID: PMC10419493 DOI: 10.3390/jcm12154971] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Revised: 07/15/2023] [Accepted: 07/21/2023] [Indexed: 08/13/2023] Open
Abstract
Concerns remain regarding the rare cardiovascular adverse events, myocarditis and pericarditis (myo/pericarditis), particularly in younger individuals following mRNA COVID-19 vaccination. Our study aimed to comprehensively assess potential safety signals related to these cardiac events following the primary and booster doses, with a specific focus on younger populations, including children as young as 6 months of age. Using the Vaccine Adverse Events Reporting System (VAERS), the United States national passive surveillance system, we conducted a retrospective pharmacovigilance study analyzing spontaneous reports of myo/pericarditis. We employed both frequentist and Bayesian methods and conducted subgroup analyses by age, sex, and vaccine dose. We observed a higher reporting rate of myo/pericarditis following the primary vaccine series, particularly in males and mainly after the second dose. However, booster doses demonstrated a lower number of reported cases, with no significant signals detected after the fourth or fifth doses. In children and young adults, we observed notable age and sex differences in the reporting of myo/pericarditis cases. Males in the 12-17 and 18-24-year-old age groups had the highest number of cases, with significant signals for both males and females after the second dose. We also identified an increased reporting for a spectrum of cardiovascular symptoms such as chest pain and dyspnea, which increased with age, and were reported more frequently than myo/pericarditis. The present study identified signals of myo/pericarditis and related cardiovascular symptoms after mRNA COVID-19 vaccination, especially among children and adolescents. These findings underline the importance for continued vaccine surveillance and the need for further studies to confirm these results and to determine their clinical implications in public health decision-making, especially for younger populations.
Collapse
Affiliation(s)
- Abdallah Alami
- School of Mathematics and Statistics, Carleton University, Ottawa, ON K1S 5B6, Canada (N.F.)
- McLaughlin Centre for Population Health Risk Assessment, University of Ottawa, Ottawa, ON K1N 6N5, Canada
| | - Paul J. Villeneuve
- Department of Neuroscience, Faculty of Science, Carleton University, Ottawa, ON K1S 5B6, Canada
| | - Patrick J. Farrell
- School of Mathematics and Statistics, Carleton University, Ottawa, ON K1S 5B6, Canada (N.F.)
| | - Donald Mattison
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON K1G 5Z3, Canada
- Risk Sciences International, Ottawa, ON K1P 5J6, Canada
- Arnold School of Public Health, University of South Carolina, Columbia, SC 29208, USA
| | - Nawal Farhat
- School of Mathematics and Statistics, Carleton University, Ottawa, ON K1S 5B6, Canada (N.F.)
| | - Nisrine Haddad
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON K1G 5Z3, Canada
| | - Kumanan Wilson
- Department of Medicine, University of Ottawa, Ottawa, ON K1H 8M5, Canada
- Bruyère Research Institute, Ottawa, ON K1R 6M1, Canada
- Ottawa Hospital Research Institute, Ottawa, ON K1Y 4E9, Canada
| | - Christopher A. Gravel
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON K1G 5Z3, Canada
- Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, QC H3A 1Y7, Canada
- Department of Mathematics and Statistics, University of Ottawa, Ottawa, ON K1N 6N5, Canada
| | - James A. G. Crispo
- Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC V6T 1Z3, Canada
- Division of Human Sciences, NOSM University, Sudbury, ON P3E2C6, Canada
| | - Santiago Perez-Lloret
- Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Buenos Aires C1033AAJ, Argentina
- Observatorio de Salud Pública, Pontificia Universidad Católica Argentina, Buenos Aires C1107AAZ, Argentina
- Department of Physiology, Faculty of Medicine, University of Buenos Aires, Buenos Aires C1121ABG, Argentina
| | - Daniel Krewski
- McLaughlin Centre for Population Health Risk Assessment, University of Ottawa, Ottawa, ON K1N 6N5, Canada
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON K1G 5Z3, Canada
- Risk Sciences International, Ottawa, ON K1P 5J6, Canada
| |
Collapse
|
3
|
Zhang BX, Lin WY, Huang TK. Stacking Ensemble of Disproportionality Indicators for Adverse Vaccine Reactions Detection-An Empirical Study on Predicting Adverse Reactions of COVID-19 Vaccines. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38082660 DOI: 10.1109/embc40787.2023.10340698] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Vaccine safety is a critical issue for public health, which has recently become more crucial than ever since COVID-19 started to spread worldwide in 2020. Many COVID-19 vaccines have been developed and used without following the traditional three clinical trial stages. Instead, most COVID-19 vaccines were approved through emergency use approval (EUA) within one year, significantly raising the risk of rare and severe adverse events. Reporting systems like the Vaccine Adverse Event Reporting System (VAERS) have been established worldwide to detect unknown and severe adverse reactions as early as possible. Although experts and researchers have been working hard to find ways to detect adverse vaccine event (AVE) signals from VAERS data, most of the contemporary methods are statistical methods based on measuring the disproportionality between vaccine-induced events and non-vaccine-induced events. This paper proposes a novel ensemble AVE detection method, which adopts a stacking ensemble of various disproportionality indicators, fusing dual-scale contingency values measured in single and cumulative yearly duration, and embraces the concept of feature concatenation. Experiments conducted on US VAERS data to predict AVE caused by COVID-19 vaccines show that our proposed method is effective. We observed that: (1) Stacking ensemble of various disproportionality indicators is superior to any single disproportionality indicator and voting ensemble method; (2) Fusing dual-scale contingency values and feature concatenation brings synergy to our proposed stacking ensemble AVE detection. Compared to the best disproportionality metric in this study, our top-performing ensemble version exhibited a 34% improvement in accuracy, 71% in precision, 29% in recall, and 77% in F-measure, with a slight decrease (8%) in specificity.
Collapse
|
4
|
Dauner DG, Zhang R, Adam TJ, Leal E, Heitlage V, Farley JF. Performance of subgrouped proportional reporting ratios in the US Food and Drug Administration (FDA) adverse event reporting system. Expert Opin Drug Saf 2023; 22:589-597. [PMID: 36800190 DOI: 10.1080/14740338.2023.2182289] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Accepted: 02/06/2023] [Indexed: 02/18/2023]
Abstract
BACKGROUND Many signal detection algorithms give the same weight to information from all products and patients, which may result in signals being masked or false positives being flagged as potential signals. Subgrouped analysis can be used to help correct for this. RESEARCH DESIGN AND METHODS The publicly available US Food and Drug Administration Adverse Event Reporting System quarterly data extract files from 1 January 2015 through 30 September 2017 were utilized. A proportional reporting ratio (PRR) analysis subgrouped by either age, sex, ADE report type, seriousness of ADE, or reporter was compared to the crude PRR analysis using sensitivity, specificity, precision, and c-statistic. RESULTS Subgrouping by age (n = 78, 34.5% increase), sex (n = 67, 15.5% increase), and reporter (n = 64, 10.3% increase) identified more signals than the crude analysis. Subgrouping by either age or sex increased both the sensitivity and precision. Subgrouping by report type or seriousness resulted in fewer signals (n = 50, -13.8% for both). Subgrouped analyses had higher c-statistic values, with age having the highest (0.468). CONCLUSIONS Subgrouping by either age or sex produced more signals with higher sensitivity and precision than the crude PRR analysis. Subgrouping by these variables can unmask potentially important associations.
Collapse
Affiliation(s)
- Daniel G Dauner
- Department of Pharmaceutical Care and Health Systems, College of Pharmacy, University of Minnesota, Minneapolis, Minnesota, USA
| | - Rui Zhang
- Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, Minnesota, USA
| | - Terrence J Adam
- Department of Pharmaceutical Care and Health Systems, College of Pharmacy, Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Eleazar Leal
- Department of Computer Science, Swenson College of Science and Engineering, University of Minnesota, Duluth, Minnesota, USA
| | - Viviene Heitlage
- Department of Pharmaceutical Care and Health Systems, College of Pharmacy, University of Minnesota, Minneapolis, Minnesota, USA
| | - Joel F Farley
- Department of Pharmaceutical Care and Health Systems, College of Pharmacy, University of Minnesota, Minneapolis, Minnesota, USA
| |
Collapse
|
5
|
Cheng AC, Buttery JP. Vaccine safety: what systems are required to ensure public confidence in vaccines? Med J Aust 2022; 217:189-190. [PMID: 35843626 PMCID: PMC9349887 DOI: 10.5694/mja2.51662] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2022] [Revised: 06/23/2022] [Accepted: 06/24/2022] [Indexed: 12/02/2022]
Affiliation(s)
| | - Jim P Buttery
- Surveillance of Adverse Events Following Vaccination In the Community (SAEFVIC), Murdoch Children’s Research Institute Melbourne VIC
| |
Collapse
|
6
|
Khademi Habibabadi S, Delir Haghighi P, Burstein F, Buttery J. Vaccine adverse event mentions in social media: Mining the language of Twitter conversations (Preprint). JMIR Med Inform 2021; 10:e34305. [PMID: 35708760 PMCID: PMC9247809 DOI: 10.2196/34305] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2021] [Revised: 02/22/2022] [Accepted: 04/11/2022] [Indexed: 11/13/2022] Open
Abstract
Background Traditional monitoring for adverse events following immunization (AEFI) relies on various established reporting systems, where there is inevitable lag between an AEFI occurring and its potential reporting and subsequent processing of reports. AEFI safety signal detection strives to detect AEFI as early as possible, ideally close to real time. Monitoring social media data holds promise as a resource for this. Objective The primary aim of this study is to investigate the utility of monitoring social media for gaining early insights into vaccine safety issues, by extracting vaccine adverse event mentions (VAEMs) from Twitter, using natural language processing techniques. The secondary aims are to document the natural language processing techniques used and identify the most effective of them for identifying tweets that contain VAEM, with a view to define an approach that might be applicable to other similar social media surveillance tasks. Methods A VAEM-Mine method was developed that combines topic modeling with classification techniques to extract maximal VAEM posts from a vaccine-related Twitter stream, with high degree of confidence. The approach does not require a targeted search for specific vaccine reaction–indicative words, but instead, identifies VAEM posts according to their language structure. Results The VAEM-Mine method isolated 8992 VAEMs from 811,010 vaccine-related Twitter posts and achieved an F1 score of 0.91 in the classification phase. Conclusions Social media can assist with the detection of vaccine safety signals as a valuable complementary source for monitoring mentions of vaccine adverse events. A social media–based VAEM data stream can be assessed for changes to detect possible emerging vaccine safety signals, helping to address the well-recognized limitations of passive reporting systems, including lack of timeliness and underreporting.
Collapse
Affiliation(s)
- Sedigheh Khademi Habibabadi
- Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
- Department of General Practice, University of Melbourne, Melbourne, Australia
| | - Pari Delir Haghighi
- Department of Human-Centred Computing, Faculty of Information Technology, Monash University, Melbourne, Australia
| | - Frada Burstein
- Department of Human-Centred Computing, Faculty of Information Technology, Monash University, Melbourne, Australia
| | - Jim Buttery
- Centre for Health Analytics, Melbourne Children's Campus, Melbourne, Australia
- Department of Paediatrics, University of Melbourne, Melbourne, Australia
| |
Collapse
|
7
|
Mesfin YM, Cheng AC, Enticott J, Lawrie J, Buttery J. Post-vaccination healthcare attendance rate as a proxy measure for syndromic surveillance of adverse events following immunisation. Aust N Z J Public Health 2021; 45:101-107. [PMID: 33617131 DOI: 10.1111/1753-6405.13052] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2020] [Revised: 05/01/2020] [Accepted: 09/01/2020] [Indexed: 11/30/2022] Open
Abstract
OBJECTIVE This study explored whether all-cause healthcare attendance rate post-vaccination could detect the two historical influenza safety episodes occurring in 2010 and 2015 using a large de-identified general practitioner (GP) consultations dataset. METHODS A retrospective observational cohort study was conducted using GP consultation data routinely collected from 2008 to 2017 in Victoria, Australia. Post-vaccination GP consultation rates were monitored, over a 22-week surveillance period each year that aligned with each year's influenza vaccination season, using the Observed minus Expected (O-E) and the Log-Likelihood Ratio (LLR) CUSUM charts. Days 1-7 post-vaccination were considered as the risk period. The LLR CUSUM was designed to detect both a 50% and two-fold rise in the odds of the baseline post-vaccination GP consultation rates. RESULTS Over the 10-year study period, more than 1.5 million seasonal influenza vaccines doses were administered to 295,091 persons. Overall, 1.29% had a GP consultation within one week of vaccination, but 98.53% of the consultations occurred in days 1-3 post-vaccination. The LLR CUSUM chart detected significant increases in the weekly rates of post-vaccination GP consultation in 2010 in children aged under ten years and in 2015 in adults aged 19-64 years. These increases were aligned by week, but one week earlier and by age category, with the historical adverse events following immunisation (AEFI) signals occurring in 2010 and 2015. However, in the absence of historical AEFI signals, increased rates of post-vaccination GP consultations were identified in three of the eight influenza vaccination years. CONCLUSION The crude post-vaccination healthcare attendance rate has the potential to offer a sensitive proxy to monitor vaccine safety signal. Implications for public health: Vaccine safety monitoring using syndromic indicator has the potential to augment the existing surveillance systems as part of an integrated vaccine safety monitoring approach.
Collapse
Affiliation(s)
- Yonatan Moges Mesfin
- Monash Centre for Health Research and Implementation, School of Public Health and Preventive Medicine, Monash University, Victoria
| | - Allen C Cheng
- Infection Prevention and Healthcare Epidemiology Unit, Alfred Health Melbourne, Victoria
| | - Joanne Enticott
- Monash Centre for Health Research and Implementation, School of Public Health and Preventive Medicine, Monash University, Victoria
| | - Jock Lawrie
- Monash Centre for Health Research and Implementation, School of Public Health and Preventive Medicine, Monash University, Victoria
| | - Jim Buttery
- Monash Centre for Health Research and Implementation, School of Public Health and Preventive Medicine, Monash University, Victoria.,Infection and Immunity, Monash Children's Hospital, Victoria
| |
Collapse
|
8
|
Efficacy of m-Health for the detection of adverse events following immunization - The stimulated telephone assisted rapid safety surveillance (STARSS) randomised control trial. Vaccine 2020; 39:332-342. [PMID: 33279317 DOI: 10.1016/j.vaccine.2020.11.056] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2020] [Revised: 11/13/2020] [Accepted: 11/19/2020] [Indexed: 01/09/2023]
Abstract
INTRODUCTION Passive surveillance is recommended globally for the detection of adverse events following immunisation (AEFI) but this has significant challenges. Use of Mobile health for vaccine safety surveillance enables a consumer-centred approach to reporting. The Stimulated Telephone Assisted Rapid Safety Surveillance (STARSS) a randomised control trial (RCT) sought to evaluate the efficacy and acceptability of SMS for AEFI surveillance. METHODS Multi-centre RCT, participants were adult vaccinees or parents of children receiving any vaccine at a trial site. At enrolment randomisation occurred to one of two SMS groups or a control group. Prompts on days 2, 7 and 14 post-immunisation, were sent to the SMS group, to ascertain if a medical event following immunisation (MEFI) had occurred. No SMS's were sent to the control participants. Those in the SMS who notified an MEFI were pre-randomised to complete a computer assisted telephone interview or a web based report to determine if an AEFI had occurred whilst an AEFI in the controls was determined by a search for passive reports. The primary outcome was the AEFI detection rate in the SMS group compared to controls. RESULTS We enrolled 6,338 participants, who were equally distributed across groups and who received 11,675 vaccines. The SMS group (4,225) received 12,675 surveillance prompts with 9.8% being non-compliant and not responding. In those that responded 90% indicated that no MEFI had been experienced and 184 had a verified AEFI. 6 control subjects had a reported AEFI. The AEFI detection rate was 13 fold greater in the SMS group when compared with controls (4.3 vs 0.3%). CONCLUSION We have demonstrated that the STARSS methodology improves AEFI detection. Our findings should inform the wider use of SMS-based surveillance which is particularly relevant since establishing robust and novel pharmacovigilance systems is critical to monitoring novel vaccines which includes potential COVID vaccines.
Collapse
|
9
|
Mesfin YM, Cheng AC, Enticott J, Lawrie J, Buttery JP. Use of telephone helpline data for syndromic surveillance of adverse events following immunization in Australia: A retrospective study, 2009 to 2017. Vaccine 2020; 38:5525-5531. [DOI: 10.1016/j.vaccine.2020.05.078] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2020] [Revised: 05/25/2020] [Accepted: 05/27/2020] [Indexed: 11/17/2022]
|
10
|
Farmacovigilancia de vacunas y su aplicación en Chile. REVISTA MÉDICA CLÍNICA LAS CONDES 2020. [DOI: 10.1016/j.rmclc.2020.03.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
|