1
|
Li Y, Li J, He J, Tao C. AE-GPT: Using Large Language Models to extract adverse events from surveillance reports-A use case with influenza vaccine adverse events. PLoS One 2024; 19:e0300919. [PMID: 38512919 PMCID: PMC10956752 DOI: 10.1371/journal.pone.0300919] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Accepted: 03/06/2024] [Indexed: 03/23/2024] Open
Abstract
Though Vaccines are instrumental in global health, mitigating infectious diseases and pandemic outbreaks, they can occasionally lead to adverse events (AEs). Recently, Large Language Models (LLMs) have shown promise in effectively identifying and cataloging AEs within clinical reports. Utilizing data from the Vaccine Adverse Event Reporting System (VAERS) from 1990 to 2016, this study particularly focuses on AEs to evaluate LLMs' capability for AE extraction. A variety of prevalent LLMs, including GPT-2, GPT-3 variants, GPT-4, and Llama2, were evaluated using Influenza vaccine as a use case. The fine-tuned GPT 3.5 model (AE-GPT) stood out with a 0.704 averaged micro F1 score for strict match and 0.816 for relaxed match. The encouraging performance of the AE-GPT underscores LLMs' potential in processing medical data, indicating a significant stride towards advanced AE detection, thus presumably generalizable to other AE extraction tasks.
Collapse
Affiliation(s)
- Yiming Li
- McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, United States of America
| | - Jianfu Li
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Jacksonville, FL, United States of America
| | - Jianping He
- McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, United States of America
| | - Cui Tao
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Jacksonville, FL, United States of America
| |
Collapse
|
2
|
Yu M, Nie S, Qiao Y, Ma Y. Guillain-Barre syndrome following COVID-19 vaccines: A review of literature. Front Immunol 2023; 14:1078197. [PMID: 36875094 PMCID: PMC9975597 DOI: 10.3389/fimmu.2023.1078197] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Accepted: 02/03/2023] [Indexed: 02/17/2023] Open
Abstract
Objective This study aimed to retrospectively analyze reported Guillain-Barré syndrome (GBS) cases that occurred after COVID-19 vaccination. Methods Case reports of GBS following COVID-19 vaccination that were published before May 14, 2022, were retrieved from PubMed. The cases were retrospectively analyzed for their basic characteristics, vaccine types, the number of vaccination doses before onset, clinical manifestations, laboratory test results, neurophysiological examination results, treatment, and prognosis. Results Retrospective analysis of 60 case reports revealed that post-COVID-19 vaccination GBS occurred mostly after the first dose of the vaccination (54 cases, 90%) and was common for DNA vaccination (38 cases, 63%), common in middle-aged and elderly people (mean age: 54.5 years), and also common in men (36 cases, 60%). The mean time from vaccination to onset was 12.3 days. The classical GBS (31 cases, 52%) was the major clinical classification and the AIDP subtype (37 cases, 71%) was the major neurophysiological subtype, but the positive rate of anti-ganglioside antibodies was low (7 cases, 20%). Bilateral facial nerve palsy (76% vs 18%) and facial palsy with distal paresthesia (38% vs 5%) were more common for DNA vaccination than for RNA vaccination. Conclusion After reviewing the literature, we proposed a possible association between the risk of GBS and the first dose of the COVID-19 vaccines, especially DNA vaccines. The higher rate of facial involvement and a lower positive rate of anti-ganglioside antibodies may be a characteristic feature of GBS following COVID-19 vaccination. The causal relationship between GBS and COVID-19 vaccination remains speculative, more research is needed to establish an association between GBS and COVID-19 vaccination. We recommend surveillance for GBS following vaccination, because it is important in determining the true incidence of GBS following COVID-19 vaccination, as well as in the development of a more safer vaccine.
Collapse
Affiliation(s)
- Miao Yu
- Department of Neurology, Shengjing Hospital of China Medical University, Shenyang, China.,Department of Neurology, Shenyang First People's Hospital, Shenyang, China
| | - Shuang Nie
- Department of Neurology, Shengjing Hospital of China Medical University, Shenyang, China
| | - Yue Qiao
- Department of Neurology, Shengjing Hospital of China Medical University, Shenyang, China
| | - Ying Ma
- Department of Neurology, Shengjing Hospital of China Medical University, Shenyang, China
| |
Collapse
|
3
|
Vaccine Vigilance System: Considerations on the Effectiveness of Vigilance Data Use in COVID-19 Vaccination. Vaccines (Basel) 2022; 10:vaccines10122115. [PMID: 36560525 PMCID: PMC9783025 DOI: 10.3390/vaccines10122115] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 12/08/2022] [Accepted: 12/08/2022] [Indexed: 12/14/2022] Open
Abstract
(1) Background: The safety of medicines has been receiving increased attention to ensure that the risks of taking medicines do not outweigh the benefits. This is the reason why, over several decades, the pharmacovigilance system has been developed. The post-authorization pharmacovigilance system is based on reports from healthcare professionals and patients on observed adverse reactions. The reports are collected in databases and progressively evaluated. However, there are emerging concerns about the effectiveness of the established passive pharmacovigilance system in accelerating circumstances, such as the COVID-19 pandemic, when billions of doses of new vaccines were administered without a long history of use. Currently, health professionals receive fragmented new information on the safety of medicines from competent authorities after a lengthy evaluation process. Simultaneously, in the context of accelerated mass vaccination, health professionals need to have access to operational information-at least on organ systems at higher risk. Therefore, the aim of this study was to perform a primary data analysis of publicly available data on suspected COVID-19 vaccine-related adverse reactions in Europe, in order to identify the predominant groups of reported medical conditions after vaccination and their association with vaccine groups, as well as to evaluate the data accessibility on specific syndromes. (2) Methods: To achieve the objectives, the data publicly available in the EudraVigilance European Database for Suspected Adverse Drug Reaction Reports were analyzed. The following tasks were defined to: (1) Identify the predominant groups of medical conditions mentioned in adverse reaction reports; (2) determine the relative frequency of reports within vaccine groups; (3) assess the feasibility of obtaining information on a possibly associated syndrome-myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS). (3) Results: The data obtained demonstrate that the predominant medical conditions induced after vaccination are relevant to the following categories: (1) "General disorders and administration site conditions", (2) "nervous system disorders", and (3) "musculoskeletal and connective tissue disorders". There are more reports for mRNA vaccines, but the relative frequency of reports per dose administered, is lower for this group of vaccines. Information on ME/CFS was not available, but reports of "chronic fatigue syndrome" are included in the database and accessible for primary analysis. (4) Conclusions: The information obtained on the predominantly reported medical conditions and the relevant vaccine groups may be useful for health professionals, patients, researchers, and medicine manufacturers. Policymakers could benefit from reflecting on the design of an active pharmacovigilance model, making full use of modern information technologies, including big data analysis of social media and networks for the detection of primary signals and building an early warning system.
Collapse
|
4
|
Flora J, Khan W, Jin J, Jin D, Hussain A, Dajani K, Khan B. Usefulness of Vaccine Adverse Event Reporting System for Machine-Learning Based Vaccine Research: A Case Study for COVID-19 Vaccines. Int J Mol Sci 2022; 23:ijms23158235. [PMID: 35897804 PMCID: PMC9368306 DOI: 10.3390/ijms23158235] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2022] [Revised: 07/06/2022] [Accepted: 07/21/2022] [Indexed: 02/04/2023] Open
Abstract
Usefulness of Vaccine-Adverse Event-Reporting System (VAERS) data and protocols required for statistical analyses were pinpointed with a set of recommendations for the application of machine learning modeling or exploratory analyses on VAERS data with a case study of COVID-19 vaccines (Pfizer-BioNTech, Moderna, Janssen). A total of 262,454 duplicate reports (29%) from 905,976 reports were identified, which were merged into a total of 643,522 distinct reports. A customized online survey was also conducted providing 211 reports. A total of 20 highest reported adverse events were first identified. Differences in results after applying various machine learning algorithms (association rule mining, self-organizing maps, hierarchical clustering, bipartite graphs) on VAERS data were noticed. Moderna reports showed injection-site-related AEs of higher frequencies by 15.2%, consistent with the online survey (12% higher reporting rate for pain in the muscle for Moderna compared to Pfizer-BioNTech). AEs {headache, pyrexia, fatigue, chills, pain, dizziness} constituted >50% of the total reports. Chest pain in male children reports was 295% higher than in female children reports. Penicillin and sulfa were of the highest frequencies (22%, and 19%, respectively). Analysis of uncleaned VAERS data demonstrated major differences from the above (7% variations). Spelling/grammatical mistakes in allergies were discovered (e.g., ~14% reports with incorrect spellings for penicillin).
Collapse
Affiliation(s)
- James Flora
- Department of Computer Science and Engineering, California State University San Bernardino, 5500 University Parkway, San Bernardino, CA 92407, USA; (J.F.); (J.J.); (K.D.)
| | - Wasiq Khan
- School of Computer Science and Mathematics, Liverpool John Moores University, Liverpool L3 3AF, UK;
| | - Jennifer Jin
- Department of Computer Science and Engineering, California State University San Bernardino, 5500 University Parkway, San Bernardino, CA 92407, USA; (J.F.); (J.J.); (K.D.)
| | - Daniel Jin
- Division of Vascular & Interventional Radiology, Department of Radiology, Loma Linda University Medical Center, Loma Linda, CA 92354, USA;
| | - Abir Hussain
- Department of Electrical Engineering, University of Sharjah, Sharjah P.O. Box 27272, United Arab Emirates;
| | - Khalil Dajani
- Department of Computer Science and Engineering, California State University San Bernardino, 5500 University Parkway, San Bernardino, CA 92407, USA; (J.F.); (J.J.); (K.D.)
| | - Bilal Khan
- Department of Computer Science and Engineering, California State University San Bernardino, 5500 University Parkway, San Bernardino, CA 92407, USA; (J.F.); (J.J.); (K.D.)
- Institute of the Environment and Sustainability, University of California Los Angeles, Los Angeles, CA 90095, USA
- Correspondence: ; Tel.: +1-(909)-537-5428
| |
Collapse
|
5
|
Luo C, Du J, Cuker A, Lautenbach E, Asch DA, Poland GA, Tao C, Chen Y. Comparability of clinical trials and spontaneous reporting data regarding COVID-19 vaccine safety. Sci Rep 2022; 12:10946. [PMID: 35768434 PMCID: PMC9243073 DOI: 10.1038/s41598-022-13809-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2022] [Accepted: 05/27/2022] [Indexed: 11/09/2022] Open
Abstract
Severe adverse events (AEs) after COVID-19 vaccination are not well studied in randomized controlled trials (RCTs) due to rarity and short follow-up. To monitor the safety of COVID-19 vaccines ("Pfizer" vaccine dose 1 and 2, "Moderna" vaccine dose 1 and 2, and "Janssen" vaccine single dose) in the U.S., especially regarding severe AEs, we compare the relative rankings of these vaccines using both RCT and the Vaccine Adverse Event Reporting System (VAERS) data. The risks of local and systemic AEs were assessed from the three pivotal COVID-19 vaccine trials and also calculated in the VAERS cohort consisting of 559,717 reports between December 14, 2020 and September 17, 2021. AE rankings of the five vaccine groups calculated separately by RCT and VAERS were consistent, especially for systemic AEs. For severe AEs reported in VAERS, the reported risks of thrombosis and GBS after Janssen vaccine were highest. The reported risk of shingles after the first dose of Moderna vaccine was highest, followed by the second dose of the Moderna vaccine. The reported risk of myocarditis was higher after the second dose of Pfizer and Moderna vaccines. The reported risk of anaphylaxis was higher after the first dose of Pfizer vaccine. Limitations of this study are the inherent biases of the spontaneous reporting system data, and only including three pivotal RCTs and no comparison with other active vaccine safety surveillance systems.
Collapse
Affiliation(s)
- Chongliang Luo
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Division of Public Health Sciences, Washington University School of Medicine in St. Louis, St. Louis, MO, USA
| | - Jingcheng Du
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Adam Cuker
- Department of Medicine and Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Ebbing Lautenbach
- Division of Infectious Diseases, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Center for Clinical Epidemiology and Biostatistics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - David A Asch
- Division of General Internal Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Leonard Davis Institute of Health Economics, Philadelphia, PA, USA
| | - Gregory A Poland
- Mayo Clinic Vaccine Research Group, Mayo Clinic, Rochester, MN, USA
| | - Cui Tao
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Yong Chen
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA.
- Leonard Davis Institute of Health Economics, Philadelphia, PA, USA.
| |
Collapse
|
6
|
Luo C, Jiang Y, Du J, Tong J, Huang J, Lo Re V, Ellenberg SS, Poland GA, Tao C, Chen Y. Prediction of post-vaccination Guillain-Barré syndrome using data from a passive surveillance system. Pharmacoepidemiol Drug Saf 2021; 30:602-609. [PMID: 33533072 PMCID: PMC8014460 DOI: 10.1002/pds.5196] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2020] [Accepted: 01/11/2021] [Indexed: 01/21/2023]
Abstract
Purpose Severe adverse events (AEs), such as Guillain‐Barré syndrome (GBS) occur rarely after influenza vaccination. We identify highly associated AEs with GBS and develop prediction models for GBS using the US Vaccine Adverse Event Reporting System (VAERS) reports following trivalent influenza vaccination (FLU3). Methods This study analyzed 80 059 reports from the US VAERS between 1990 and 2017. Several AEs were identified as highly associated with GBS and were used to develop the prediction model. Some common and mild AEs that were suspected to be underreported when GBS occurred simultaneously were removed from the final model. The analyses were validated using European influenza vaccine AEs data from EudraVigilance. Results Of the 80 059 reports, 1185 (1.5%) were annotated as GBS related. Twenty‐four AEs were identified as having strong association with GBS. The full prediction model, using age, sex, and all 24 AEs achieved an area under the receiver operating characteristic (ROC) curve (AUC) of 85.4% (90% CI: [83.8%, 86.9%]). After excluding the nine (e.g., pruritus, rash, injection site pain) likely underreported AEs, the final AUC became 77.5% (90% CI: [75.5%, 79.6%]). Two hundred and one (0.25%) reports were predicted as of high risk of GBS (predicted probability >25%) and 84 actually developed GBS. Conclusion The prediction performance demonstrated the potential of developing risk‐prediction models utilizing the VAERS cohort. Excluding the likely underreported AEs sacrificed some prediction power but made the model more interpretable and feasible. The high absolute risk of even a small number of AE combinations suggests the promise of GBS prediction within the VAERS dataset.
Collapse
Affiliation(s)
- Chongliang Luo
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Ying Jiang
- Department of Neurology and Multiple Sclerosis Research Center, The Third Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Jingcheng Du
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Jiayi Tong
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Jing Huang
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Vincent Lo Re
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Susan S Ellenberg
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Gregory A Poland
- Mayo Clinic Vaccine Research Group, Mayo Clinic, Rochester, Minnesota, USA
| | - Cui Tao
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Yong Chen
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| |
Collapse
|
7
|
Huang Y, Luo C, Jiang Y, Du J, Tao C, Chen Y, Hao Y. Bayesian Network to Predict the Risk of Post Influenza Vaccination Guillain-Barré Syndrome: A Development and Validation Study (Preprint). JMIR Public Health Surveill 2020; 8:e25658. [PMID: 35333192 PMCID: PMC8994148 DOI: 10.2196/25658] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2020] [Revised: 12/27/2020] [Accepted: 02/02/2022] [Indexed: 11/18/2022] Open
Abstract
Background Identifying the key factors of Guillain-Barré syndrome (GBS) and predicting its occurrence are vital for improving the prognosis of patients with GBS. However, there are scarcely any publications on a forewarning model of GBS. A Bayesian network (BN) model, which is known to be an accurate, interpretable, and interaction-sensitive graph model in many similar domains, is worth trying in GBS risk prediction. Objective The aim of this study is to determine the most significant factors of GBS and further develop and validate a BN model for predicting GBS risk. Methods Large-scale influenza vaccine postmarketing surveillance data, including 79,165 US (obtained from the Vaccine Adverse Event Reporting System between 1990 and 2017) and 12,495 European (obtained from the EudraVigilance system between 2003 and 2016) adverse events (AEs) reports, were extracted for model development and validation. GBS, age, gender, and the top 50 prevalent AEs were included for initial BN construction using the R package bnlearn. Results Age, gender, and 10 AEs were identified as the most significant factors of GBS. The posttest probability of GBS suggested that male vaccinees aged 50-64 years and without erythema should be on the alert or be warned by clinicians about an increased risk of GBS, especially when they also experience symptoms of asthenia, hypesthesia, muscular weakness, or paresthesia. The established BN model achieved an area under the receiver operating characteristic curve of 0.866 (95% CI 0.865-0.867), sensitivity of 0.752 (95% CI 0.749-0.756), specificity of 0.882 (95% CI 0.879-0.885), and accuracy of 0.882 (95% CI 0.879-0.884) for predicting GBS risk during the internal validation and obtained values of 0.829, 0.673, 0.854, and 0.843 for area under the receiver operating characteristic curve, sensitivity, specificity, and accuracy, respectively, during the external validation. Conclusions The findings of this study illustrated that a BN model can effectively identify the most significant factors of GBS, improve understanding of the complex interactions among different postvaccination symptoms through its graphical representation, and accurately predict the risk of GBS. The established BN model could further assist clinical decision-making by providing an estimated risk of GBS for a specific vaccinee or be developed into an open-access platform for vaccinees’ self-monitoring.
Collapse
Affiliation(s)
- Yun Huang
- Department of Medical Statistics, Sun Yat-Sen University, Guangzhou, China
- Guangdong Provincial Center for Disease Control and Prevention, Guangzhou, China
| | - Chongliang Luo
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, United States
- Division of Public Health Sciences, Washington University School of Medicine in St. Louis, St. Louis, MO, United States
| | - Ying Jiang
- Department of Neurology and Multiple Sclerosis Research Center, The Third Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China
| | - Jingcheng Du
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, United States
| | - Cui Tao
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, United States
| | - Yong Chen
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, United States
| | - Yuantao Hao
- Department of Medical Statistics, Sun Yat-Sen University, Guangzhou, China
- Peking University Center for Public Health and Epidemic Preparedness & Response, Beijing, China
| |
Collapse
|