1
|
Wang X, Xu X, Liu Z, Tong W. Bidirectional Encoder Representations from Transformers-like large language models in patient safety and pharmacovigilance: A comprehensive assessment of causal inference implications. Exp Biol Med (Maywood) 2023; 248:1908-1917. [PMID: 38084745 PMCID: PMC10798182 DOI: 10.1177/15353702231215895] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Accepted: 10/24/2023] [Indexed: 01/06/2024] Open
Abstract
Causality assessment is vital in patient safety and pharmacovigilance (PSPV) for safety signal detection, adverse reaction management, and regulatory submission. Large language models (LLMs), especially those designed with transformer architecture, are revolutionizing various fields, including PSPV. While attempts to utilize Bidirectional Encoder Representations from Transformers (BERT)-like LLMs for causal inference in PSPV are underway, a detailed evaluation of "fit-for-purpose" BERT-like model selection to enhance causal inference performance within PSPV applications remains absent. This study conducts an in-depth exploration of BERT-like LLMs, including generic pre-trained BERT LLMs, domain-specific pre-trained LLMs, and domain-specific pre-trained LLMs with safety knowledge-specific fine-tuning, for causal inference in PSPV. Our investigation centers around (1) the influence of data complexity and model architecture, (2) the correlation between the BERT size and its impact, and (3) the role of domain-specific training and fine-tuning on three publicly accessible PSPV data sets. The findings suggest that (1) BERT-like LLMs deliver consistent predictive power across varied data complexity levels, (2) the predictive performance and causal inference results do not directly correspond to the BERT-like model size, and (3) domain-specific pre-trained LLMs, with or without safety knowledge-specific fine-tuning, surpass generic pre-trained BERT models in causal inference. The findings are valuable to guide the future application of LLMs in a broad range of application.
Collapse
Affiliation(s)
- Xingqiao Wang
- Department of Information Science, University of Arkansas at Little Rock, Little Rock, AR 72204, USA
| | - Xiaowei Xu
- Department of Information Science, University of Arkansas at Little Rock, Little Rock, AR 72204, USA
| | - Zhichao Liu
- Nonclinical Drug Safety, Boehringer Ingelheim Pharmaceuticals, Inc., Ridgefield, CT 06877, USA
| | - Weida Tong
- FDA/National Center for Toxicological Research, Jefferson, AR 72079, USA
| |
Collapse
|
2
|
Salvo F, Micallef J, Lahouegue A, Chouchana L, Létinier L, Faillie JL, Pariente A. Will the future of pharmacovigilance be more automated? Expert Opin Drug Saf 2023; 22:541-548. [PMID: 37435796 DOI: 10.1080/14740338.2023.2227091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2023] [Accepted: 06/15/2023] [Indexed: 07/13/2023]
Abstract
INTRODUCTION Artificial intelligence (AI) based tools offer new opportunities for pharmacovigilance (PV) activities. Nevertheless, their contribution to PV needs to be tailored to preserve and strengthen medical and pharmacological expertise in drug safety. AREAS COVERED This work aims to describe PV tasks in which the contribution of AI and intelligent automation (IA) tools is required, in the context of a continuous increase of spontaneous reporting cases and regulatory tasks. A narrative review with expert selection of pertinent references was performed through Medline. Two areas were covered, management of spontaneous reporting cases and signal detection. PERSPECTIVE The use of AI and IA tools will assist a large spectrum of PV activities, both in public and private PV systems, in particular for tasks of low added value (e.g. initial quality check, verification of essential regulatory information, search for duplicates). Testing, validating, and integrating these tools in the PV routine are the actual challenges for modern PV systems, to guarantee high-quality standards in terms of case management and signal detection.
Collapse
Affiliation(s)
- Francesco Salvo
- University of Bordeaux, Inserm, BPH, Team AHeaD, Bordeaux, France
- CHU de Bordeaux, Service de Pharmacologie Medicale, Bordeaux, France
| | - Joelle Micallef
- Pharmacovigilance Centre, Department of Clinical Pharmacology and Pharmacovigilance, University of Aix Marseille, INSERM UMR 1106 Institut de Neurosciences des Systèmes, Marseille, France
| | - Amir Lahouegue
- Department of Pharmacovigilance and Medical Information, Astrazeneca, Courbevoie, France
| | - Laurent Chouchana
- Regional Center of Pharmacovigilance, Pharmacology Department, Cochin Port Royal University Hospital, Paris, France
| | - Louis Létinier
- University of Bordeaux, Inserm, BPH, Team AHeaD, Bordeaux, France
- CHU de Bordeaux, Service de Pharmacologie Medicale, Bordeaux, France
- Synapse Medicine, Bordeaux, France
| | - Jean-Luc Faillie
- Inserm, Departement de Pharmacologie Medicale Et Toxicologie, Centre Regional de PV, Institut Desbrest D'epidemiologie Et de Sante Publique, CHU de Montpellier, Universite Montpellier, Montpellier, France
| | - Antoine Pariente
- University of Bordeaux, Inserm, BPH, Team AHeaD, Bordeaux, France
- CHU de Bordeaux, Service de Pharmacologie Medicale, Bordeaux, France
| |
Collapse
|
3
|
Patera AC, Maidment J, Maroj B, Mohamed A, Twomey K. A Science-Based Methodology Framework for the Assessment of Combination Safety Risks in Clinical Trials. Pharmaceut Med 2023; 37:183-202. [PMID: 37099245 DOI: 10.1007/s40290-023-00465-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/26/2023] [Indexed: 04/27/2023]
Abstract
Multiple components factor into the assessment of combination safety risks when two or more novel individual products are used in combination in clinical trials. These include, but are not limited to, biology, biochemistry, pharmacology, class effects, and preclinical and clinical findings (such as adverse drug reactions, drug target and mechanism of action, target expression, signaling, and drug-drug interactions). This paper presents a science-based methodology framework for the assessment of combination safety risks when two or more investigational products are used in clinical trials. The aim of this methodology framework is to improve prediction of the risks, to enable the appropriate safety risk mitigation and management to be put in place for the combination, and the development of the project combination safety strategy.
Collapse
Affiliation(s)
- Andriani C Patera
- Patient Safety Oncology, Oncology R&D, AstraZeneca, 101 Orchard Ridge Way, Gaithersburg, MD, 20878, USA.
| | - Julie Maidment
- Patient Safety Oncology, Oncology R&D, AstraZeneca, Cambridge, UK
| | - Brijesh Maroj
- Patient Safety Oncology, Oncology R&D, AstraZeneca, Cambridge, UK
| | - Ahmed Mohamed
- Patient Safety Oncology, Oncology R&D, AstraZeneca, 101 Orchard Ridge Way, Gaithersburg, MD, 20878, USA
| | - Ken Twomey
- Patient Safety Oncology, Oncology R&D, AstraZeneca, Cambridge, UK
| |
Collapse
|
4
|
Murphy RM, Dongelmans DA, Kom IYD, Calixto I, Abu-Hanna A, Jager KJ, de Keizer NF, Klopotowska JE. Drug-related causes attributed to acute kidney injury and their documentation in intensive care patients. J Crit Care 2023; 75:154292. [PMID: 36959015 DOI: 10.1016/j.jcrc.2023.154292] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Revised: 03/14/2023] [Accepted: 03/14/2023] [Indexed: 03/25/2023]
Abstract
PURPOSE To investigate drug-related causes attributed to acute kidney injury (DAKI) and their documentation in patients admitted to the Intensive Care Unit (ICU). METHODS This study was conducted in an academic hospital in the Netherlands by reusing electronic health record (EHR) data of adult ICU admissions between November 2015 to January 2020. First, ICU admissions with acute kidney injury (AKI) stage 2 or 3 were identified. Subsequently, three modes of DAKI documentation in EHR were examined: diagnosis codes (structured data), allergy module (semi-structured data), and clinical notes (unstructured data). RESULTS n total 8124 ICU admissions were included, with 542 (6.7%) ICU admissions experiencing AKI stage 2 or 3. The ICU physicians deemed 102 of these AKI cases (18.8%) to be drug-related. These DAKI cases were all documented in the clinical notes (100%), one in allergy module (1%) and none via diagnosis codes. The clinical notes required the highest time investment to analyze. CONCLUSIONS Drug-related causes comprise a substantial part of AKI in the ICU patients. However, current unstructured DAKI documentation practice via clinical notes hampers our ability to gain better insights about DAKI occurrence. Therefore, both automating DAKI identification from the clinical notes and increasing structured DAKI documentation should be encouraged.
Collapse
Affiliation(s)
- Rachel M Murphy
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health, Digital Health, Amsterdam, the Netherlands; Amsterdam Public Health, Quality of Care, Amsterdam, the Netherlands.
| | - Dave A Dongelmans
- Amsterdam Public Health, Quality of Care, Amsterdam, the Netherlands; Amsterdam UMC location University of Amsterdam, Department of Intensive Care Medicine, Meibergdreef 9, Amsterdam, the Netherlands
| | - Izak Yasrebi-de Kom
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health, Methodology, Amsterdam, the Netherlands
| | - Iacer Calixto
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health, Methodology, Amsterdam, the Netherlands; Amsterdam Public Health, Mental Health, Amsterdam, the Netherlands
| | - Ameen Abu-Hanna
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health, Methodology, Amsterdam, the Netherlands; Amsterdam Public Health, Aging & Later Life, Amsterdam, the Netherlands
| | - Kitty J Jager
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health, Quality of Care, Amsterdam, the Netherlands; Amsterdam Public Health, Aging & Later Life, Amsterdam, the Netherlands; Amsterdam Cardiovascular Sciences, Pulmonary hypertension & thrombosis, Amsterdam, the Netherlands
| | - Nicolette F de Keizer
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health, Digital Health, Amsterdam, the Netherlands; Amsterdam Public Health, Quality of Care, Amsterdam, the Netherlands
| | - Joanna E Klopotowska
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health, Digital Health, Amsterdam, the Netherlands; Amsterdam Public Health, Quality of Care, Amsterdam, the Netherlands
| |
Collapse
|
5
|
Yalçın N, Kaşıkcı M, Çelik HT, Allegaert K, Demirkan K, Yiğit Ş, Yurdakök M. Development and validation of a machine learning-based detection system to improve precision screening for medication errors in the neonatal intensive care unit. Front Pharmacol 2023; 14:1151560. [PMID: 37124199 PMCID: PMC10140576 DOI: 10.3389/fphar.2023.1151560] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Accepted: 04/04/2023] [Indexed: 05/02/2023] Open
Abstract
Aim: To develop models that predict the presence of medication errors (MEs) (prescription, preparation, administration, and monitoring) using machine learning in NICU patients. Design: Prospective, observational cohort study randomized with machine learning (ML) algorithms. Setting: A 22-bed capacity NICU in Ankara, Turkey, between February 2020 and July 2021. Results: A total of 11,908 medication orders (28.9 orders/patient) for 412 NICU patients (5.53 drugs/patient/day) who received 2,280 prescriptions over 32,925 patient days were analyzed. At least one physician-related ME and nurse-related ME were found in 174 (42.2%) and 235 (57.0%) of the patients, respectively. The parameters that had the highest correlation with ME occurrence and subsequently included in the model were: total number of drugs, anti-infective drugs, nervous system drugs, 5-min APGAR score, postnatal age, alimentary tract and metabolism drugs, and respiratory system drugs as patient-related parameters, and weekly working hours of nurses, weekly working hours of physicians, and number of nurses' monthly shifts as care provider-related parameters. The obtained model showed high performance to predict ME (AUC: 0.920; 95% CI: 0.876-0.970) presence and is accessible online (http://softmed.hacettepe.edu.tr/NEO-DEER_Medication_Error/). Conclusion: This is the first developed and validated model to predict the presence of ME using work environment and pharmacotherapy parameters with high-performance ML algorithms in NICU patients. This approach and the current model hold the promise of implementation of targeted/precision screening to prevent MEs in neonates. Clinical Trial Registration: ClinicalTrials.gov, identifier NCT04899960.
Collapse
Affiliation(s)
- Nadir Yalçın
- Department of Clinical Pharmacy, Faculty of Pharmacy, Hacettepe University, Ankara, Türkiye
- *Correspondence: Nadir Yalçın,
| | - Merve Kaşıkcı
- Department of Biostatistics, Faculty of Medicine, Hacettepe University, Ankara, Türkiye
| | - Hasan Tolga Çelik
- Division of Neonatology, Department of Child Health and Diseases, Faculty of Medicine, Hacettepe University, Ankara, Türkiye
| | - Karel Allegaert
- Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Belgium
- Department of Development and Regeneration, KU Leuven, Belgium
- Department of Hospital Pharmacy, Erasmus Medical Center, Rotterdam, Netherlands
| | - Kutay Demirkan
- Department of Clinical Pharmacy, Faculty of Pharmacy, Hacettepe University, Ankara, Türkiye
| | - Şule Yiğit
- Division of Neonatology, Department of Child Health and Diseases, Faculty of Medicine, Hacettepe University, Ankara, Türkiye
| | - Murat Yurdakök
- Division of Neonatology, Department of Child Health and Diseases, Faculty of Medicine, Hacettepe University, Ankara, Türkiye
| |
Collapse
|