Artificial Intelligence for Unstructured Healthcare Data: Application to Coding of Patient Reporting of Adverse Drug Reactions

SMS-Based Active Surveillance of Adverse Events following Immunization in Children: The VigiVax Study

Underreporting is the main limitation of spontaneous reporting systems. This cohort-event monitoring study aims to examine the potential of short message service (SMS)-based surveillance compared to traditional surveillance systems. Using VigiVax software, parents of vaccinated children aged two years or younger, in the period March 2021-May 2022, received a single SMS inquiry about adverse events following immunization (AEFI). Responses were collected, validated by health operators and integrated with the information on electronic immunization registries. AEFI reports were automatically submitted to the Italian Pharmacovigilance system. Among 254,160 SMS messages sent, corresponding to 451,656 administered doses (AD), 71,643 responses were collected (28.2% response rate), and 21,231 of them (8.3%) reported AEFI. After a seriousness assessment based on clinical criteria, 50 reports (0.24%) were classified as serious. Among these, a causality assessment identified 31 reports at least potentially related to the vaccination (RR: 6.86/100,000 AD). Febrile seizures following MMRV (measles, mumps, rubella, varicella) vaccination accounted for 11 of these 31 cases, with an incidence of 32 per 100,000 AD. No fatal outcomes were reported. Our findings support the highly favorable risk profile of pediatric vaccinations and the possibility to improve spontaneous reporting through the integration of digital technologies.

An artificial intelligence algorithm for co-clustering to help in pharmacovigilance before and during the COVID-19 pandemic

AIMS

Monitoring drug safety in real-world settings is the primary aim of pharmacovigilance. Frequent adverse drug reactions (ADRs) are usually identified during drug development. Rare ones are mostly characterized through post-marketing scrutiny, increasingly with the use of data mining and disproportionality approaches, which lead to new drug safety signals. Nonetheless, waves of excessive numbers of reports, often stirred up by social media, may overwhelm and distort this process, as observed recently with levothyroxine or COVID-19 vaccines. As human resources become rarer in the field of pharmacovigilance, we aimed to evaluate the performance of an unsupervised co-clustering method to help the monitoring of drug safety.

METHODS

A dynamic latent block model (dLBM), based on a time-dependent co-clustering generative method, was used to summarize all regional ADR reports (n = 45 269) issued between 1 January 2012 and 28 February 2022. After analysis of their intra and extra interrelationships, all reports were grouped into different cluster types (time, drug, ADR).

RESULTS

Our model clustered all reports in 10 time, 10 ADR and 9 drug collections. Based on such clustering, three prominent societal problems were detected, subsequent to public health concerns about drug safety, including a prominent media hype about the perceived safety of COVID-19 vaccines. The dLBM also highlighted some specific drug-ADR relationships, such as the association between antiplatelets, anticoagulants and bleeding.

CONCLUSIONS

Co-clustering and dLBM appear as promising tools to explore large pharmacovigilance databases. They allow, 'unsupervisedly', the detection, exploration and strengthening of safety signals, facilitating the analysis of massive upsurges of reports.

Disponibilité de l'information médicale requise pour la déclaration d'une réaction indésirable médicamenteuse à Santé Canada: une étude exploratoire

Background

Since 2019, health care facilities have been required to report serious adverse drug reactions (ADRs) to Health Canada.

Objectives

To describe the availability of information required for reporting an ADR to Health Canada from medical records using 2 methods (systematic and in-depth reporting) and to compare the time required to find the information.

Methods

This retrospective and prospective descriptive study involved serious ADRs occurring in a mother-child centre and reported between April 1, 2021, and March 31, 2023. The variables needed to complete the Health Canada reporting form were collected using 2 distinct methods.

Results

Among the 270 serious ADRs reported retrospectively, 140 were sampled. The average availability of variables was 82.3% (standard deviation [SD] 11.3%), with average data collection time of 50 (SD 25) minutes. For the prospective part of the study, 15 serious ADRs were studied. The availability of variables was 82.8% (SD 6.9%) and 91.9% (SD 7.8%), for systematic and in-depth reporting, respectively, with data collection times of 44 (SD 17) and 130 (SD 33) minutes, respectively.

Conclusions

The challenge of finding, in patients' medical records, all of the information needed for reporting an ADR to Health Canada required an in-depth approach. However, the in-depth method took 3 times as long as a search limited to places in the record where specific information should be found. To improve record keeping, additional training for clinicians could be considered and, potentially, development of a computerized clinical record that includes a dedicated form for documenting ADRs.

BERT based natural language processing for triage of adverse drug reaction reports shows close to human-level performance

Post-marketing reports of suspected adverse drug reactions are important for establishing the safety profile of a medicinal product. However, a high influx of reports poses a challenge for regulatory authorities as a delay in identification of previously unknown adverse drug reactions can potentially be harmful to patients. In this study, we use natural language processing (NLP) to predict whether a report is of serious nature based solely on the free-text fields and adverse event terms in the report, potentially allowing reports mislabelled at time of reporting to be detected and prioritized for assessment. We consider four different NLP models at various levels of complexity, bootstrap their train-validation data split to eliminate random effects in the performance estimates and conduct prospective testing to avoid the risk of data leakage. Using a Swedish BERT based language model, continued language pre-training and final classification training, we achieve close to human-level performance in this task. Model architectures based on less complex technical foundation such as bag-of-words approaches and LSTM neural networks trained with random initiation of weights appear to perform less well, likely due to the lack of robustness that a base of general language training provides.

Will the future of pharmacovigilance be more automated?

INTRODUCTION

Artificial intelligence (AI) based tools offer new opportunities for pharmacovigilance (PV) activities. Nevertheless, their contribution to PV needs to be tailored to preserve and strengthen medical and pharmacological expertise in drug safety.

AREAS COVERED

This work aims to describe PV tasks in which the contribution of AI and intelligent automation (IA) tools is required, in the context of a continuous increase of spontaneous reporting cases and regulatory tasks. A narrative review with expert selection of pertinent references was performed through Medline. Two areas were covered, management of spontaneous reporting cases and signal detection.

PERSPECTIVE

The use of AI and IA tools will assist a large spectrum of PV activities, both in public and private PV systems, in particular for tasks of low added value (e.g. initial quality check, verification of essential regulatory information, search for duplicates). Testing, validating, and integrating these tools in the PV routine are the actual challenges for modern PV systems, to guarantee high-quality standards in terms of case management and signal detection.

What place for intelligent automation and artificial intelligence to preserve and strengthen vigilance expertise in the face of increasing declarations?

In 2018, the "Ateliers de Giens" (Giens Workshops) devoted a workshop to artificial intelligence (AI) and led its experts to confirm the potential contribution and theoretical benefit of AI in clinical research, pharmacovigilance, and in improving the efficiency of care. The 2022 workshop is a continuation of this reflection on AI and intelligent automation (IA) by focusing on its contribution to pharmacovigilance and the applications and tasks could be optimized to preserve and strengthen medical and pharmacological expertise in pharmacovigilance. The evolution of pharmacovigilance work is characterized by many tasks with low added value, a growing volume of pharmacovigilance reporting of suspected side effects, and a scarcity of medical staff with expertise in clinical pharmacology and pharmacovigilance and human resources to support this growing need. Together, these parameters contribute to an embolization of the pharmacovigilance system at risk of missing its primary mission: to identify and characterize a risk or even a health alert on a drug. The participants of the workshop (representatives of the Regional Pharmacovigilance Centres (CRPV), the French National Agency for Safety of Medicinal Products (ANSM), patients, the pharmaceutical industry, or start-ups working in the development of AI in the field of medicine) shared their experiences, their pilot projects and their expectations on the expected potential, theoretical or proven, AI and IA. This work has made it possible to identify the needs and challenges that AI or IA represent, in the current or future modes of organization of pharmacovigilance activities. This approach led to the development of a SWOT matrix (strengths, weaknesses, opportunities, threats), a basis for reflection to identify critical points and consider four main recommendations: (1) preserve and develop business expertise in pharmacovigilance (including research and development in methods) with the integration of new technologies; (2) improve the quality of pharmacovigilance reports; (3) adapt technical and regulatory means; (4) implement a development strategy for AI and IA tools at the service of expertise.

Artificial Intelligence in Pharmacovigilance: An Introduction to Terms, Concepts, Applications, and Limitations

The tools of artificial intelligence (AI) have enormous potential to enhance activities in pharmacovigilance. Pharmacovigilance experts need not be AI experts, but they should know enough about AI to explore the possibilities of collaboration with those who are. Modern concepts of AI date from Alan Turing's work, especially his paper on "the imitation game", in the late 1940s and early 1950s. Its scope today includes computational skills, including the formulation of mathematical proofs; visual perception, including facial recognition and virtual reality; decision making by expert systems; aspects of language, such as language processing, speech recognition, creative composition, and translation; and combinations of these, e.g. in self-driving vehicles. Machines can be programmed with the ability to learn, using neural networks that mimic cognitive actions of the human brain, leading to deep structural learning. Limitations of AI include difficulties with language, arising from the need to understand context and interpret ambiguities, which particularly affect translation, and inadequacies of databases, requiring careful preparation and curation. New techniques may cause unforeseen difficulties via unexpected malfunctioning. Relevant terms and concepts include different types of machine learning, neural networks, natural language programming, ontologies, and expert systems. Adoption of the tools of AI in pharmacovigilance has been slow. Machine learning, in conjunction with natural language processing and data mining, to study adverse drug reactions in databases such as those found in electronic health records, claims databases, and social media, has the potential to enhance the characterization of known adverse effects and reactions and detect new signals.

Industry Perspective on Artificial Intelligence/Machine Learning in Pharmacovigilance

TransCelerate reports on the results of 2019, 2020, and 2021 member company (MC) surveys on the use of intelligent automation in pharmacovigilance processes. MCs increased the number and extent of implementation of intelligent automation solutions throughout Individual Case Safety Report (ICSR) processing, especially with rule-based automations such as robotic process automation, lookups, and workflows, moving from planning to piloting to implementation over the 3 survey years. Companies remain highly interested in other technologies such as machine learning (ML) and artificial intelligence, which can deliver a human-like interpretation of data and decision making rather than just automating tasks. Intelligent automation solutions are usually used in combination with more than one technology being used simultaneously for the same ICSR process step. Challenges to implementing intelligent automation solutions include finding/having appropriate training data for ML models and the need for harmonized regulatory guidance.

Application of artificial intelligence in a real-world research for predicting the risk of liver metastasis in T1 colorectal cancer

Background

Liver is the most common metastatic site of colorectal cancer (CRC) and liver metastasis (LM) determines subsequent treatment as well as prognosis of patients, especially in T1 patients. T1 CRC patients with LM are recommended to adopt surgery and systematic treatments rather than endoscopic therapy alone. Nevertheless, there is still no effective model to predict the risk of LM in T1 CRC patients. Hence, we aim to construct an accurate predictive model and an easy-to-use tool clinically.

Methods

We integrated two independent CRC cohorts from Surveillance Epidemiology and End Results database (SEER, training dataset) and Xijing hospital (testing dataset). Artificial intelligence (AI) and machine learning (ML) methods were adopted to establish the predictive model.

Results

A total of 16,785 and 326 T1 CRC patients from SEER database and Xijing hospital were incorporated respectively into the study. Every single ML model demonstrated great predictive capability, with an area under the curve (AUC) close to 0.95 and a stacking bagging model displaying the best performance (AUC = 0.9631). Expectedly, the stacking model exhibited a favorable discriminative ability and precisely screened out all eight LM cases from 326 T1 patients in the outer validation cohort. In the subgroup analysis, the stacking model also demonstrated a splendid predictive ability for patients with tumor size ranging from one to50mm (AUC = 0.956).

Conclusion

We successfully established an innovative and convenient AI model for predicting LM in T1 CRC patients, which was further verified in the external dataset. Ultimately, we designed a novel and easy-to-use decision tree, which only incorporated four fundamental parameters and could be successfully applied in clinical practice.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12935-021-02424-7.

Development and validation of a machine learning-augmented algorithm for diabetes screening in community and primary care settings: A population-based study

BACKGROUND

Opportunely screening for diabetes is crucial to reduce its related morbidity, mortality, and socioeconomic burden. Machine learning (ML) has excellent capability to maximize predictive accuracy. We aim to develop ML-augmented models for diabetes screening in community and primary care settings.

METHODS

8425 participants were involved from a population-based study in Hubei, China since 2011. The dataset was split into a development set and a testing set. Seven different ML algorithms were compared to generate predictive models. Non-laboratory features were employed in the ML model for community settings, and laboratory test features were further introduced in the ML+lab models for primary care. The area under the receiver operating characteristic curve (AUC), area under the precision-recall curve (auPR), and the average detection costs per participant of these models were compared with their counterparts based on the New China Diabetes Risk Score (NCDRS) currently recommended for diabetes screening.

RESULTS

The AUC and auPR of the ML model were 0·697and 0·303 in the testing set, seemingly outperforming those of NCDRS by 10·99% and 64·67%, respectively. The average detection cost of the ML model was 12·81% lower than that of NCDRS with the same sensitivity (0·72). Moreover, the average detection cost of the ML+FPG model is the lowest among the ML+lab models and less than that of the ML model and NCDRS+FPG model.

CONCLUSION

The ML model and the ML+FPG model achieved higher predictive accuracy and lower detection costs than their counterpart based on NCDRS. Thus, the ML-augmented algorithm is potential to be employed for diabetes screening in community and primary care settings.

Validation of Artificial Intelligence to Support the Automatic Coding of Patient Adverse Drug Reaction Reports, Using Nationwide Pharmacovigilance Data

INTRODUCTION

Adverse drug reaction reports are usually manually assessed by pharmacovigilance experts to detect safety signals associated with drugs. With the recent extension of reporting to patients and the emergence of mass media-related sanitary crises, adverse drug reaction reports currently frequently overwhelm pharmacovigilance networks. Artificial intelligence could help support the work of pharmacovigilance experts during such crises, by automatically coding reports, allowing them to prioritise or accelerate their manual assessment. After a previous study showing first results, we developed and compared state-of-the-art machine learning models using a larger nationwide dataset, aiming to automatically pre-code patients' adverse drug reaction reports.

OBJECTIVES

We aimed to determine the best artificial intelligence model identifying adverse drug reactions and assessing seriousness in patients reports from the French national pharmacovigilance web portal.

METHODS

Reports coded by 27 Pharmacovigilance Centres between March 2017 and December 2020 were selected (n = 11,633). For each report, the Portable Document Format form containing free-text information filled by the patient, and the corresponding encodings of adverse event symptoms (in Medical Dictionary for Regulatory Activities Preferred Terms) and seriousness were obtained. This encoding by experts was used as the reference to train and evaluate models, which contained input data processing and machine-learning natural language processing to learn and predict encodings. We developed and compared different approaches for data processing and classifiers. Performance was evaluated using receiver operating characteristic area under the curve (AUC), F-measure, sensitivity, specificity and positive predictive value. We used data from 26 Pharmacovigilance Centres for training and internal validation. External validation was performed using data from the remaining Pharmacovigilance Centres during the same period.

RESULTS

Internal validation: for adverse drug reaction identification, Term Frequency-Inverse Document Frequency (TF-IDF) + Light Gradient Boosted Machine (LGBM) achieved an AUC of 0.97 and an F-measure of 0.80. The Cross-lingual Language Model (XLM) [transformer] obtained an AUC of 0.97 and an F-measure of 0.78. For seriousness assessment, FastText + LGBM achieved an AUC of 0.85 and an F-measure of 0.63. CamemBERT (transformer) + Light Gradient Boosted Machine obtained an AUC of 0.84 and an F-measure of 0.63. External validation for both adverse drug reaction identification and seriousness assessment tasks yielded consistent and robust results.

CONCLUSIONS

Our artificial intelligence models showed promising performance to automatically code patient adverse drug reaction reports, with very similar results across approaches. Our system has been deployed by national health authorities in France since January 2021 to facilitate pharmacovigilance of COVID-19 vaccines. Further studies will be needed to validate the performance of the tool in real-life settings.

Black Swan Events and Intelligent Automation for Routine Safety Surveillance

Effective identification of previously implausible safety signals is a core component of successful pharmacovigilance. Timely, reliable, and efficient data ingestion and related processing are critical to this. The term 'black swan events' was coined by Taleb to describe events with three attributes: unpredictability, severe and widespread consequences, and retrospective bias. These rare events are not well understood at their emergence but are often rationalized in retrospect as predictable. Pharmacovigilance strives to rapidly respond to potential black swan events associated with medicine or vaccine use. Machine learning (ML) is increasingly being explored in data ingestion tasks. In contrast to rule-based automation approaches, ML can use historical data (i.e., 'training data') to effectively predict emerging data patterns and support effective data intake, processing, and organisation. At first sight, this reliance on previous data might be considered a limitation when building ML models for effective data ingestion in systems that look to focus on the identification of potential black swan events. We argue that, first, some apparent black swan events-although unexpected medically-will exhibit data attributes similar to those of other safety data and not prove algorithmically unpredictable, and, second, standard and emerging ML approaches can still be robust to such data outliers with proper awareness and consideration in ML system design and with the incorporation of specific mitigatory and support strategies. We argue that effective approaches to managing data on potential black swan events are essential for trust and outline several strategies to address data on potential black swan events during data ingestion.

Intelligent Telehealth in Pharmacovigilance: A Future Perspective

Pharmacovigilance improves patient safety by detecting and preventing adverse drug events. However, challenges exist that limit adverse drug event detection, resulting in many adverse drug events being underreported or inaccurately reported. One challenge includes having access to large data sets from various sources including electronic health records and wearable medical devices. Artificial intelligence, including machine learning methods, such as natural language processing and deep learning, can detect and extract information about adverse drug events, thus automating the pharmacovigilance process and improving the surveillance of known and documented adverse drug events. In addition, with the increased demand for telehealth services, for managing both acute and chronic diseases, artificial intelligence methods can play a role in detecting and preventing adverse drug events. In this review, we discuss two use cases of how artificial intelligence methods may be useful to improve the quality of pharmacovigilance and the role of artificial intelligence in telehealth practices.

Detection of Drug-Induced Thrombocytopenia Signals in Children Using Routine Electronic Medical Records

Background: Drug-induced thrombocytopenia (DITP) is a severe adverse reaction and a significantly under-recognized clinical problem in children. However, for post-marketing pharmacovigilance purposes, detection of DITP signals is crucial. This study aimed to develop a signal detection model for DITP using the pediatric electronic medical records (EMR) data.

Methods: This study used the electronic medical records collected at Beijing Children’s Hospital between 2009 and 2020. A two-stage modeling method was developed to detect the signal of DITP. In the first stage, we calculated the crude incidence by mining cases of thrombocytopenia to select the potential suspected drugs. In the second stage, we constructed propensity score–matched retrospective cohorts of specific screened drugs from the first stage and estimated the odds ratio (OR) and 95% confidence interval (CI) using conditional logistic regression models. The novelty of the signal was assessed by current evidence.

Results: In the study, from a total of 839 drugs, 21 drugs were initially screened as potentially inducing thrombocytopenia. In total, we identified 18 positive DITP associations. Of these, potential DITP risk of nystatin (OR: 1.75, 95% CI: 1.37–2.22) and latamoxef sodium (OR: 1.61, 95% CI: 1.38–1.88) were two new DITP signals in both children and adults. Six associations between thrombocytopenia and drugs including imipenem (OR: 1.69, 95% CI: 1.16–2.45), teicoplanin (OR: 4.75, 95% CI: 3.33–6.78), fusidic acid (OR: 2.81, 95% CI: 2.06–3.86), ceftizoxime sodium (OR: 1.83, 95% CI: 1.36–2.45), ceftazidime (OR: 2.16, 95% CI: 1.58–2.95), and cefepime (OR: 5.06, 95% CI: 3.77–6.78) were considered as new signals in children.

Conclusion: This study developed a two-stage algorithm to detect safety signals of DITP and found eighteen positive signals of DITP, including six new signals in a pediatric population. This method is a promising tool for pharmacovigilance based on EMR data.