A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis

Accuracy of a Proprietary Large Language Model in Labeling Obstetric Incident Reports

BACKGROUND

Using the data collected through incident reporting systems is challenging, as it is a large volume of primarily qualitative information. Large language models (LLMs), such as ChatGPT, provide novel capabilities in text summarization and labeling that could support safety data trending and early identification of opportunities to prevent patient harm. This study assessed the capability of a proprietary LLM (GPT-3.5) to automatically label a cross-sectional sample of real-world obstetric incident reports.

METHODS

A sample of 370 incident reports submitted to inpatient obstetric units between December 2022 and May 2023 was extracted. Human-annotated labels were assigned by a clinician reviewer and considered gold standard. The LLM was prompted to label incident reports relying solely on its pretrained knowledge and information included in the prompt. Primary outcomes assessed were sensitivity, specificity, positive predictive value, and negative predictive value. A secondary outcome assessed the human-perceived quality of the model's justification for the label(s) applied.

RESULTS

The LLM demonstrated the ability to label incident reports with high sensitivity and specificity. The model applied a total of 79 labels compared to the reviewer's 49 labels. Overall sensitivity for the model was 85.7%, and specificity was 97.9%. Positive and negative predictive values were 53.2% and 99.6%, respectively. For 60.8% of labels, the reviewer approved of the model's justification for applying the label.

CONCLUSION

The proprietary LLM demonstrated the ability to label obstetric incident reports with high sensitivity and specificity. LLMs offer the potential to enable more efficient use of data from incident reporting systems.

The Potential of Using Generative AI/NLP to Identify and Analyse Critical Incidents in a Critical Incident Reporting System (CIRS): A Feasibility Case-Control Study

BACKGROUND

To enhance patient safety in healthcare, it is crucial to address the underreporting of issues in Critical Incident Reporting Systems (CIRSs). This study aims to evaluate the effectiveness of generative Artificial Intelligence and Natural Language Processing (AI/NLP) in reviewing CIRS cases by comparing its performance with human reviewers and categorising these cases into relevant topics.

METHODS

A case-control feasibility study was conducted using CIRS cases from the German CIRS-Anaesthesiology subsystem. Each case was reviewed by a human expert and by an AI/NLP model (ChatGPT-3.5). Two CIRS experts blindly assessed these reviews, rating them on linguistic quality, recognisable expertise, logical derivability, and overall quality using six-point Likert scales.

RESULTS

On average, the CIRS experts correctly classified 80% of human CIRS reviews as created by a human and misclassified 45.8% of AI reviews as written by a human. Ratings on a scale of 1 (very good) to 6 (failed) revealed a comparable performance between human- and AI-generated reviews across the dimensions of linguistic expression (p = 0.39), recognisable expertise (p = 0.89), logical derivability (p = 0.84), and overall quality (p = 0.87). The AI model was able to categorise the cases into relevant topics independently.

CONCLUSIONS

This feasibility study demonstrates the potential of generative AI/NLP in analysing and categorising cases from the CIRS. This could have implications for improving incident reporting in healthcare. Therefore, additional research is required to verify and expand upon these discoveries.

Development and Evaluation of Patient Safety Interventions: Perspectives of Operational Safety Leaders and Patient Safety Organizations

OBJECTIVES

The purpose of this study is to understand how patient safety professionals from healthcare facilities and patient safety organizations develop patient safety interventions and the resources used to support intervention development.

METHODS

Semistructured interviews were conducted with patient safety professionals at nine healthcare facilities and nine patient safety organizations. Interview data were qualitatively analyzed, and findings were organized by the following: patient safety solutions and interventions, use of external databases, and evaluation of patient safety solutions.

RESULTS

Development of patient safety interventions across healthcare facilities and patient safety organizations was similar and included literature searches, internal brainstorming, and interviews. Nearly all patient safety professionals at healthcare facilities reported contacting colleagues at other healthcare facilities to learn about similar safety issues and potential interventions. Additionally, less than half of patient safety professionals at healthcare facilities and patient safety organizations interviewed report data to publicly available patient safety databases. Finally, most patient safety professionals at healthcare facilities and patient safety organizations stated that they evaluate the effectiveness of patient safety interventions; however, they mentioned methods that may be less rigorous including audits, self-reporting, and subjective judgment.

CONCLUSIONS

Patient safety professionals often utilize similar methods and resources to develop and evaluate patient safety interventions; however, many of these efforts are not coordinated across healthcare organizations and could benefit from working collectively in a systematic fashion. Additionally, healthcare facilities and patient safety organizations face similar challenges and there are several opportunities for optimization on a national level that may improve patient safety.

Validation of a natural language processing algorithm using national reporting data to improve identification of anesthesia-related ADVerse evENTs: The "ADVENTURE" study

BACKGROUND

Reporting and analysis of adverse events (AE) is associated with improved health system learning, quality outcomes, and patient safety. Manual text analysis is time-consuming, costly, and prone to human errors. We aimed to demonstrate the feasibility of novel machine learning and natural language processing (NLP) approaches for early predictions of adverse events and provide input to direct quality improvement and patient safety initiatives.

METHODS

We used machine learning to analyze 9559 continuously reported AE by clinicians and healthcare systems to the French National Health accreditor (HAS) between January 1, 2009, and December 31, 2020 . We validated the labeling of 135,000 unique de-identified AE reports and determined the associations between different system's root causes and patient consequences. The model was validated by independent expert anesthesiologists.

RESULTS

The machine learning (ML) and Artificial Intelligence (AI) model trained on 9559 AE datasets accurately categorized 8800 (88%) of reported AE. The three most frequent AE types were "difficult orotracheal intubation" (16.9% of AE reports), "medication error" (10.5%), and "post-induction hypotension" (6.9%). The accuracy of the AI model reached 70.9% sensitivity, 96.6% specificity for "difficult intubation", 43.2% sensitivity, and 98.9% specificity for "medication error."

CONCLUSIONS

This unsupervised ML method provides an accurate, automated, AI-supported search algorithm that ranks and helps to understand complex risk patterns and has greater speed, precision, and clarity when compared to manual human data extraction. Machine learning and Natural language processing (NLP) models can effectively be used to process natural language AE reports and augment expert clinician input. This model can support clinical applications and methodological standards and used to better inform and enhance decision-making for improved risk management and patient safety.

TRIAL REGISTRATION

The study was approved by the ethics committee of the French Society of Anesthesiology (IRB 00010254-2020-20) and the CNIL (CNIL: 118 58 95) and the study was registered with ClinicalTrials.gov (NCT: NCT05185479).

ChatGPT for Automated Qualitative Research: Content Analysis

BACKGROUND

Data analysis approaches such as qualitative content analysis are notoriously time and labor intensive because of the time to detect, assess, and code a large amount of data. Tools such as ChatGPT may have tremendous potential in automating at least some of the analysis.

OBJECTIVE

The aim of this study was to explore the utility of ChatGPT in conducting qualitative content analysis through the analysis of forum posts from people sharing their experiences on reducing their sugar consumption.

METHODS

Inductive and deductive content analysis were performed on 537 forum posts to detect mechanisms of behavior change. Thorough prompt engineering provided appropriate instructions for ChatGPT to execute data analysis tasks. Data identification involved extracting change mechanisms from a subset of forum posts. The precision of the extracted data was assessed through comparison with human coding. On the basis of the identified change mechanisms, coding schemes were developed with ChatGPT using data-driven (inductive) and theory-driven (deductive) content analysis approaches. The deductive approach was informed by the Theoretical Domains Framework using both an unconstrained coding scheme and a structured coding matrix. In total, 10 coding schemes were created from a subset of data and then applied to the full data set in 10 new conversations, resulting in 100 conversations each for inductive and unconstrained deductive analysis. A total of 10 further conversations coded the full data set into the structured coding matrix. Intercoder agreement was evaluated across and within coding schemes. ChatGPT output was also evaluated by the researchers to assess whether it reflected prompt instructions.

RESULTS

The precision of detecting change mechanisms in the data subset ranged from 66% to 88%. Overall κ scores for intercoder agreement ranged from 0.72 to 0.82 across inductive coding schemes and from 0.58 to 0.73 across unconstrained coding schemes and structured coding matrix. Coding into the best-performing coding scheme resulted in category-specific κ scores ranging from 0.67 to 0.95 for the inductive approach and from 0.13 to 0.87 for the deductive approaches. ChatGPT largely followed prompt instructions in producing a description of each coding scheme, although the wording for the inductively developed coding schemes was lengthier than specified.

CONCLUSIONS

ChatGPT appears fairly reliable in assisting with qualitative analysis. ChatGPT performed better in developing an inductive coding scheme that emerged from the data than adapting an existing framework into an unconstrained coding scheme or coding directly into a structured matrix. The potential for ChatGPT to act as a second coder also appears promising, with almost perfect agreement in at least 1 coding scheme. The findings suggest that ChatGPT could prove useful as a tool to assist in each phase of qualitative content analysis, but multiple iterations are required to determine the reliability of each stage of analysis.

Construction of a Multi-Label Classifier for Extracting Multiple Incident Factors From Medication Incident Reports in Residential Care Facilities: Natural Language Processing Approach

BACKGROUND

Medication safety in residential care facilities is a critical concern, particularly when nonmedical staff provide medication assistance. The complex nature of medication-related incidents in these settings, coupled with the psychological impact on health care providers, underscores the need for effective incident analysis and preventive strategies. A thorough understanding of the root causes, typically through incident-report analysis, is essential for mitigating medication-related incidents.

OBJECTIVE

We aimed to develop and evaluate a multilabel classifier using natural language processing to identify factors contributing to medication-related incidents using incident report descriptions from residential care facilities, with a focus on incidents involving nonmedical staff.

METHODS

We analyzed 2143 incident reports, comprising 7121 sentences, from residential care facilities in Japan between April 1, 2015, and March 31, 2016. The incident factors were annotated using sentences based on an established organizational factor model and previous research findings. The following 9 factors were defined: procedure adherence, medicine, resident, resident family, nonmedical staff, medical staff, team, environment, and organizational management. To assess the label criteria, 2 researchers with relevant medical knowledge annotated a subset of 50 reports; the interannotator agreement was measured using Cohen κ. The entire data set was subsequently annotated by 1 researcher. Multiple labels were assigned to each sentence. A multilabel classifier was developed using deep learning models, including 2 Bidirectional Encoder Representations From Transformers (BERT)-type models (Tohoku-BERT and a University of Tokyo Hospital BERT pretrained with Japanese clinical text: UTH-BERT) and an Efficiently Learning Encoder That Classifies Token Replacements Accurately (ELECTRA), pretrained on Japanese text. Both sentence- and report-level training were performed; the performance was evaluated by the F1-score and exact match accuracy through 5-fold cross-validation.

RESULTS

Among all 7121 sentences, 1167, 694, 2455, 23, 1905, 46, 195, 1104, and 195 included "procedure adherence," "medicine," "resident," "resident family," "nonmedical staff," "medical staff," "team," "environment," and "organizational management," respectively. Owing to limited labels, "resident family" and "medical staff" were omitted from the model development process. The interannotator agreement values were higher than 0.6 for each label. A total of 10, 278, and 1855 reports contained no, 1, and multiple labels, respectively. The models trained using the report data outperformed those trained using sentences, with macro F1-scores of 0.744, 0.675, and 0.735 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. The report-trained models also demonstrated better exact match accuracy, with 0.411, 0.389, and 0.399 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. Notably, the accuracy was consistent even when the analysis was confined to reports containing multiple labels.

CONCLUSIONS

The multilabel classifier developed in our study demonstrated potential for identifying various factors associated with medication-related incidents using incident reports from residential care facilities. Thus, this classifier can facilitate prompt analysis of incident factors, thereby contributing to risk management and the development of preventive strategies.

In-Hospital Violence and Its Impact on Critical Care Practitioners

OBJECTIVES

To provide a narrative review of hospital violence (HV) and its impact on critical care clinicians.

DATA SOURCES

Detailed search strategy using PubMed and OVID Medline for English language articles describing HV, risk factors, precipitating events, consequences, and mitigation strategies.

STUDY SELECTION

Studies that specifically addressed HV involving critical care medicine clinicians or their practice settings were selected. The time frame was limited to the last 15 years to enhance relevance to current practice.

DATA EXTRACTION

Relevant descriptions or studies were reviewed, and abstracted data were parsed by setting, clinician type, location, social media events, impact, outcomes, and responses (agency, facility, health system, individual).

DATA SYNTHESIS

HV is globally prevalent, especially in complex care environments, and correlates with a variety of factors including ICU stay duration, conflict, and has recently expanded to out-of-hospital occurrences; online violence as well as stalking is increasingly prevalent. An overlap with violent extremism and terrorism that impacts healthcare facilities and clinicians is similarly relevant. A number of approaches can reduce HV occurrence including, most notably, conflict management training, communication initiatives, and visitor flow and access management practices. Rescue training for HV occurrences seems prudent.

CONCLUSIONS

HV is a global problem that impacts clinicians and imperils patient care. Specific initiatives to reduce HV drivers include individual training and system-wide adaptations. Future methods to identify potential perpetrators may leverage machine learning/augmented intelligence approaches.

Validation and clinical discovery demonstration of breast cancer data from a real-world data extraction platform

Objective

To validate and demonstrate the clinical discovery utility of a novel patient-mediated, medical record collection and data extraction platform developed to improve access and utilization of real-world clinical data.

Materials and Methods

Clinical variables were extracted from the medical records of 1011 consented patients with breast cancer. To validate the extracted data, case report forms completed using the structured data output of the platform were compared to manual chart review for 50 randomly-selected patients with metastatic breast cancer. To demonstrate the platform's clinical discovery utility, we identified 194 patients with early-stage clinical data who went on to develop distant metastases and utilized the platform-extracted data to assess associations between time to distant metastasis (TDM) and early-stage tumor histology, molecular type, and germline BRCA status.

Results

The platform-extracted data for the validation cohort had 97.6% precision (91.98%-100% by variable type) and 81.48% recall (58.15%-95.00% by variable type) compared to manual chart review. In our discovery cohort, the shortest TDM was significantly associated with metaplastic (739.0 days) and inflammatory histologies (1005.8 days), HR-/HER2- molecular types (1187.4 days), and positive BRCA status (1042.5 days) as compared to other histologies, molecular types, and negative BRCA status, respectively. Multivariable analyses did not produce statistically significant results.

Discussion

The precision and recall of platform-extracted clinical data are reported, although specificity could not be assessed. The data can generate clinically-relevant insights.

Conclusion

The structured real-world data produced by a novel patient-mediated, medical record-extraction platform are reliable and can power clinical discovery.

Publication Trends of Research on Adverse Event and Patient Safety in Nursing Research: A 8-Year Bibliometric Analysis

BACKGROUND

Adverse events (AEs), which are associated with medical system instability, poor clinical outcomes, and increasing socioeconomic burden, represent a negative outcome of the healthcare system and profoundly influence patient safety. However, research into AEs remains at a developmental stage according to the existing literature, and no previous studies have systematically reviewed the current state of research in the field of AEs. Therefore, the aims of this study were to interpret the results of published research in the field of AEs through bibliometric analysis and to analyze the trends and patterns in the data, which will be important for subsequent innovations in the field.

METHODS

A statistical and retrospective visualization bibliometric analysis was performed on July 28, 2022. The research data were extracted from the Web of Science Core Collection, and bibliometric citation analysis was performed using Microsoft Excel, VOSviewer 1.6.18, CiteSpace 6.1.R2, and the Online Analysis Platform of Literature Metrology ( http://bibliometric.com/ ).

RESULTS

A total of 1035 publications on AEs were included in the analysis. The number of articles increased annually from 2014 to 2022. Among them, the United States (n = 318) made the largest contribution, and Chung-Ang University (n = 20) was the affiliation with the greatest influence in this field. Despite notable international cooperation, a regional concentration of research literature production was observed in economically more developed countries. In terms of authors, Stone ND (n = 9) was the most productive author in the research of AEs. Most of the publications concerning AEs were cited from internationally influential nursing journals, and the Journal of Nursing Management (n = 62) was the most highly published journal. Regarding referencing, the article titled "Medical error-the third leading cause of death in the US" received the greatest attention on this topic (51 citations).

CONCLUSIONS

After systematically reviewed the current state of research in the field of AEs through bibliometric analysis, and AEs highlighted medication errors, patient safety, according reporting, and quality improvement as essential developments and research hotspots in this field. Furthermore, thematic analysis identified 2 new directions in research, concerned with psychological safety, nurse burnout, and with important research value and broad application prospects in the future.

Development of a scoring system to quantify errors from semantic characteristics in incident reports

OBJECTIVES

Incident reporting systems are widely used to identify risks and enable organisational learning. Free-text descriptions contain important information about factors associated with incidents. This study aimed to develop error scores by extracting information about the presence of error factors in incidents using an original decision-making model that partly relies on natural language processing techniques.

METHODS

We retrospectively analysed free-text data from reports of incidents between January 2012 and December 2022 from Nagoya University Hospital, Japan. The sample data were randomly allocated to equal-sized training and validation datasets. We conducted morphological analysis on free text to segment terms from sentences in the training dataset. We calculated error scores for terms, individual reports and reports from staff groups according to report volume size and compared these with conventional classifications by patient safety experts. We also calculated accuracy, recall, precision and F-score values from the proposed 'report error score'.

RESULTS

Overall, 114 013 reports were included. We calculated 36 131 'term error scores' from the 57 006 reports in the training dataset. There was a significant difference in error scores between reports of incidents categorised by experts as arising from errors (p<0.001, d=0.73 (large)) and other incidents. The accuracy, recall, precision and F-score values were 0.8, 0.82, 0.85 and 0.84, respectively. Group error scores were positively associated with expert ratings (correlation coefficient, 0.66; 95% CI 0.54 to 0.75, p<0.001) for all departments.

CONCLUSION

Our error scoring system could provide insights to improve patient safety using aggregated incident report data.

Adverse Event Signal Detection Using Patients' Concerns in Pharmaceutical Care Records: Evaluation of Deep Learning Models

BACKGROUND

Early detection of adverse events and their management are crucial to improving anticancer treatment outcomes, and listening to patients' subjective opinions (patients' voices) can make a major contribution to improving safety management. Recent progress in deep learning technologies has enabled various new approaches for the evaluation of safety-related events based on patient-generated text data, but few studies have focused on the improvement of real-time safety monitoring for individual patients. In addition, no study has yet been performed to validate deep learning models for screening patients' narratives for clinically important adverse event signals that require medical intervention. In our previous work, novel deep learning models have been developed to detect adverse event signals for hand-foot syndrome or adverse events limiting patients' daily lives from the authored narratives of patients with cancer, aiming ultimately to use them as safety monitoring support tools for individual patients.

OBJECTIVE

This study was designed to evaluate whether our deep learning models can screen clinically important adverse event signals that require intervention by health care professionals. The applicability of our deep learning models to data on patients' concerns at pharmacies was also assessed.

METHODS

Pharmaceutical care records at community pharmacies were used for the evaluation of our deep learning models. The records followed the SOAP format, consisting of subjective (S), objective (O), assessment (A), and plan (P) columns. Because of the unique combination of patients' concerns in the S column and the professional records of the pharmacists, this was considered a suitable data for the present purpose. Our deep learning models were applied to the S records of patients with cancer, and the extracted adverse event signals were assessed in relation to medical actions and prescribed drugs.

RESULTS

From 30,784 S records of 2479 patients with at least 1 prescription of anticancer drugs, our deep learning models extracted true adverse event signals with more than 80% accuracy for both hand-foot syndrome (n=152, 91%) and adverse events limiting patients' daily lives (n=157, 80.1%). The deep learning models were also able to screen adverse event signals that require medical intervention by health care providers. The extracted adverse event signals could reflect the side effects of anticancer drugs used by the patients based on analysis of prescribed anticancer drugs. "Pain or numbness" (n=57, 36.3%), "fever" (n=46, 29.3%), and "nausea" (n=40, 25.5%) were common symptoms out of the true adverse event signals identified by the model for adverse events limiting patients' daily lives.

CONCLUSIONS

Our deep learning models were able to screen clinically important adverse event signals that require intervention for symptoms. It was also confirmed that these deep learning models could be applied to patients' subjective information recorded in pharmaceutical care records accumulated during pharmacists' daily work.

Exploring approaches to contemporary clinical incident analysis methods within acute care settings: a scoping review protocol

OBJECTIVE

This review will explore the literature on contemporary incident analysis methods used in acute hospital settings, identifying types and characteristics of these methods and how they are used to minimize, prevent, or learn from errors and improve patient safety.

INTRODUCTION

Safety is a major focus in health care; however, despite best efforts, errors and incidents still occur, leading to harm or potential harm to patients, families, carers, staff, or the organization. Incident analysis methods aim to reduce risk of harm. Traditional methods have been criticized for failing to consider the complexity of health care and the dynamic nature of acute care settings. Alternative methodologies are being sought to achieve higher levels of patient safety and care quality care in hospitals. Learning from errors and communicating with those involved in incidents are key requirements in contemporary incident analysis.

INCLUSION CRITERIA

This review will consider empirical research published since 2013, reporting on the use of clinical incident analysis methods within acute care settings. The review will explore ways in which consumers or stakeholders (eg, clinicians or other hospital workers, patients, families, carers, visitors) have been included in these analysis methods and how data have been used to support changes in the service or organization.

METHODS

Following JBI methods and PRISMA-ScR reporting guidance, we will search PubMed, CINAHL (EBSCOhost), Embase, Scopus, the Cochrane Library, Web of Science, and ProQuest Dissertations and Theses. Studies will be reviewed independently, with results presented in tables, figures, and narrative summaries according to the concepts of interest.

Identifying and mapping measures of medication safety during transfer of care in a digital era: a scoping literature review

BACKGROUND

Measures to evaluate high-risk medication safety during transfers of care should span different safety dimensions across all components of these transfers and reflect outcomes and opportunities for proactive safety management.

OBJECTIVES

To scope measures currently used to evaluate safety interventions targeting insulin, anticoagulants and other high-risk medications during transfers of care and evaluate their comprehensiveness as a portfolio.

METHODS

Embase, Medline, Cochrane and CINAHL databases were searched using scoping methodology for studies evaluating the safety of insulin, anticoagulants and other high-risk medications during transfer of care. Measures identified were extracted into a spreadsheet, collated and mapped against three frameworks: (1) 'Key Components of an Ideal Transfer of Care', (2) work systems, processes and outcomes and (3) whether measures captured past harms, events in real time or areas of concern. The potential for digital health systems to support proactive measures was explored.

RESULTS

Thirty-five studies were reviewed with 162 measures in use. Once collated, 29 discrete categories of measures were identified. Most were outcome measures such as adverse events. Process measures included communication and issue identification and resolution. Clinic enrolment was the only work system measure. Twenty-four measures captured past harm (eg, adverse events) and six indicated future risk (eg, patient feedback for organisations). Two real-time measures alerted healthcare professionals to risks using digital systems. No measures were of advance care planning or enlisting support.

CONCLUSION

The measures identified are insufficient for a comprehensive portfolio to assess safety of key medications during transfer of care. Further measures are required to reflect all components of transfers of care and capture the work system factors contributing to outcomes in order to support proactive intervention to reduce unwanted variation and prevent adverse outcomes. Advances in digital technology and its employment within integrated care provide opportunities for the development of such measures.

Adverse event signal extraction from cancer patients' narratives focusing on impact on their daily-life activities

Adverse event (AE) management is important to improve anti-cancer treatment outcomes, but it is known that some AE signals can be missed during clinical visits. In particular, AEs that affect patients' activities of daily living (ADL) need careful monitoring as they may require immediate medical intervention. This study aimed to build deep-learning (DL) models for extracting signals of AEs limiting ADL from patients' narratives. The data source was blog posts written in Japanese by breast cancer patients. After pre-processing and annotation for AE signals, three DL models (BERT, ELECTRA, and T5) were trained and tested in three different approaches for AE signal identification. The performances of the trained models were evaluated in terms of precision, recall, and F1 scores. From 2,272 blog posts, 191 and 702 articles were identified as describing AEs limiting ADL or not limiting ADL, respectively. Among tested DL modes and approaches, T5 showed the best F1 scores to identify articles with AE limiting ADL or all AE: 0.557 and 0.811, respectively. The most frequent AE signals were "pain or numbness", "fatigue" and "nausea". Our results suggest that this AE monitoring scheme focusing on patients' ADL has potential to reinforce current AE management provided by medical staff.

COVID-19 Vaccine Adverse Event Detection Based on Multi-Label Classification With Various Label Selection Strategies

Analyzing massive VAERS reports without medical context may lead to incorrect conclusions about vaccine adverse events (VAE). Facilitating VAE detection promotes continual safety improvement for new vaccines. This study proposes a multi-label classification method with various term-and topic-based label selection strategies to improve the accuracy and efficiency of VAE detection. Topic modeling methods are first used to generate rule-based label dependencies from Medical Dictionary for Regulatory Activities terms in VAE reports with two hyper-parameters. Multiple label selection strategies, namely one-vs-rest (OvsR), problem transformation (PT), algorithm adaption (AA), and deep learning (DL) methods, are used in multi-label classification to examine the model performance, respectively. Experimental results indicated that the topic-based PT methods improve the accuracy by up to 33.69% using a COVID-19 VAE reporting data set, which improves the robustness and interpretability of our models. In addition, the topic-based OvsR methods achieve an optimal accuracy of up to 98.88%. The accuracy of the AA methods with topic-based labels increased by up to 87.36%. By contrast, the state-of-art LSTM- and BERT-based DL methods have relatively poor performance with accuracy rates of 71.89% and 64.63%, respectively. Our findings reveal that the proposed method effectively improves the model accuracy and strengthens VAE interpretability by using different label selection strategies and domain knowledge in multi-label classification for VAE detection.

Towards Medical Billing Automation: NLP for Outpatient Clinician Note Classification

Objectives

Our primary objective was to develop a natural language processing approach that accurately predicts outpatient Evaluation and Management (E/M) level of service (LoS) codes using clinicians' notes from a health system electronic health record. A secondary objective was to investigate the impact of clinic note de-identification on document classification performance.

Methods

We used retrospective outpatient office clinic notes from four medical and surgical specialties. Classification models were fine-tuned on the clinic notes datasets and stratified by subspecialty. The success criteria for the classification tasks were the classification accuracy and F1-scores on internal test data. For the secondary objective, the dataset was de-identified using Named Entity Recognition (NER) to remove protected health information (PHI), and models were retrained.

Results

The models demonstrated similar predictive performance across different specialties, except for internal medicine, which had the lowest classification accuracy across all model architectures. The models trained on the entire note corpus achieved an E/M LoS CPT code classification accuracy of 74.8% (CI 95: 74.1-75.6). However, the de-identified note corpus showed a markedly lower classification accuracy of 48.2% (CI 95: 47.7-48.6) compared to the model trained on the identified notes.

Conclusion

The study demonstrates the potential of NLP-based document classifiers to accurately predict E/M LoS CPT codes using clinical notes from various medical and procedural specialties. The models' performance suggests that the classification task's complexity merits further investigation. The de-identification experiment demonstrated that de-identification may negatively impact classifier performance. Further research is needed to validate the performance of our NLP classifiers in different healthcare settings and patient populations and to investigate the potential implications of de-identification on model performance.

A proactive learning approach toward building adaptive capacity during COVID-19: A radiology case study

The COVID-19 pandemic has challenged organizations to adapt under uncertainty and time pressure, with no pre-existing protocols or guidelines available. For organizations to learn to adapt effectively, there is a need to understand the perspectives of the frontline workforce involved in everyday operations. This study implemented a survey-tool to elicit narratives of successful adaptation based on the lived experiences frontline radiology staff at a large multispecialty pediatric hospital. Fifty-eight members of the radiology frontline staff responded to the tool between July and October of 2020. Qualitative analysis of the free-text data revealed five categories of themes that underpinned adaptive capacity of the radiology department during the pandemic: information flow, attitudes and initiative, new and adjusted workflows, availability and utilization of resources, and collaboration and teamwork. Enablers of adaptive capacity included timely and clear communication about procedures and policies from the leadership to frontline staff, and revised workflows with flexible work arrangements, such as remote patient screening. Responses to multiple choice questions in the tool helped identify the main categories of challenges faced by staff, factors that enabled successful adaptation, and resources used. The study demonstrates the use of a survey-tool to proactively identify frontline adaptations. The paper also reports a system-wide intervention resulting directly from a discovery enabled by the findings based on the use of RETIPS in the radiology department. In general, the tool could be used in concert with existing learning mechanisms, such as safety event reporting systems, to inform leadership-level decisions to support adaptive capacity.

The effect of refresher training on fact description in medical incident report writing in the Japanese language

To maintain the effectiveness of the training (1st-Training Session: 1st-TS) to accurate describe facts in the medical incident reports (IRs) in Japanese, a refresher TS was designed and its effectiveness was examined. First, textual analysis showed that IRs' accuracy significantly decreased six months after the 1st-TS. Based on this result, the refresher TS was designed and conducted with 64 residents. To verify the refresher TS' effectiveness, IRs after the 1st-TS, six months later, and after the refresher TS were compared via text analysis. The results showed that the refresher TS restored the description rate of patient's background, safety check procedures, original work procedures, information on equipment used, reporter's actions, and post-incident response. The questionnaire was also administered and showed that the refresher TS contributed to residents' motivation to learn about IRs. In conclusion, the refresher TS contributed to sustaining the effect of the 1st-TS on accurately describing IRs.

Interorganizational health information exchange-related patient safety incidents: A descriptive register-based qualitative study

PURPOSE

The current literature related to patient safety of interorganizational health information is fragmented. This study aims to identify interorganizational health information exchange-related patient safety incidents occurring in the emergency department, emergency medical services, and home care. The research also aimed to describe the causes and consequences of these incidents.

METHODS

A total of sixty (n = 60) interorganizational health information exchange-related patient safety incident free text reports were analyzed. The reports were reported in the emergency department, emergency medical services, or home care between January 2016 and December 2019 in one hospital district in Finland.

RESULTS

The identified interorganizational health information exchange-related incidents were grouped under two main categories: "Inadequate documentation"; and "Inadequate use of information". The causes of these incidents were grouped under the two main categories "Factors related to the healthcare professional " and "Organizational factors", while the consequences of these incidents fell under the two main categories "Adverse events" and "Additional actions to prevent, avoid, and correct adverse events".

CONCLUSION

This study shows that the inadequate documentation and use of information is mainly caused by factors related to the healthcare professional and organization, including technical problems. These incidents cause adverse events and additional actions to prevent, avoid, and correct the events. The sociotechnical perspective, including factors related to health care professionals, organization, and technology, should be emphasized in patient safety development of inter-organizational health information exchange and it will be the focus of our future research. Continuous research and development work is needed because the processes and information systems used in health care are constantly evolving.

Thematic mapping of perioperative incident reporting data to relational coordination domains

Communication failure is a common root cause of adverse clinical events. Problematic communication domains are difficult to decipher, and communication improvement strategies are scarce. This study compared perioperative incident reports (IR) identifying potential communication failures with the results of a contemporaneous peri-operative Relational Coordination (RC) survey. We hypothesised that IR-prevalent themes would map to areas-of-weakness identified in the RC survey. Perioperative IRs filed between 2018 and 2020 (n = 6,236) were manually reviewed to identify communication failures (n = 1049). The IRs were disaggregated into seven RC theory domains and compared with the RC survey. Report disaggregation ratings demonstrated a three-way inter-rater agreement of 91.2%. Of the 1,049 communication failure-related IRs, shared knowledge deficits (n = 479, 46%) or accurate communication (n = 465, 44%) were most frequently identified. Communication frequency failures (n = 3, 0.3%) were rarely coded. Comparatively, shared knowledge was the weakest domain in the RC survey, while communication frequency was the strongest, correlating well with our IR data. Linking IR with RC domains offers a novel approach to assessing the specific elements of communication failures with an acute care facility. This approach provides a deployable mechanism to trend intra- and inter-domain progress in communication success, and develop targeted interventions to mitigate against communication failure-related adverse events.

Adverse drug event detection using natural language processing: A scoping review of supervised learning methods

To reduce adverse drug events (ADEs), hospitals need a system to support them in monitoring ADE occurrence routinely, rapidly, and at scale. Natural language processing (NLP), a computerized approach to analyze text data, has shown promising results for the purpose of ADE detection in the context of pharmacovigilance. However, a detailed qualitative assessment and critical appraisal of NLP methods for ADE detection in the context of ADE monitoring in hospitals is lacking. Therefore, we have conducted a scoping review to close this knowledge gap, and to provide directions for future research and practice. We included articles where NLP was applied to detect ADEs in clinical narratives within electronic health records of inpatients. Quantitative and qualitative data items relating to NLP methods were extracted and critically appraised. Out of 1,065 articles screened for eligibility, 29 articles met the inclusion criteria. Most frequent tasks included named entity recognition (n = 17; 58.6%) and relation extraction/classification (n = 15; 51.7%). Clinical involvement was reported in nine studies (31%). Multiple NLP modelling approaches seem suitable, with Long Short Term Memory and Conditional Random Field methods most commonly used. Although reported overall performance of the systems was high, it provides an inflated impression given a steep drop in performance when predicting the ADE entity or ADE relation class. When annotating corpora, treating an ADE as a relation between a drug and non-drug entity seems the best practice. Future research should focus on semi-automated methods to reduce the manual annotation effort, and examine implementation of the NLP methods in practice.

Natural language processing and network analysis in patients withdrawing from life-sustaining treatments: a retrospective cohort study

BACKGROUND

Providing palliative care to patients who withdraw from life-sustaining treatments is crucial; however, delays or the absence of such services are prevalent. This study used natural language processing and network analysis to identify the role of medications as early palliative care referral triggers.

METHODS

We conducted a retrospective observational study of 119 adult patients receiving specialized palliative care after endotracheal tube withdrawal in intensive care units of a Taiwan-based medical center between July 2016 and June 2018. Patients were categorized into early integration and late referral groups based on the median survival time. Using natural language processing, we analyzed free texts from electronic health records. The Palliative trigger index was also calculated for comparison, and network analysis was performed to determine the co-occurrence of terms between the two groups.

RESULTS

Broad-spectrum antibiotics, antifungal agents, diuretics, and opioids had high Palliative trigger index. The most common co-occurrences in the early integration group were micafungin and voriconazole (co-correlation = 0.75). However, in the late referral group, piperacillin and penicillin were the most common co-occurrences (co-correlation = 0.843).

CONCLUSION

Treatments for severe infections, chronic illnesses, and analgesics are possible triggers for specialized palliative care consultations. The Palliative trigger index and network analysis indicated the need for palliative care in patients withdrawing from life-sustaining treatments. This study recommends establishing a therapeutic control system based on computerized order entry and integrating it into a shared-decision model.

Development of a Novel Scoring System to Quantify the Severity of Incident Reports: An Exploratory Research Study

Incident reporting systems have been widely adopted to collect information about patient safety incidents. Much of the value of incident reports lies in the free-text section. Computer processing of semantic information may be helpful to analyze this. We developed a novel scoring system for decision making to assess the severity of incidents using the semantic characteristics of the text in incident reports, and compared its results with experts' opinions. We retrospectively analyzed free-text data from incident reports from January 2012 to September 2021 at Nagoya University Hospital, Aichi, Japan. The sample was allocated to training and validation datasets using the hold-out method. Morphological analysis was used to segment terms in the training dataset. We calculated a severity term score, a severity report score and severity group score, by report volume size, and compared these with conventional severity classifications by patient safety experts and reporters. We allocated 96,082 incident reports into two groups. We calculated 1,802 severity term scores from the 48,041 reports in the training dataset. There was a significant difference in severity report score between reports categorized as severe and not severe by experts (95% confidence interval [CI] -0.83 to -0.80, p < 0.001, d = 0.81). Severity group scores were positively associated with severity ratings from experts and reporters (correlation coefficients 0.73 [95% CI 0.63-0.80, p < 0.001] and 0.79 [95% CI 0.71-0.85, p < 0.001]) for all departments. Our severity scoring system could therefore contribute to better organizational patient safety.

Performance of machine learning algorithms for surgical site infection case detection and prediction: A systematic review and meta-analysis

Background

Medical researchers and clinicians have shown much interest in developing machine learning (ML) algorithms to detect/predict surgical site infections (SSIs). However, little is known about the overall performance of ML algorithms in predicting SSIs and how to improve the algorithm's robustness. We conducted a systematic review and meta-analysis to summarize the performance of ML algorithms in SSIs case detection and prediction and to describe the impact of using unstructured and textual data in the development of ML algorithms.

Methods

MEDLINE, EMBASE, CINAHL, CENTRAL and Web of Science were searched from inception to March 25, 2021. Study characteristics and algorithm development information were extracted. Performance statistics (e.g., sensitivity, area under the receiver operating characteristic curve [AUC]) were pooled using a random effect model. Stratified analysis was applied to different study characteristic levels. Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Diagnostic Test Accuracy Studies (PRISMA-DTA) was followed.

Results

Of 945 articles identified, 108 algorithms from 32 articles were included in this review. The overall pooled estimate of the SSI incidence rate was 3.67%, 95% CI: 3.58-3.76. Mixed-use of structured and textual data-based algorithms (pooled estimates of sensitivity 0.83, 95% CI: 0.78-0.87, specificity 0.92, 95% CI: 0.86-0.95, AUC 0.92, 95% CI: 0.89-0.94) outperformed algorithms solely based on structured data (sensitivity 0.56, 95% CI:0.43-0.69, specificity 0.95, 95% CI:0.91-0.97, AUC = 0.90, 95% CI: 0.87-0.92).

Conclusions

ML algorithms developed with structured and textual data provided optimal performance. External validation of ML algorithms is needed to translate current knowledge into clinical practice.

Developing EMR-based algorithms to Identify hospital adverse events for health system performance evaluation and improvement: Study protocol

Background

Measurement of care quality and safety mainly relies on abstracted administrative data. However, it is well studied that administrative data-based adverse event (AE) detection methods are suboptimal due to lack of clinical information. Electronic medical records (EMR) have been widely implemented and contain detailed and comprehensive information regarding all aspects of patient care, offering a valuable complement to administrative data. Harnessing the rich clinical data in EMRs offers a unique opportunity to improve detection, identify possible risk factors of AE and enhance surveillance. However, the methodological tools for detection of AEs within EMR need to be developed and validated. The objectives of this study are to develop EMR-based AE algorithms from hospital EMR data and assess AE algorithm’s validity in Canadian EMR data.

Methods

Patient EMR structured and text data from acute care hospitals in Calgary, Alberta, Canada will be linked with discharge abstract data (DAD) between 2010 and 2020 (n~1.5 million). AE algorithms development. First, a comprehensive list of AEs will be generated through a systematic literature review and expert recommendations. Second, these AEs will be mapped to EMR free texts using Natural Language Processing (NLP) technologies. Finally, an expert panel will assess the clinical relevance of the developed NLP algorithms. AE algorithms validation: We will test the newly developed AE algorithms on 10,000 randomly selected EMRs between 2010 to 2020 from Calgary, Alberta. Trained reviewers will review the selected 10,000 EMR charts to identify AEs that had occurred during hospitalization. Performance indicators (e.g., sensitivity, specificity, positive predictive value, negative predictive value, F₁ score, etc.) of the developed AE algorithms will be assessed using chart review data as the reference standard.

Discussion

The results of this project can be widely implemented in EMR based healthcare system to accurately and timely detect in-hospital AEs.

Artificial intelligence and its impact on the domains of universal health coverage, health emergencies and health promotion: An overview of systematic reviews

BACKGROUND

Artificial intelligence is fueling a new revolution in medicine and in the healthcare sector. Despite the growing evidence on the benefits of artificial intelligence there are several aspects that limit the measure of its impact in people's health. It is necessary to assess the current status on the application of AI towards the improvement of people's health in the domains defined by WHO's Thirteenth General Programme of Work (GPW13) and the European Programme of Work (EPW), to inform about trends, gaps, opportunities, and challenges.

OBJECTIVE

To perform a systematic overview of systematic reviews on the application of artificial intelligence in the people's health domains as defined in the GPW13 and provide a comprehensive and updated map on the application specialties of artificial intelligence in terms of methodologies, algorithms, data sources, outcomes, predictors, performance, and methodological quality.

METHODS

A systematic search in MEDLINE, EMBASE, Cochrane and IEEEXplore was conducted between January 2015 and June 2021 to collect systematic reviews using a combination of keywords related to the domains of universal health coverage, health emergencies protection, and better health and wellbeing as defined by the WHO's PGW13 and EPW. Eligibility criteria was based on methodological quality and the inclusion of practical implementation of artificial intelligence. Records were classified and labeled using ICD-11 categories into the domains of the GPW13. Descriptors related to the area of implementation, type of modeling, data entities, outcomes and implementation on care delivery were extracted using a structured form and methodological aspects of the included reviews studies was assessed using the AMSTAR checklist.

RESULTS

The search strategy resulted in the screening of 815 systematic reviews from which 203 were assessed for eligibility and 129 were included in the review. The most predominant domain for artificial intelligence applications was Universal Health Coverage (N = 98) followed by Health Emergencies (N = 16) and Better Health and Wellbeing (N = 15). Neoplasms area on Universal Health Coverage was the disease area featuring most of the applications (21.7 %, N = 28). The reviews featured analytics primarily over both public and private data sources (67.44 %, N = 87). The most used type of data was medical imaging (31.8 %, N = 41) and predictors based on regions of interest and clinical data. The most prominent subdomain of Artificial Intelligence was Machine Learning (43.4 %, N = 56), in which Support Vector Machine method was predominant (20.9 %, N = 27). Regarding the purpose, the application of Artificial Intelligence I is focused on the prediction of the diseases (36.4 %, N = 47). With respect to the validation, more than a half of the reviews (54.3 %, N = 70) did not report a validation procedure and, whenever available, the main performance indicator was the accuracy (28.7 %, N = 37). According to the methodological quality assessment, a third of the reviews (34.9 %, N = 45) implemented methods for analysis the risk of bias and the overall AMSTAR score below was 5 (4.01 ± 1.93) on all the included systematic reviews.

CONCLUSION

Artificial intelligence is being used for disease modelling, diagnose, classification and prediction in the three domains of GPW13. However, the evidence is often limited to laboratory and the level of adoption is largely unbalanced between ICD-11 categoriesand diseases. Data availability is a determinant factor on the developmental stage of artificial intelligence applications. Most of the reviewed studies show a poor methodological quality and are at high risk of bias, which limits the reproducibility of the results and the reliability of translating these applications to real clinical scenarios. The analyzed papers show results only in laboratory and testing scenarios and not in clinical trials nor case studies, limiting the supporting evidence to transfer artificial intelligence to actual care delivery.

Identifying low-quality patterns in accident reports from textual data

Accident investigation reports provide useful knowledge to support companies to propose preventive and mitigative measures. However, the information presented in accident report databases is normally large, complex, filled with errors and has missing and/or redundant data. In this article, we propose text mining and natural language processing techniques to investigate low-quality accident reports. We adopted machine learning (ML) to detect and investigate inconsistencies on accident reports. The methodology was applied to 626 documents collected from an actual hydroelectric power company. The initial ML performances indicated data divergences and concerns related to the report structure. Then, the accident database was restructured to a more proper form confirming the supposition about the quality of the reports investigated. The proposed approach can be used as a diagnostic tool to improve the design of accident investigation reports to provide a more useful source of knowledge to support decisions in the safety context.

Usability and Accessibility of Publicly Available Patient Safety Databases

OBJECTIVES

The aims of the study were to identify publicly available patient safety report databases and to determine whether these databases support safety analyst and data scientist use to identify patterns and trends.

METHODS

An Internet search was conducted to identify publicly available patient safety databases that contained patient safety reports. Each database was analyzed to identify features that enable patient safety analyst and data scientist use of these databases.

RESULTS

Seven databases (6 hosted by federal agencies, 1 hosted by a nonprofit organization) containing more than 28.3 million safety reports were identified. Some, but not all, databases contained features to support patient safety analyst use: 57.1% provided the ability to sort/compare/filter data, 42.9% provided data visualization, and 85.7% enabled free-text search. None of the databases provided regular updates or monitoring and only one database suggested solutions to patient safety reports. Analysis of features to support data scientist use showed that only 42.9% provided an application programing interface, most (85.7%) provided batch downloading, all provided documentation about the database, and 71.4% provided a data dictionary. All databases provided open access. Only 28.6% provided a data diagram.

CONCLUSIONS

Patient safety databases should be improved to support patient safety analyst use by, at a minimum, allowing for data to be sorted/compared/filtered, providing data visualization, and enabling free-text search. Databases should also enable data scientist use by, at a minimum, providing an application programing interface, batch downloading, and a data dictionary.

Training residents in medical incident report writing to improve incident investigation quality and efficiency enables accurate fact gathering

We assessed whether training on writing readable and accurate medical incident reports (IRs) improves the quality of fact description. In this training, 124 residents created fictional IRs. We provided tips, including using When, Where, Who, What, Why, How. We compared the fictional IRs with and without tips, and the trainees' and non-trainees' IRs submitted in the first five months after training. Results indicated that the subject words in IRs were more clarified and the readability was improved. The fictional IRs using tips were more accurate, with increased descriptions of the patient's background, reporter's actions, team members' actions and conversations, safety check procedures, result of the error, and post-incident response. The reporter's actions, work procedures, and environment were more clarified in the trainees' IRs than in the non-trainees' IRs. This training may help analysts comprehend the sequence of and underlying factors for reporter's actions based on IRs.

[The COVID-19 pandemic's impacts on adverse events associated with care]

Since early 2020, the onset of the COVID-19 pandemic, physicians have continued to report adverse events associated with care. Patients also continued to participate in the hospital satisfaction surveys. To date, no study in France has measured the impact of the pandemic on adverse events and patient satisfaction. We looked at the characteristics of these adverse events in relation to the pandemic and put patients' feelings into perspective. A qualitative and observational retrospective study of the REX and MCO48 databases was carried out. The quantitative study of the REX database was supplemented by a qualitative analysis of the declarations. The adverse events more often affects middle-aged men aged 60 years, while deaths occur in older patients with more complex pathologies and more urgent management. The nature of these events is different depending on the reporting period: Those reported in the first wave are more urgent, occur less frequently in the operating room than in the emergency room, and are considered less preventable than those reported in the second wave. The latter are more similar to the events that usually occur. The implementation of effective barriers, particularly within the teams, has made it possible to reduce the impact of the second wave on the occurrence of these events, the role of communication seems essential. The overall patient satisfaction score as well as those for medical and paramedical care has increased, which may reflect patient solidarity with caregivers. The attitude of active resilience on the part of all actors has been a major element in risk management during this crisis and it is essential to capitalize on these collaborative processes for the future.

Harnessing Event Report Data to Identify Diagnostic Error During the COVID-19 Pandemic

INTRODUCTION

COVID-19 exposed systemic gaps with increased potential for diagnostic error. This project implemented a new approach leveraging electronic safety reporting to identify and categorize diagnostic errors during the pandemic.

METHODS

All safety event reports from March 1, 2020, to February 28, 2021, at an academic medical center were evaluated using two complementary pathways (Pathway 1: all reports with explicit mention of COVID-19; Pathway 2: all reports without explicit mention of COVID-19 where natural language processing [NLP] plus logic-based stratification was applied to identify potential cases). Cases were evaluated by manual review to identify diagnostic error/delay and categorize error type using a recently proposed classification framework of eight categories of pandemic-related diagnostic errors.

RESULTS

A total of 14,230 reports were included, with 95 (0.7%) identified as cases of diagnostic error/delay. Pathway 1 (n = 1,780 eligible reports) yielded 45 reports with diagnostic error/delay (positive predictive value [PPV] = 2.5%), of which 35.6% (16/45) were attributed to pandemic-related strain. In Pathway 2, the NLP-based algorithm flagged 110 safety reports for manual review from 12,450 eligible reports. Of these, 50 reports had diagnostic error/delay (PPV = 45.5%); 94.0% (47/50) were related to strain. Errors from all eight categories of the taxonomy were found on analysis.

CONCLUSION

An event reporting-based strategy including use of simple-NLP-identified COVID-19-related diagnostic errors/delays uncovered several safety concerns related to COVID-19. An NLP-based approach can complement traditional reporting and be used as a just-in-time monitoring system to enable early detection of emerging risks from large volumes of safety reports.

A New Method to Extract Health-Related Quality of Life Data From Social Media Testimonies: Algorithm Development and Validation

Background

Monitoring social media has been shown to be a useful means to capture patients’ opinions and feelings about medical issues, ranging from diseases to treatments. Health-related quality of life (HRQoL) is a useful indicator of overall patients’ health, which can be captured online.

Objective

This study aimed to describe a social media listening algorithm able to detect the impact of diseases or treatments on specific dimensions of HRQoL based on posts written by patients in social media and forums.

Methods

Using a web crawler, 19 forums in France were harvested, and messages related to patients’ experience with disease or treatment were specifically collected. The SF-36 (Short Form Health Survey) and EQ-5D (Euro Quality of Life 5 Dimensions) HRQoL surveys were mixed and adapted for a tailored social media listening system. This was carried out to better capture the variety of expression on social media, resulting in 5 dimensions of the HRQoL, which are physical, psychological, activity-based, social, and financial. Models were trained using cross-validation and hyperparameter optimization. Oversampling was used to increase the infrequent dimension: after annotation, SMOTE (synthetic minority oversampling technique) was used to balance the proportions of the dimensions among messages.

Results

The training set was composed of 1399 messages, randomly taken from a batch of 20,000 health-related messages coming from forums. The algorithm was able to detect a general impact on HRQoL (sensitivity of 0.83 and specificity of 0.74), a physical impact (0.67 and 0.76), a psychic impact (0.82 and 0.60), an activity-related impact (0.73 and 0.78), a relational impact (0.73 and 0.70), and a financial impact (0.79 and 0.74).

Conclusions

The development of an innovative method to extract health data from social media as real time assessment of patients’ HRQoL is useful to a patient-centered medical care. As a source of real-world data, social media provide a complementary point of view to understand patients’ concerns and unmet needs, as well as shedding light on how diseases and treatments can be a burden in their daily lives.

Artificial Intelligence to Improve Health Outcomes in the NICU and PICU: A Systematic Review

CONTEXT

Artificial intelligence (AI) technologies are increasingly used in pediatrics and have the potential to help inpatient physicians provide high-quality care for critically ill children.

OBJECTIVE

We aimed to describe the use of AI to improve any health outcome(s) in neonatal and pediatric intensive care.

DATA SOURCE

PubMed, IEEE Xplore, Cochrane, and Web of Science databases.

STUDY SELECTION

We used peer-reviewed studies published between June 1, 2010, and May 31, 2020, in which researchers described (1) AI, (2) pediatrics, and (3) intensive care. Studies were included if researchers assessed AI use to improve at least 1 health outcome (eg, mortality).

DATA EXTRACTION

Data extraction was conducted independently by 2 researchers. Articles were categorized by direct or indirect impact of AI, defined by the European Institute of Innovation and Technology Health joint report.

RESULTS

Of the 287 publications screened, 32 met inclusion criteria. Approximately 22% (n = 7) of studies revealed a direct impact and improvement in health outcomes after AI implementation. Majority were in prototype testing, and few were deployed into an ICU setting. Among the remaining 78% (n = 25) AI models outperformed standard clinical modalities and may have indirectly influenced patient outcomes. Quantitative assessment of health outcomes using statistical measures, such as area under the receiver operating curve (56%; n = 18) and specificity (38%; n = 12), revealed marked heterogeneity in metrics and standardization.

CONCLUSIONS

Few studies have revealed that AI has directly improved health outcomes for pediatric critical care patients. Further prospective, experimental studies are needed to assess AI's impact by using established implementation frameworks, standardized metrics, and validated outcome measures.

How Clinicians Perceive Artificial Intelligence-Assisted Technologies in Diagnostic Decision Making: Mixed Methods Approach

Background

With the rapid development of artificial intelligence (AI) and related technologies, AI algorithms are being embedded into various health information technologies that assist clinicians in clinical decision making.

Objective

This study aimed to explore how clinicians perceive AI assistance in diagnostic decision making and suggest the paths forward for AI-human teaming for clinical decision making in health care.

Methods

This study used a mixed methods approach, utilizing hierarchical linear modeling and sentiment analysis through natural language understanding techniques.

Results

A total of 114 clinicians participated in online simulation surveys in 2020 and 2021. These clinicians studied family medicine and used AI algorithms to aid in patient diagnosis. Their overall sentiment toward AI-assisted diagnosis was positive and comparable with diagnoses made without the assistance of AI. However, AI-guided decision making was not congruent with the way clinicians typically made decisions in diagnosing illnesses. In a quantitative survey, clinicians reported perceiving current AI assistance as not likely to enhance diagnostic capability and negatively influenced their overall performance (β=–0.421, P=.02). Instead, clinicians’ diagnostic capabilities tended to be associated with well-known parameters, such as education, age, and daily habit of technology use on social media platforms.

Conclusions

This study elucidated clinicians’ current perceptions and sentiments toward AI-enabled diagnosis. Although the sentiment was positive, the current form of AI assistance may not be linked with efficient decision making, as AI algorithms are not well aligned with subjective human reasoning in clinical diagnosis. Developers and policy makers in health could gather behavioral data from clinicians in various disciplines to help align AI algorithms with the unique subjective patterns of reasoning that humans employ in clinical diagnosis.

The Utility of Different Data Standards to Document Adverse Drug Event Symptoms and Diagnoses: Mixed Methods Study

BACKGROUND

Existing systems to document adverse drug events often use free text data entry, which produces nonstandardized and unstructured data that are prone to misinterpretation. Standardized terminology may improve data quality; however, it is unclear which data standard is most appropriate for documenting adverse drug event symptoms and diagnoses.

OBJECTIVE

This study aims to compare the utility, strengths, and weaknesses of different data standards for documenting adverse drug event symptoms and diagnoses.

METHODS

We performed a mixed methods substudy of a multicenter retrospective chart review. We reviewed the research records of prospectively diagnosed adverse drug events at 5 Canadian hospitals. A total of 2 pharmacy research assistants independently entered the symptoms and diagnoses for the adverse drug events using four standards: Medical Dictionary for Regulatory Activities (MedDRA), Systematized Nomenclature of Medicine (SNOMED) Clinical Terms, SNOMED Adverse Reaction (SNOMED ADR), and International Classification of Diseases (ICD) 11th Revision. Disagreements between research assistants regarding the case-specific utility of data standards were discussed until a consensus was reached. We used consensus ratings to determine the proportion of adverse drug events covered by a data standard and coded and analyzed field notes from the consensus sessions.

RESULTS

We reviewed 573 adverse drug events and found that MedDRA and ICD-11 had excellent coverage of adverse drug event symptoms and diagnoses. MedDRA had the highest number of matches between the research assistants, whereas ICD-11 had the fewest. SNOMED ADR had the lowest proportion of adverse drug event coverage. The research assistants were most likely to encounter terminological challenges with SNOMED ADR and usability challenges with ICD-11, whereas least likely to encounter challenges with MedDRA.

CONCLUSIONS

Usability, comprehensiveness, and accuracy are important features of data standards for documenting adverse drug event symptoms and diagnoses. On the basis of our results, we recommend the use of MedDRA.

Realizing the Power of Text Mining and Natural Language Processing for Analyzing Patient Safety Event Narratives: The Challenges and Path Forward

ABSTRACT

Patient safety event (PSE) reports are a useful lens to understand hazards and patient safety risks in healthcare systems. However, patient safety officers and analysts in healthcare systems and safety organizations are challenged to make sense of the ever-increasing volume of PSE reports, including the free-text narratives. As a result, there is a growing emphasis on applying text mining and natural language processing (NLP) approaches to assist in the processing and understanding of these narratives. Although text mining and NLP in healthcare have advanced significantly over the past decades, the utility of the resulting models, ontologies, and algorithms to analyze PSE narratives are limited given the unique difference and challenges in content and language between PSE narratives and clinical documentation. To promote the application of text mining and NLP for PSE narratives, these unique challenges must be addressed. Improving data access, developing NLP resources to practically use contributing factor taxonomies, and developing and adopting shared specifications for interoperability will help create an infrastructure and environment that unlocks the collaborative potential between patient safety, research, and machine learning communities, in the development of reproducible and generalizable methods and models to better understand and improve patient safety and patient care.

Reducing medication errors for adults in hospital settings

BACKGROUND

Medication errors are preventable events that may cause or lead to inappropriate medication use or patient harm while the medication is in the control of the healthcare professional or patient. Medication errors in hospitalised adults may cause harm, additional costs, and even death.

OBJECTIVES

To determine the effectiveness of interventions to reduce medication errors in adults in hospital settings.

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, five other databases and two trials registers on 16 January 2020.  SELECTION CRITERIA: We included randomised controlled trials (RCTs) and interrupted time series (ITS) studies investigating interventions aimed at reducing medication errors in hospitalised adults, compared with usual care or other interventions. Outcome measures included adverse drug events (ADEs), potential ADEs, preventable ADEs, medication errors, mortality, morbidity, length of stay, quality of life and identified/solved discrepancies. We included any hospital setting, such as inpatient care units, outpatient care settings, and accident and emergency departments.

DATA COLLECTION AND ANALYSIS

We followed the standard methodological procedures expected by Cochrane and the Effective Practice and Organisation of Care (EPOC) Group. Where necessary, we extracted and reanalysed ITS study data using piecewise linear regression, corrected for autocorrelation and seasonality, where possible.  MAIN RESULTS: We included 65 studies: 51 RCTs and 14 ITS studies, involving 110,875 participants. About half of trials gave rise to 'some concerns' for risk of bias during the randomisation process and one-third lacked blinding of outcome assessment. Most ITS studies presented low risk of bias. Most studies came from high-income countries or high-resource settings. Medication reconciliation -the process of comparing a patient's medication orders to the medications that the patient has been taking- was the most common type of intervention studied. Electronic prescribing systems, barcoding for correct administering of medications, organisational changes, feedback on medication errors, education of professionals and improved medication dispensing systems were other interventions studied. Medication reconciliation Low-certainty evidence suggests that medication reconciliation (MR) versus no-MR may reduce medication errors (odds ratio [OR] 0.55, 95% confidence interval (CI) 0.17 to 1.74; 3 studies; n=379). Compared to no-MR, MR probably reduces ADEs (OR 0.38, 95%CI 0.18 to 0.80; 3 studies, n=1336 ; moderate-certainty evidence), but has little to no effect on length of stay (mean difference (MD) -0.30 days, 95%CI -1.93 to 1.33 days; 3 studies, n=527) and quality of life (MD -1.51, 95%CI -10.04 to 7.02; 1 study, n=131).  Low-certainty evidence suggests that, compared to MR by other professionals, MR by pharmacists may reduce medication errors (OR 0.21, 95%CI 0.09 to 0.48; 8 studies, n=2648) and may increase ADEs (OR 1.34, 95%CI 0.73 to 2.44; 3 studies, n=2873). Compared to MR by other professionals, MR by pharmacists may have little to no effect on length of stay (MD -0.25, 95%CI -1.05 to 0.56; 6 studies, 3983). Moderate-certainty evidence shows that this intervention probably has little to no effect on mortality during hospitalisation (risk ratio (RR) 0.99, 95%CI 0.57 to 1.7; 2 studies, n=1000), and on readmissions at one month (RR 0.93, 95%CI 0.76 to 1.14; 2 studies, n=997); and low-certainty evidence suggests that the intervention may have little to no effect on quality of life (MD 0.00, 95%CI -14.09 to 14.09; 1 study, n=724).  Low-certainty evidence suggests that database-assisted MR conducted by pharmacists, versus unassisted MR conducted by pharmacists, may reduce potential ADEs (OR 0.26, 95%CI 0.10 to 0.64; 2 studies, n=3326), and may have no effect on length of stay (MD 1.00, 95%CI -0.17 to 2.17; 1 study, n=311).  Low-certainty evidence suggests that MR performed by trained pharmacist technicians, versus pharmacists, may have little to no difference on length of stay (MD -0.30, 95%CI -2.12 to 1.52; 1 study, n=183). However, the CI is compatible with important beneficial and detrimental effects. Low-certainty evidence suggests that MR before admission may increase the identification of discrepancies compared with MR after admission (MD 1.27, 95%CI 0.46 to 2.08; 1 study, n=307). However, the CI is compatible with important beneficial and detrimental effects. Moderate-certainty evidence shows that multimodal interventions probably increase discrepancy resolutions compared to usual care (RR 2.14, 95%CI 1.81 to 2.53; 1 study, n=487). Computerised physician order entry (CPOE)/clinical decision support systems (CDSS) Moderate-certainty evidence shows that CPOE/CDSS probably reduce medication errors compared to paper-based systems (OR 0.74, 95%CI 0.31 to 1.79; 2 studies, n=88).  Moderate-certainty evidence shows that, compared with standard CPOE/CDSS, improved CPOE/CDSS probably reduce medication errors (OR 0.85, 95%CI 0.74 to 0.97; 2 studies, n=630). Low-certainty evidence suggests that prioritised alerts provided by CPOE/CDSS may prevent ADEs compared to non-prioritised (inconsequential) alerts (MD 1.98, 95%CI 1.65 to 2.31; 1 study; participant numbers unavailable). Barcode identification of participants/medications Low-certainty evidence suggests that barcoding may reduce medication errors (OR 0.69, 95%CI 0.59 to 0.79; 2 studies, n=50,545). Reduced working hours Low-certainty evidence suggests that reduced working hours may reduce serious medication errors (RR 0.83, 95%CI 0.63 to 1.09; 1 study, n=634). However, the CI is compatible with important beneficial and detrimental effects. Feedback on prescribing errors Low-certainty evidence suggests that feedback on prescribing errors may reduce medication errors (OR 0.47, 95%CI 0.33 to 0.67; 4 studies, n=384). Dispensing system Low-certainty evidence suggests that dispensing systems in surgical wards may reduce medication errors (OR 0.61, 95%CI 0.47 to 0.79; 2 studies, n=1775).

AUTHORS' CONCLUSIONS

Low- to moderate-certainty evidence suggests that, compared to usual care, medication reconciliation, CPOE/CDSS, barcoding, feedback and dispensing systems in surgical wards may reduce medication errors and ADEs. However, the results are imprecise for some outcomes related to medication reconciliation and CPOE/CDSS. The evidence for other interventions is very uncertain. Powered and methodologically sound studies are needed to address the identified evidence gaps. Innovative, synergistic strategies -including those that involve patients- should also be evaluated.

Natural language processing and machine learning to assist radiation oncology incident learning

PURPOSE

To develop a Natural Language Processing (NLP) and Machine Learning (ML) pipeline that can be integrated into an Incident Learning System (ILS) to assist radiation oncology incident learning by semi-automating incident classification. Our goal was to develop ML models that can generate label recommendations, arranged according to their likelihoods, for three data elements in Canadian NSIR-RT taxonomy.

METHODS

Over 6000 incident reports were gathered from the Canadian national ILS as well as our local ILS database. Incident descriptions from these reports were processed using various NLP techniques. The processed data with the expert-generated labels were used to train and evaluate over 500 multi-output ML algorithms. The top three models were identified and tuned for each of three different taxonomy data elements, namely: (1) process step where the incident occurred, (2) problem type of the incident and (3) the contributing factors of the incident. The best-performing model after tuning was identified for each data element and tested on unseen data.

RESULTS

The MultiOutputRegressor extended Linear SVR models performed best on the three data elements. On testing, our models ranked the most appropriate label 1.48 ± 0.03, 1.73 ± 0.05 and 2.66 ± 0.08 for process-step, problem-type and contributing factors respectively.

CONCLUSIONS

We developed NLP-ML models that can perform incident classification. These models will be integrated into our ILS to generate a drop-down menu. This semi-automated feature has the potential to improve the usability, accuracy and efficiency of our radiation oncology ILS.

Improving broad-coverage medical entity linking with semantic type prediction and large-scale datasets

OBJECTIVES

Biomedical natural language processing tools are increasingly being applied for broad-coverage information extraction-extracting medical information of all types in a scientific document or a clinical note. In such broad-coverage settings, linking mentions of medical concepts to standardized vocabularies requires choosing the best candidate concepts from large inventories covering dozens of types. This study presents a novel semantic type prediction module for biomedical NLP pipelines and two automatically-constructed, large-scale datasets with broad coverage of semantic types.

METHODS

We experiment with five off-the-shelf biomedical NLP toolkits on four benchmark datasets for medical information extraction from scientific literature and clinical notes. All toolkits adopt a staged approach of mention detection followed by two stages of medical entity linking: (1) generating a list of candidate concepts, and (2) picking the best concept among them. We introduce a semantic type prediction module to alleviate the problem of overgeneration of candidate concepts by filtering out irrelevant candidate concepts based on the predicted semantic type of a mention. We present MedType, a fully modular semantic type prediction model which we integrate into the existing NLP toolkits. To address the dearth of broad-coverage training data for medical information extraction, we further present WikiMed and PubMedDS, two large-scale datasets for medical entity linking.

RESULTS

Semantic type filtering improves medical entity linking performance across all toolkits and datasets, often by several percentage points of F-1. Further, pretraining MedType on our novel datasets achieves state-of-the-art performance for semantic type prediction in biomedical text.

CONCLUSIONS

Semantic type prediction is a key part of building accurate NLP pipelines for broad-coverage information extraction from biomedical text. We make our source code and novel datasets publicly available to foster reproducible research.

Artificial Intelligence for Identifying the Prevention of Medication Incidents Causing Serious or Moderate Harm: An Analysis Using Incident Reporters' Views

The purpose of this study was to describe incident reporters’ views identified by artificial intelligence concerning the prevention of medication incidents that were assessed, causing serious or moderate harm to patients. The information identified the most important risk management areas in these medication incidents. This was a retrospective record review using medication-related incident reports from one university hospital in Finland between January 2017 and December 2019 (n = 3496). Of these, incidents that caused serious or moderate harm to patients (n = 137) were analysed using artificial intelligence. Artificial intelligence classified reporters’ views on preventing incidents under the following main categories: (1) treatment, (2) working, (3) practices, and (4) setting and multiple sub-categories. The following risk management areas were identified: (1) verification, documentation and up-to-date drug doses, drug lists and other medication information, (2) carefulness and accuracy in managing medications, (3) ensuring the flow of information and communication regarding medication information and safeguarding continuity of patient care, (4) availability, update and compliance with instructions and guidelines, (5) multi-professional cooperation, and (6) adequate human resources, competence and suitable workload. Artificial intelligence was found to be useful and effective to classifying text-based data, such as the free text of incident reports.

Finding Potential Adverse Events in the Unstructured Text of Electronic Health Care Records: Development of the Shakespeare Method

BACKGROUND

Big data tools provide opportunities to monitor adverse events (patient harm associated with medical care) (AEs) in the unstructured text of electronic health care records (EHRs). Writers may explicitly state an apparent association between treatment and adverse outcome ("attributed") or state the simple treatment and outcome without an association ("unattributed"). Many methods for finding AEs in text rely on predefining possible AEs before searching for prespecified words and phrases or manual labeling (standardization) by investigators. We developed a method to identify possible AEs, even if unknown or unattributed, without any prespecifications or standardization of notes. Our method was inspired by word-frequency analysis methods used to uncover the true authorship of disputed works credited to William Shakespeare. We chose two use cases, "transfusion" and "time-based." Transfusion was chosen because new transfusion AE types were becoming recognized during the study data period; therefore, we anticipated an opportunity to find unattributed potential AEs (PAEs) in the notes. With the time-based case, we wanted to simulate near real-time surveillance. We chose time periods in the hope of detecting PAEs due to contaminated heparin from mid-2007 to mid-2008 that were announced in early 2008. We hypothesized that the prevalence of contaminated heparin may have been widespread enough to manifest in EHRs through symptoms related to heparin AEs, independent of clinicians' documentation of attributed AEs.

OBJECTIVE

We aimed to develop a new method to identify attributed and unattributed PAEs using the unstructured text of EHRs.

METHODS

We used EHRs for adult critical care admissions at a major teaching hospital (2001-2012). For each case, we formed a group of interest and a comparison group. We concatenated the text notes for each admission into one document sorted by date, and deleted replicate sentences and lists. We identified statistically significant words in the group of interest versus the comparison group. Documents in the group of interest were filtered to those words, followed by topic modeling on the filtered documents to produce topics. For each topic, the three documents with the maximum topic scores were manually reviewed to identify PAEs.

RESULTS

Topics centered around medical conditions that were unique to or more common in the group of interest, including PAEs. In each use case, most PAEs were unattributed in the notes. Among the transfusion PAEs was unattributed evidence of transfusion-associated cardiac overload and transfusion-related acute lung injury. Some of the PAEs from mid-2007 to mid-2008 were increased unattributed events consistent with AEs related to heparin contamination.

CONCLUSIONS

The Shakespeare method could be a useful supplement to AE reporting and surveillance of structured EHR data. Future improvements should include automation of the manual review process.

Modern Clinical Text Mining: A Guide and Review

Electronic health records (EHRs) are becoming a vital source of data for healthcare quality improvement, research, and operations. However, much of the most valuable information contained in EHRs remains buried in unstructured text. The field of clinical text mining has advanced rapidly in recent years, transitioning from rule-based approaches to machine learning and, more recently, deep learning. With new methods come new challenges, however, especially for those new to the field. This review provides an overview of clinical text mining for those who are encountering it for the first time (e.g., physician researchers, operational analytics teams, machine learning scientists from other domains). While not a comprehensive survey, this review describes the state of the art, with a particular focus on new tasks and methods developed over the past few years. It also identifies key barriers between these remarkable technical advances and the practical realities of implementation in health systems and in industry.

Viewpoint: An Information Extraction Framework for Disability Determination Using a Mental Functioning Use-Case (Preprint)

Natural language processing (NLP) in health care enables transformation of complex narrative information into high value products such as clinical decision support and adverse event monitoring in real time via the electronic health record (EHR). However, information technologies for mental health have consistently lagged because of the complexity of measuring and modeling mental health and illness. The use of NLP to support management of mental health conditions is a viable topic that has not been explored in depth. This paper provides a framework for the advanced application of NLP methods to identify, extract, and organize information on mental health and functioning to inform the decision-making process applied to assessing mental health. We present a use-case related to work disability, guided by the disability determination process of the US Social Security Administration (SSA). From this perspective, the following questions must be addressed about each problem that leads to a disability benefits claim: When did the problem occur and how long has it existed? How severe is it? Does it affect the person’s ability to work? and What is the source of the evidence about the problem? Our framework includes 4 dimensions of medical information that are central to assessing disability—temporal sequence and duration, severity, context, and information source. We describe key aspects of each dimension and promising approaches for application in mental functioning. For example, to address temporality, a complete functional timeline must be created with all relevant aspects of functioning such as intermittence, persistence, and recurrence. Severity of mental health symptoms can be successfully identified and extracted on a 4-level ordinal scale from absent to severe. Some NLP work has been reported on the extraction of context for specific cases of wheelchair use in clinical settings. We discuss the links between the task of information source assessment and work on source attribution, coreference resolution, event extraction, and rule-based methods. Gaps were identified in NLP applications that directly applied to the framework and in existing relevant annotated data sets. We highlighted NLP methods with the potential for advanced application in the field of mental functioning. Findings of this work will inform the development of instruments for supporting SSA adjudicators in their disability determination process. The 4 dimensions of medical information may have relevance for a broad array of individuals and organizations responsible for assessing mental health function and ability. Further, our framework with 4 specific dimensions presents significant opportunity for the application of NLP in the realm of mental health and functioning beyond the SSA setting, and it may support the development of robust tools and methods for decision-making related to clinical care, program implementation, and other outcomes.

Natural language processing with deep learning for medical adverse event detection from free-text medical narratives: A case study of detecting total hip replacement dislocation

BACKGROUND

Accurate and timely detection of medical adverse events (AEs) from free-text medical narratives can be challenging. Natural language processing (NLP) with deep learning has already shown great potential for analyzing free-text data, but its application for medical AE detection has been limited.

METHOD

In this study, we developed deep learning based NLP (DL-NLP) models for efficient and accurate hip dislocation AE detection following primary total hip replacement from standard (radiology notes) and non-standard (follow-up telephone notes) free-text medical narratives. We benchmarked these proposed models with traditional machine learning based NLP (ML-NLP) models, and also assessed the accuracy of International Classification of Diseases (ICD) and Current Procedural Terminology (CPT) codes in capturing these hip dislocation AEs in a multi-center orthopaedic registry.

RESULTS

All DL-NLP models outperformed all of the ML-NLP models, with a convolutional neural network (CNN) model achieving the best overall performance (Kappa = 0.97 for radiology notes, and Kappa = 1.00 for follow-up telephone notes). On the other hand, the ICD/CPT codes of the patients who sustained a hip dislocation AE were only 75.24% accurate.

CONCLUSIONS

We demonstrated that a DL-NLP model can be used in largescale orthopaedic registries for accurate and efficient detection of hip dislocation AEs. The NLP model in this study was developed with data from the most frequently used electronic medical record (EMR) system in the U.S., Epic. This NLP model could potentially be implemented in other Epic-based EMR systems to improve AE detection, and consequently, quality of care and patient outcomes.

Development and Validation of a Deep Learning Model for Detection of Allergic Reactions Using Safety Event Reports Across Hospitals

IMPORTANCE

Although critical to patient safety, health care-related allergic reactions are challenging to identify and monitor.

OBJECTIVE

To develop a deep learning model to identify allergic reactions in the free-text narrative of hospital safety reports and evaluate its generalizability, efficiency, productivity, and interpretability.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study analyzed hospital safety reports filed between May 2004 and January 2019 at Brigham and Women's Hospital and between April 2006 and June 2018 at Massachusetts General Hospital in Boston. Training and validating a deep learning model involved extracting safety reports using 101 expert-curated keywords from Massachusetts General Hospital (data set I). The model was then evaluated on 3 data sets: reports without keywords (data set II), reports from a different time frame (data set III), and reports from a different hospital (Brigham and Women's Hospital; data set IV). Statistical analyses were performed between March 1, 2019, and July 18, 2020.

MAIN OUTCOMES AND MEASURES

The area under the receiver operating characteristic curve and area under the precision-recall curve were used on data set I. The precision at top-k was used on data sets II to IV.

RESULTS

A total of 299 028 safety reports with 172 854 patients were included. Of these patients, 86 544 were women (50.1%) and the median (interquartile range [IQR]) age was 59.7 (43.8-71.6) years. The deep learning model achieved an area under the receiver operating characteristic curve of 0.979 (95% CI, 0.973-0.985) and an area under the precision-recall curve of 0.809 (95% CI, 0.773-0.845). The model achieved precisions at the top 100 model-identified cases of 0.930 in data set II, 0.960 in data set III, and 0.990 in data set IV. Compared with the keyword-search approach, the deep learning model reduced the number of cases for manual review by 63.8% and identified 24.2% more cases of confirmed allergic reactions. The model highlighted important words (eg, rash, hives, and Benadryl) in prediction and extended the list of expert-curated keywords through an attention layer.

CONCLUSIONS AND RELEVANCE

This study showed that a deep learning model can accurately and efficiently identify allergic reactions using free-text narratives written by a variety of health care professionals. This model could be used to improve allergy care, potentially enabling real-time event surveillance and guidance for medical errors and system improvement.

Evaluation of a Concept Mapping Task Using Named Entity Recognition and Normalization in Unstructured Clinical Text

In this pilot study, we explore the feasibility and accuracy of using a query in a commercial natural language processing engine in a named entity recognition and normalization task to extract a wide spectrum of clinical concepts from free text clinical letters. Editorial guidance developed by two independent clinicians was used to annotate sixty anonymized clinic letters to create the gold standard. Concepts were categorized by semantic type, and labels were applied to indicate contextual attributes such as negation. The natural language processing (NLP) engine was Linguamatics I2E version 5.3.1, equipped with an algorithm for contextualizing words and phrases and an ontology of terms from Intelligent Medical Objects to which those tokens were mapped. Performance of the engine was assessed on a training set of the documents using precision, recall, and the F1 score, with subset analysis for semantic type, accurate negation, exact versus partial conceptual matching, and discontinuous text. The engine underwent tuning, and the final performance was determined for a test set. The test set showed an F1 score of 0.81 and 0.84 using strict and relaxed criteria respectively when appropriate negation was not required and 0.75 and 0.77 when it was. F1 scores were higher when concepts were derived from continuous text only. This pilot study showed that a commercially available NLP engine delivered good overall results for identifying a wide spectrum of structured clinical concepts. Such a system holds promise for extracting concepts from free text to populate problem lists or for data mining projects.

Can Unified Medical Language System-based semantic representation improve automated identification of patient safety incident reports by type and severity?

Objective

The study sought to evaluate the feasibility of using Unified Medical Language System (UMLS) semantic features for automated identification of reports about patient safety incidents by type and severity.

Materials and Methods

Binary support vector machine (SVM) classifier ensembles were trained and validated using balanced datasets of critical incident report texts (n_type = 2860, n_severity = 1160) collected from a state-wide reporting system. Generalizability was evaluated on different and independent hospital-level reporting system. Concepts were extracted from report narratives using the UMLS Metathesaurus, and their relevance and frequency were used as semantic features. Performance was evaluated by F-score, Hamming loss, and exact match score and was compared with SVM ensembles using bag-of-words (BOW) features on 3 testing datasets (type/severity: n_benchmark = 286/116, n_original = 444/4837, n_independent =6000/5950).

Results

SVMs using semantic features met or outperformed those based on BOW features to identify 10 different incident types (F-score [semantics/BOW]: benchmark = 82.6%/69.4%; original = 77.9%/68.8%; independent = 78.0%/67.4%) and extreme-risk events (F-score [semantics/BOW]: benchmark = 87.3%/87.3%; original = 25.5%/19.8%; independent = 49.6%/52.7%). For incident type, the exact match score for semantic classifiers was consistently higher than BOW across all test datasets (exact match [semantics/BOW]: benchmark = 48.9%/39.9%; original = 57.9%/44.4%; independent = 59.5%/34.9%).

Discussion

BOW representations are not ideal for the automated identification of incident reports because they do not account for text semantics. UMLS semantic representations are likely to better capture information in report narratives, and thus may explain their superior performance.

Conclusions

UMLS-based semantic classifiers were effective in identifying incidents by type and extreme-risk events, providing better generalizability than classifiers using BOW.