Using multiclass classification to automate the identification of patient safety incident reports by type and severity

Construction of a Multi-Label Classifier for Extracting Multiple Incident Factors From Medication Incident Reports in Residential Care Facilities: Natural Language Processing Approach

BACKGROUND

Medication safety in residential care facilities is a critical concern, particularly when nonmedical staff provide medication assistance. The complex nature of medication-related incidents in these settings, coupled with the psychological impact on health care providers, underscores the need for effective incident analysis and preventive strategies. A thorough understanding of the root causes, typically through incident-report analysis, is essential for mitigating medication-related incidents.

OBJECTIVE

We aimed to develop and evaluate a multilabel classifier using natural language processing to identify factors contributing to medication-related incidents using incident report descriptions from residential care facilities, with a focus on incidents involving nonmedical staff.

METHODS

We analyzed 2143 incident reports, comprising 7121 sentences, from residential care facilities in Japan between April 1, 2015, and March 31, 2016. The incident factors were annotated using sentences based on an established organizational factor model and previous research findings. The following 9 factors were defined: procedure adherence, medicine, resident, resident family, nonmedical staff, medical staff, team, environment, and organizational management. To assess the label criteria, 2 researchers with relevant medical knowledge annotated a subset of 50 reports; the interannotator agreement was measured using Cohen κ. The entire data set was subsequently annotated by 1 researcher. Multiple labels were assigned to each sentence. A multilabel classifier was developed using deep learning models, including 2 Bidirectional Encoder Representations From Transformers (BERT)-type models (Tohoku-BERT and a University of Tokyo Hospital BERT pretrained with Japanese clinical text: UTH-BERT) and an Efficiently Learning Encoder That Classifies Token Replacements Accurately (ELECTRA), pretrained on Japanese text. Both sentence- and report-level training were performed; the performance was evaluated by the F1-score and exact match accuracy through 5-fold cross-validation.

RESULTS

Among all 7121 sentences, 1167, 694, 2455, 23, 1905, 46, 195, 1104, and 195 included "procedure adherence," "medicine," "resident," "resident family," "nonmedical staff," "medical staff," "team," "environment," and "organizational management," respectively. Owing to limited labels, "resident family" and "medical staff" were omitted from the model development process. The interannotator agreement values were higher than 0.6 for each label. A total of 10, 278, and 1855 reports contained no, 1, and multiple labels, respectively. The models trained using the report data outperformed those trained using sentences, with macro F1-scores of 0.744, 0.675, and 0.735 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. The report-trained models also demonstrated better exact match accuracy, with 0.411, 0.389, and 0.399 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. Notably, the accuracy was consistent even when the analysis was confined to reports containing multiple labels.

CONCLUSIONS

The multilabel classifier developed in our study demonstrated potential for identifying various factors associated with medication-related incidents using incident reports from residential care facilities. Thus, this classifier can facilitate prompt analysis of incident factors, thereby contributing to risk management and the development of preventive strategies.

A large dataset of annotated incident reports on medication errors

Incident reports of medication errors are valuable learning resources for improving patient safety. However, pertinent information is often contained within unstructured free text, which prevents automated analysis and limits the usefulness of these data. Natural language processing can structure this free text automatically and retrieve relevant past incidents and learning materials, but to be able to do so requires a large, fully annotated and validated corpus of incident reports. We present a corpus of 58,658 machine-annotated incident reports of medication errors that can be used to advance the development of information extraction models and subsequent incident learning. We report the best F1-scores for the annotated dataset: 0.97 and 0.76 for named entity recognition and intention/factuality analysis, respectively, for the cross-validation exercise. Our dataset contains 478,175 named entities and differentiates between incident types by recognising discrepancies between what was intended and what actually occurred. We explain our annotation workflow and technical validation and provide access to the validation datasets and machine annotator for labelling future incident reports of medication errors.

Risk Management and Patient Safety in the Artificial Intelligence Era: A Systematic Review

BACKGROUND

Healthcare systems represent complex organizations within which multiple factors (physical environment, human factor, technological devices, quality of care) interconnect to form a dense network whose imbalance is potentially able to compromise patient safety. In this scenario, the need for hospitals to expand reactive and proactive clinical risk management programs is easily understood, and artificial intelligence fits well in this context. This systematic review aims to investigate the state of the art regarding the impact of AI on clinical risk management processes. To simplify the analysis of the review outcomes and to motivate future standardized comparisons with any subsequent studies, the findings of the present review will be grouped according to the possibility of applying AI in the prevention of the different incident type groups as defined by the ICPS.

MATERIALS AND METHODS

On 3 November 2023, a systematic review of the literature according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was carried out using the SCOPUS and Medline (via PubMed) databases. A total of 297 articles were identified. After the selection process, 36 articles were included in the present systematic review.

RESULTS AND DISCUSSION

The studies included in this review allowed for the identification of three main "incident type" domains: clinical process, healthcare-associated infection, and medication. Another relevant application of AI in clinical risk management concerns the topic of incident reporting.

CONCLUSIONS

This review highlighted that AI can be applied transversely in various clinical contexts to enhance patient safety and facilitate the identification of errors. It appears to be a promising tool to improve clinical risk management, although its use requires human supervision and cannot completely replace human skills. To facilitate the analysis of the present review outcome and to enable comparison with future systematic reviews, it was deemed useful to refer to a pre-existing taxonomy for the identification of adverse events. However, the results of the present study highlighted the usefulness of AI not only for risk prevention in clinical practice, but also in improving the use of an essential risk identification tool, which is incident reporting. For this reason, the taxonomy of the areas of application of AI to clinical risk processes should include an additional class relating to risk identification and analysis tools. For this purpose, it was considered convenient to use ICPS classification.

A Machine Learning Approach with Human-AI Collaboration for Automated Classification of Patient Safety Event Reports: Algorithm Development and Validation Study

BACKGROUND

Adverse events refer to incidents with potential or actual harm to patients in hospitals. These events are typically documented through patient safety event (PSE) reports, which consist of detailed narratives providing contextual information on the occurrences. Accurate classification of PSE reports is crucial for patient safety monitoring. However, this process faces challenges due to inconsistencies in classifications and the sheer volume of reports. Recent advancements in text representation, particularly contextual text representation derived from transformer-based language models, offer a promising solution for more precise PSE report classification. Integrating the machine learning (ML) classifier necessitates a balance between human expertise and artificial intelligence (AI). Central to this integration is the concept of explainability, which is crucial for building trust and ensuring effective human-AI collaboration.

OBJECTIVE

This study aims to investigate the efficacy of ML classifiers trained using contextual text representation in automatically classifying PSE reports. Furthermore, the study presents an interface that integrates the ML classifier with the explainability technique to facilitate human-AI collaboration for PSE report classification.

METHODS

This study used a data set of 861 PSE reports from a large academic hospital's maternity units in the Southeastern United States. Various ML classifiers were trained with both static and contextual text representations of PSE reports. The trained ML classifiers were evaluated with multiclass classification metrics and the confusion matrix. The local interpretable model-agnostic explanations (LIME) technique was used to provide the rationale for the ML classifier's predictions. An interface that integrates the ML classifier with the LIME technique was designed for incident reporting systems.

RESULTS

The top-performing classifier using contextual representation was able to obtain an accuracy of 75.4% (95/126) compared to an accuracy of 66.7% (84/126) by the top-performing classifier trained using static text representation. A PSE reporting interface has been designed to facilitate human-AI collaboration in PSE report classification. In this design, the ML classifier recommends the top 2 most probable event types, along with the explanations for the prediction, enabling PSE reporters and patient safety analysts to choose the most suitable one. The LIME technique showed that the classifier occasionally relies on arbitrary words for classification, emphasizing the necessity of human oversight.

CONCLUSIONS

This study demonstrates that training ML classifiers with contextual text representations can significantly enhance the accuracy of PSE report classification. The interface designed in this study lays the foundation for human-AI collaboration in the classification of PSE reports. The insights gained from this research enhance the decision-making process in PSE report classification, enabling hospitals to more efficiently identify potential risks and hazards and enabling patient safety analysts to take timely actions to prevent patient harm.

The use of natural language processing in detecting and predicting falls within the healthcare setting: a systematic review

Falls are a common problem associated with significant morbidity, mortality, and economic costs. Current fall prevention policies in local healthcare settings are often guided by information provided by fall risk assessment tools, incident reporting, and coding data. This review was conducted with the aim of identifying studies which utilized natural language processing (NLP) for the automated detection and prediction of falls in the healthcare setting. The databases Ovid Medline, Ovid Embase, Ovid Emcare, PubMed, CINAHL, IEEE Xplore, and Ei Compendex were searched from 2012 until April 2023. Retrospective derivation, validation, and implementation studies wherein patients experienced falls within a healthcare setting were identified for inclusion. The initial search yielded 2611 publications for title and abstract screening. Full-text screening was conducted on 105 publications, resulting in 26 unique studies that underwent qualitative analyses. Studies applied NLP towards falls risk factor identification, known falls detection, future falls prediction, and falls severity stratification with reasonable success. The NLP pipeline was reviewed in detail between studies and models utilizing rule-based, machine learning (ML), deep learning (DL), and hybrid approaches were examined. With a growing literature surrounding falls prediction in both inpatient and outpatient environments, the absence of studies examining the impact of these models on patient and system outcomes highlights the need for further implementation studies. Through an exploration of the application of NLP techniques, it may be possible to develop models with higher performance in automated falls prediction and detection.

Blockchain for Patient Safety: Use Cases, Opportunities and Open Challenges

Medical errors are recognized as major threats to patient safety worldwide. Lack of streamlined communication and an inability to share and exchange data are among the contributory factors affecting patient safety. To address these challenges, blockchain can be utilized to ensure a secure, transparent and decentralized data exchange among stakeholders. In this study, we discuss six use cases that can benefit from blockchain to gain operational effectiveness and efficiency in the patient safety context. The role of stakeholders, system requirements, opportunities and challenges are discussed in each use case in detail. Connecting stakeholders and data in complex healthcare systems, blockchain has the potential to provide an accountable and collaborative milieu for the delivery of safe care. By reviewing the potential of blockchain in six use cases, we suggest that blockchain provides several benefits, such as an immutable and transparent structure and decentralized architecture, which may help transform health care and enhance patient safety. While blockchain offers remarkable opportunities, it also presents open challenges in the form of trust, privacy, scalability and governance. Future research may benefit from including additional use cases and developing smart contracts to present a more comprehensive view on potential contributions and challenges to explore the feasibility of blockchain-based solutions in the patient safety context.

Development of a Novel Scoring System to Quantify the Severity of Incident Reports: An Exploratory Research Study

Incident reporting systems have been widely adopted to collect information about patient safety incidents. Much of the value of incident reports lies in the free-text section. Computer processing of semantic information may be helpful to analyze this. We developed a novel scoring system for decision making to assess the severity of incidents using the semantic characteristics of the text in incident reports, and compared its results with experts' opinions. We retrospectively analyzed free-text data from incident reports from January 2012 to September 2021 at Nagoya University Hospital, Aichi, Japan. The sample was allocated to training and validation datasets using the hold-out method. Morphological analysis was used to segment terms in the training dataset. We calculated a severity term score, a severity report score and severity group score, by report volume size, and compared these with conventional severity classifications by patient safety experts and reporters. We allocated 96,082 incident reports into two groups. We calculated 1,802 severity term scores from the 48,041 reports in the training dataset. There was a significant difference in severity report score between reports categorized as severe and not severe by experts (95% confidence interval [CI] -0.83 to -0.80, p < 0.001, d = 0.81). Severity group scores were positively associated with severity ratings from experts and reporters (correlation coefficients 0.73 [95% CI 0.63-0.80, p < 0.001] and 0.79 [95% CI 0.71-0.85, p < 0.001]) for all departments. Our severity scoring system could therefore contribute to better organizational patient safety.

Using Healthcare Resources Wisely: A Predictive Support System Regarding the Severity of Patient Falls

Background

An injurious fall is one of the main indicators of care quality in healthcare facilities. Despite several fall screen tools being widely used to evaluate a patient's fall risk, they are frequently unable to reveal the severity level of patient falls. The purpose of this study is to build a practical system useful to predict the severity level of in-hospital falls. This practice is done in order to better allocate limited healthcare resources and to improve overall patient safety.

Methods

Four hundred and forty-six patients who experienced fall events at a large Taiwanese hospital were referenced. Eight predictors were used to ascertain the severity of patient falls solely based on the above study population. Multinomial logistic regression, Naïve Bayes, random forest, support vector machine, eXtreme gradient boosting, deep learning, and ensemble learning were adopted to establish predictive models. Accuracy, F1 score, precision, and recall were utilized to assess the models' performance.

Results

Compared to other learners, random forest exhibited satisfying predictive performance in terms of all metrics (accuracy: 0.844, F1 score: 0.850, precision: 0.839, and recall: 0.875 for the test dataset), and it was adopted as the base learner for a severity-level predictive system which is web-based. Furthermore, age, ability of independent activity, patient sources, use of assistive devices, and fall history within the past 12 months were deemed the top five important risk factors for evaluating fall severity.

Conclusions

The application of machine learning techniques for predicting the severity level of patient falls may result in some benefits to monitor fall severity and to better allocate limited healthcare resources.

Analyzing adverse drug reaction using statistical and machine learning methods: A systematic review

BACKGROUND

Adverse drug reactions (ADRs) are unintended negative drug-induced responses. Determining the association between drugs and ADRs is crucial, and several methods have been proposed to demonstrate this association. This systematic review aimed to examine the analytical tools by considering original articles that utilized statistical and machine learning methods for detecting ADRs.

METHODS

A systematic literature review was conducted based on articles published between 2015 and 2020. The keywords used were statistical, machine learning, and deep learning methods for detecting ADR signals. The study was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (PRISMA) guidelines.

RESULTS

We reviewed 72 articles, of which 51 and 21 addressed statistical and machine learning methods, respectively. Electronic medical record (EMR) data were exclusively analyzed using the regression method. For FDA Adverse Event Reporting System (FAERS) data, components of the disproportionality method were preferable. DrugBank was the most used database for machine learning. Other methods accounted for the highest and supervised methods accounted for the second highest.

CONCLUSIONS

Using the 72 main articles, this review provides guidelines on which databases are frequently utilized and which analysis methods can be connected. For statistical analysis, >90% of the cases were analyzed by disproportionate or regression analysis with each spontaneous reporting system (SRS) data or electronic medical record (EMR) data; for machine learning research, however, there was a strong tendency to analyze various data combinations. Only half of the DrugBank database was occupied, and the k-nearest neighbor method accounted for the greatest proportion.

A Machine Learning Approach to Reclassifying Miscellaneous Patient Safety Event Reports

BACKGROUND AND OBJECTIVES

Medical errors are a leading cause of death in the United States. Despite widespread adoption of patient safety reporting systems to address medical errors, making sense of the reports collected in these systems is challenging in practice. Event classification taxonomies used in many reporting systems can be complex and difficult to understand by frontline reporters, leading reporters to classify reports as "miscellaneous" as opposed to assigning a specific event-type category, which may facilitate analysis.

METHODS

To assist patient safety analysts in their analysis of "miscellaneous" reports, we developed an ensemble machine learning natural language processing model to reclassify these reports. We integrated the model into a clinical workflow dashboard, evaluated user feedback, and compared differences in user thresholds for model performance.

RESULTS AND CONCLUSIONS

Integrating an ensemble model to classify "miscellaneous" event reports with an interactive visualization was helpful to patient safety analysts review "miscellaneous" reports. However, patient safety analysts have different thresholds for model reclassification depending on their role and experience with "miscellaneous" event reports.

Content Analysis of Patient Safety Incident Reports for Older Adult Patient Transfers, Handovers, and Discharges: Do They Serve Organizations, Staff, or Patients?

OBJECTIVE

The aim of the study was to analyze content of incident reports during patient transitions in the context of care of older people, cardiology, orthopedics, and stroke.

METHODS

A structured search strategy identified incident reports involving patient transitions (March 2014-August 2014, January 2015-June 2015) within 2 National Health Service Trusts (in upper and lower quartiles of incident reports/100 admissions) in care of older people, cardiology, orthopedics, and stroke. Content analysis identified the following: incident classifications; active failures; latent conditions; patient/relative involvement; and evidence of individual or organizational learning. Reported harm was interpreted with reference to National Reporting and Learning System criteria.

RESULTS

A total 278 incident reports were analyzed. Fourteen incident classifications were identified, with pressure ulcers the modal category (n = 101,36%), followed by falls (n = 32, 12%), medication (n = 31, 11%), and documentation (n = 29, 10%). Half (n = 139, 50%) of incident reports related to interunit/department/team transfers. Latent conditions were explicit in 33 (12%) reports; most frequently, these related to inadequate resources/staff and concomitant time pressures (n = 13). Patient/family involvement was explicit in 61 (22%) reports. Patient well-being was explicit in 24 (9%) reports. Individual and organizational learning was evident in 3% and 7% of reports, respectively. Reported harm was significantly lower than coder-interpreted harm (P < 0.0001).

CONCLUSIONS

Incident report quality was suboptimal for individual and organizational learning. Underreporting level of harm suggests reporter bias, which requires reducing as much as practicable. System-level interventions are warranted to encourage use of staff reflective skills, emphasizing joint ownership of incidents. Co-producing incident reports with other clinicians involved in the transition and patients/relatives could optimize organizational learning.

Novel Telephone-Based Interactive Voice Response System for Incident Reporting

BACKGROUND

The voluntary reporting of medical errors and near misses is a well-established patient safety reporting mechanism. However, studies suggest that these incident reporting systems (IRSs) detect less than 10% of all adverse events. Improving the process of reporting can facilitate more informative and timely data capture while providing more opportunities to improve health care quality and safety. The purpose of this study was to understand the barriers to incident reporting via the existing Web-based IRS and develop solutions to increase the ease and efficiency of reporting.

METHODS

A survey of staff in a diagnostic imaging department in St. Catharines, Ontario was performed to identify barriers to incident reporting. Based on the barriers identified, two methods of incident reporting were tested in successive phases: (1) a phone-based voice message mailbox, in the computed tomography suite; and (2) a phone-based structured interactive voice response system (IVRS), across the entire department. We measured the rate of incident reports/day and time required to complete reports.

OUTCOMES

The three most common barriers to reporting identified were lack of time, complexity of reporting system, and lack of feedback. There was a significant difference in reports per day for the IVRS (mean [M] = 3.43, standard deviation [SD] = 2.71) compared to the IRS (M = 0.99, SD = 0.55); t(31) = 4.58, p ≤ 0.00001. There was also a significant difference in the average time to make a report for the IVRS (M = 97 seconds [s], SD = 30 s) compared to the IRS (M = 644 s, SD = 90 s); t(4) =13.55, p = 0.00025.

CONCLUSION

IVRS is an innovative approach to incident reporting that may prove to be more efficient than Web-based approaches and encourage higher reporting rates.

Developing an Analytical Pipeline to Classify Patient Safety Event Reports Using Optimized Predictive Algorithms

BACKGROUND

 Patient safety event reports provide valuable insight into systemic safety issues but deriving insights from these reports requires computational tools to efficiently parse through large volumes of qualitative data. Natural language processing (NLP) combined with predictive learning provides an automated approach to evaluating these data and supporting the work of patient safety analysts.

OBJECTIVES

 The objective of this study was to use NLP and machine learning techniques to develop a generalizable, scalable, and reliable approach to classifying event reports for the purpose of driving improvements in the safety and quality of patient care.

METHODS

 Datasets for 14 different labels (themes) were vectorized using a bag-of-words, tf-idf, or document embeddings approach and then applied to a series of classification algorithms via a hyperparameter grid search to derive an optimized model. Reports were also analyzed for terms strongly associated with each theme using an adjusted F-score calculation.

RESULTS

 F₁ score for each optimized model ranged from 0.951 ("Fall") to 0.544 ("Environment"). The bag-of-words approach proved optimal for 12 of 14 labels, and the naïve Bayes algorithm performed best for nine labels. Linear support vector machine was demonstrated as optimal for three labels and XGBoost for four of the 14 labels. Labels with more distinctly associated terms performed better than less distinct themes, as shown by a Pearson's correlation coefficient of 0.634.

CONCLUSIONS

 We were able to demonstrate an analytical pipeline that broadly applies NLP and predictive modeling to categorize patient safety reports from multiple facilities. This pipeline allows analysts to more rapidly identify and structure information contained in patient safety data, which can enhance the evaluation and the use of this information over time.

Multi-feature fusion framework for sarcasm identification on twitter data: A machine learning based approach

Sarcasm is the main reason behind the faulty classification of tweets. It brings a challenge in natural language processing (NLP) as it hampers the method of finding people’s actual sentiment. Various feature engineering techniques are being investigated for the automatic detection of sarcasm. However, most related techniques have always concentrated only on the content-based features in sarcastic expression, leaving the contextual information in isolation. This leads to a loss of the semantics of words in the sarcastic expression. Another drawback is the sparsity of the training data. Due to the word limit of microblog, the feature vector’s values for each sample constructed by BoW produces null features. To address the above-named problems, a Multi-feature Fusion Framework is proposed using two classification stages. The first stage classification is constructed with the lexical feature only, extracted using the BoW technique, and trained using five standard classifiers, including SVM, DT, KNN, LR, and RF, to predict the sarcastic tendency. In stage two, the constructed lexical sarcastic tendency feature is fused with eight other proposed features for modelling a context to obtain a final prediction. The effectiveness of the developed framework is tested with various experimental analysis to obtain classifiers’ performance. The evaluation shows that our constructed classification models based on the developed novel feature fusion obtained results with a precision of 0.947 using a Random Forest classifier. Finally, the obtained results were compared with the results of three baseline approaches. The comparison outcome shows the significance of the proposed framework.

Evaluating resampling methods and structured features to improve fall incident report identification by the severity level

OBJECTIVE

This study aims to improve the classification of the fall incident severity level by considering data imbalance issues and structured features through machine learning.

MATERIALS AND METHODS

We present an incident report classification (IRC) framework to classify the in-hospital fall incident severity level by addressing the imbalanced class problem and incorporating structured attributes. After text preprocessing, bag-of-words features, structured text features, and structured clinical features were extracted from the reports. Next, resampling techniques were incorporated into the training process. Machine learning algorithms were used to build classification models. IRC systems were trained, validated, and tested using a repeated and randomly stratified shuffle-split cross-validation method. Finally, we evaluated the system performance using the F1-measure, precision, and recall over 15 stratified test sets.

RESULTS

The experimental results demonstrated that the classification system setting considering both data imbalance issues and structured features outperformed the other system settings (with a mean macro-averaged F1-measure of 0.733). Considering the structured features and resampling techniques, this classification system setting significantly improved the mean F1-measure for the rare class by 30.88% (P value < .001) and the mean macro-averaged F1-measure by 8.26% from the baseline system setting (P value < .001). In general, the classification system employing the random forest algorithm and random oversampling method outperformed the others.

CONCLUSIONS

Structured features provide essential information for categorizing the fall incident severity level. Resampling methods help rebalance the class distribution of the original incident report data, which improves the performance of machine learning models. The IRC framework presented in this study effectively automates the identification of fall incident reports by the severity level.

Using convolutional neural networks to identify patient safety incident reports by type and severity

OBJECTIVE

To evaluate the feasibility of a convolutional neural network (CNN) with word embedding to identify the type and severity of patient safety incident reports.

MATERIALS AND METHODS

A CNN with word embedding was applied to identify 10 incident types and 4 severity levels. Model training and validation used data sets (n_type = 2860, n_severity = 1160) collected from a statewide incident reporting system. Generalizability was evaluated using an independent hospital-level reporting system. CNN architectures were examined by varying layer size and hyperparameters. Performance was evaluated by F score, precision, recall, and compared to binary support vector machine (SVM) ensembles on 3 testing data sets (type/severity: n_benchmark = 286/116, n_original = 444/4837, n_independent = 6000/5950).

RESULTS

A CNN with 6 layers was the most effective architecture, outperforming SVMs with better generalizability to identify incidents by type and severity. The CNN achieved high F scores (> 85%) across all test data sets when identifying common incident types including falls, medications, pressure injury, and aggression. When identifying common severity levels (medium/low), CNN outperformed SVMs, improving F scores by 11.9%-45.1% across all 3 test data sets.

DISCUSSION

Automated identification of incident reports using machine learning is challenging because of a lack of large labelled training data sets and the unbalanced distribution of incident classes. The standard classification strategy is to build multiple binary classifiers and pool their predictions. CNNs can extract hierarchical features and assist in addressing class imbalance, which may explain their success in identifying incident report types.

CONCLUSION

A CNN with word embedding was effective in identifying incidents by type and severity, providing better generalizability than SVMs.

Application of Augmented Intelligence for Pharmacovigilance Case Seriousness Determination

INTRODUCTION

Identification of adverse events and determination of their seriousness ensures timely detection of potential patient safety concerns. Adverse event seriousness is a key factor in defining reporting timelines and is often performed manually by pharmacovigilance experts. The dramatic increase in the volume of safety reports necessitates exploration of scalable solutions that also meet reporting timeline requirements.

OBJECTIVE

The aim of this study was to develop an augmented intelligence methodology for automatically identifying adverse event seriousness in spontaneous, solicited, and medical literature safety reports. Deep learning models were evaluated for accuracy and/or the F1 score against a ground truth labeled by pharmacovigilance experts.

METHODS

Using a stratified random sample of safety reports received by Celgene, we developed three neural networks for addressing identification of adverse event seriousness: (1) a binary adverse-event level seriousness classifier; (2) a classifier for determining seriousness categorization at the adverse-event level; and (3) an annotator for identifying seriousness criteria terms to provide supporting evidence at the document level.

RESULTS

The seriousness classifier achieved an accuracy of 83.0% in post-marketing reports, 92.9% in solicited reports, and 86.3% in medical literature reports. F1 scores for seriousness categorization were 77.7 for death, 78.9 for hospitalization, and 75.5 for important medical events. The seriousness annotator achieved an F1 score of 89.9 in solicited reports, and 75.2 in medical literature reports.

CONCLUSIONS

The results of this study indicate that a neural network approach can provide an accurate and scalable solution for potentially augmenting pharmacovigilance practitioner determination of adverse event seriousness in spontaneous, solicited, and medical literature reports.

Use of machine learning in geriatric clinical care for chronic diseases: a systematic literature review

Objectives

Geriatric clinical care is a multidisciplinary assessment designed to evaluate older patients’ (age 65 years and above) functional ability, physical health, and cognitive well-being. The majority of these patients suffer from multiple chronic conditions and require special attention. Recently, hospitals utilize various artificial intelligence (AI) systems to improve care for elderly patients. The purpose of this systematic literature review is to understand the current use of AI systems, particularly machine learning (ML), in geriatric clinical care for chronic diseases.

Materials and Methods

We restricted our search to eight databases, namely PubMed, WorldCat, MEDLINE, ProQuest, ScienceDirect, SpringerLink, Wiley, and ERIC, to analyze research articles published in English between January 2010 and June 2019. We focused on studies that used ML algorithms in the care of geriatrics patients with chronic conditions.

Results

We identified 35 eligible studies and classified in three groups: psychological disorder (n = 22), eye diseases (n = 6), and others (n = 7). This review identified the lack of standardized ML evaluation metrics and the need for data governance specific to health care applications.

Conclusion

More studies and ML standardization tailored to health care applications are required to confirm whether ML could aid in improving geriatric clinical care.

Can Unified Medical Language System-based semantic representation improve automated identification of patient safety incident reports by type and severity?

Objective

The study sought to evaluate the feasibility of using Unified Medical Language System (UMLS) semantic features for automated identification of reports about patient safety incidents by type and severity.

Materials and Methods

Binary support vector machine (SVM) classifier ensembles were trained and validated using balanced datasets of critical incident report texts (n_type = 2860, n_severity = 1160) collected from a state-wide reporting system. Generalizability was evaluated on different and independent hospital-level reporting system. Concepts were extracted from report narratives using the UMLS Metathesaurus, and their relevance and frequency were used as semantic features. Performance was evaluated by F-score, Hamming loss, and exact match score and was compared with SVM ensembles using bag-of-words (BOW) features on 3 testing datasets (type/severity: n_benchmark = 286/116, n_original = 444/4837, n_independent =6000/5950).

Results

SVMs using semantic features met or outperformed those based on BOW features to identify 10 different incident types (F-score [semantics/BOW]: benchmark = 82.6%/69.4%; original = 77.9%/68.8%; independent = 78.0%/67.4%) and extreme-risk events (F-score [semantics/BOW]: benchmark = 87.3%/87.3%; original = 25.5%/19.8%; independent = 49.6%/52.7%). For incident type, the exact match score for semantic classifiers was consistently higher than BOW across all test datasets (exact match [semantics/BOW]: benchmark = 48.9%/39.9%; original = 57.9%/44.4%; independent = 59.5%/34.9%).

Discussion

BOW representations are not ideal for the automated identification of incident reports because they do not account for text semantics. UMLS semantic representations are likely to better capture information in report narratives, and thus may explain their superior performance.

Conclusions

UMLS-based semantic classifiers were effective in identifying incidents by type and extreme-risk events, providing better generalizability than classifiers using BOW.

Automatic Incident Triage in Radiation Oncology Incident Learning System

The Radiotherapy Incident Reporting and Analysis System (RIRAS) receives incident reports from Radiation Oncology facilities across the US Veterans Health Affairs (VHA) enterprise and Virginia Commonwealth University (VCU). In this work, we propose a computational pipeline for analysis of radiation oncology incident reports. Our pipeline uses machine learning (ML) and natural language processing (NLP) based methods to predict the severity of the incidents reported in the RIRAS platform using the textual description of the reported incidents. These incidents in RIRAS are reviewed by a radiation oncology subject matter expert (SME), who initially triages some incidents based on the salient elements in the incident report. To automate the triage process, we used the data from the VHA treatment centers and the VCU radiation oncology department. We used NLP combined with traditional ML algorithms, including support vector machine (SVM) with linear kernel, and compared it against the transfer learning approach with the universal language model fine-tuning (ULMFiT) algorithm. In RIRAS, severities are divided into four categories; A, B, C, and D, with A being the most severe to D being the least. In this work, we built models to predict High (A & B) vs. Low (C & D) severity instead of all the four categories. Models were evaluated with macro-averaged precision, recall, and F1-Score. The Traditional ML machine learning (SVM-linear) approach did well on the VHA dataset with 0.78 F1-Score but performed poorly on the VCU dataset with 0.5 F1-Score. The transfer learning approach did well on both datasets with 0.81 F1-Score on VHA dataset and 0.68 F1-Score on the VCU dataset. Overall, our methods show promise in automating the triage and severity determination process from radiotherapy incident reports.

Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review

BACKGROUND

Artificial intelligence (AI) provides opportunities to identify the health risks of patients and thus influence patient safety outcomes.

OBJECTIVE

The purpose of this systematic literature review was to identify and analyze quantitative studies utilizing or integrating AI to address and report clinical-level patient safety outcomes.

METHODS

We restricted our search to the PubMed, PubMed Central, and Web of Science databases to retrieve research articles published in English between January 2009 and August 2019. We focused on quantitative studies that reported positive, negative, or intermediate changes in patient safety outcomes using AI apps, specifically those based on machine-learning algorithms and natural language processing. Quantitative studies reporting only AI performance but not its influence on patient safety outcomes were excluded from further review.

RESULTS

We identified 53 eligible studies, which were summarized concerning their patient safety subcategories, the most frequently used AI, and reported performance metrics. Recognized safety subcategories were clinical alarms (n=9; mainly based on decision tree models), clinical reports (n=21; based on support vector machine models), and drug safety (n=23; mainly based on decision tree models). Analysis of these 53 studies also identified two essential findings: (1) the lack of a standardized benchmark and (2) heterogeneity in AI reporting.

CONCLUSIONS

This systematic review indicates that AI-enabled decision support systems, when implemented correctly, can aid in enhancing patient safety by improving error detection, patient stratification, and drug management. Future work is still needed for robust validation of these systems in prospective and real-world clinical environments to understand how well AI can predict safety outcomes in health care settings.

Clinical Text Data in Machine Learning: Systematic Review

Background

Clinical narratives represent the main form of communication within health care, providing a personalized account of patient history and assessments, and offering rich information for clinical decision making. Natural language processing (NLP) has repeatedly demonstrated its feasibility to unlock evidence buried in clinical narratives. Machine learning can facilitate rapid development of NLP tools by leveraging large amounts of text data.

Objective

The main aim of this study was to provide systematic evidence on the properties of text data used to train machine learning approaches to clinical NLP. We also investigated the types of NLP tasks that have been supported by machine learning and how they can be applied in clinical practice.

Methods

Our methodology was based on the guidelines for performing systematic reviews. In August 2018, we used PubMed, a multifaceted interface, to perform a literature search against MEDLINE. We identified 110 relevant studies and extracted information about text data used to support machine learning, NLP tasks supported, and their clinical applications. The data properties considered included their size, provenance, collection methods, annotation, and any relevant statistics.

Results

The majority of datasets used to train machine learning models included only hundreds or thousands of documents. Only 10 studies used tens of thousands of documents, with a handful of studies utilizing more. Relatively small datasets were utilized for training even when much larger datasets were available. The main reason for such poor data utilization is the annotation bottleneck faced by supervised machine learning algorithms. Active learning was explored to iteratively sample a subset of data for manual annotation as a strategy for minimizing the annotation effort while maximizing the predictive performance of the model. Supervised learning was successfully used where clinical codes integrated with free-text notes into electronic health records were utilized as class labels. Similarly, distant supervision was used to utilize an existing knowledge base to automatically annotate raw text. Where manual annotation was unavoidable, crowdsourcing was explored, but it remains unsuitable because of the sensitive nature of data considered. Besides the small volume, training data were typically sourced from a small number of institutions, thus offering no hard evidence about the transferability of machine learning models. The majority of studies focused on text classification. Most commonly, the classification results were used to support phenotyping, prognosis, care improvement, resource management, and surveillance.

Conclusions

We identified the data annotation bottleneck as one of the key obstacles to machine learning approaches in clinical NLP. Active learning and distant supervision were explored as a way of saving the annotation efforts. Future research in this field would benefit from alternatives such as data augmentation and transfer learning, or unsupervised learning, which do not require data annotation.

Toward systems-centered analysis of patient safety events: Improving root cause analysis by optimized incident classification and information presentation

BACKGROUND

Systems-centered root cause analysis (RCA) of patient safety events presents unique advantages as it aims to disclose vulnerabilities of healthcare systems. However, the increasing number of collected events poses the problems of low efficiency and information overload for traditional RCA.

OBJECTIVES

This study aims to improve systems-centered RCA by developing optimized information extraction and presentation.

METHODS

We experimented supervised machine-learning methods to extract safety-related information from 3333 de-identified patient safety event reports from two independent sources. Based on the extracted information, we further evaluated how optimized information presentation could help facilitate the disclosure of system vulnerabilities in traditional RCA.

RESULTS

Multilabel text classification is effective in identifying safety-related information from the narrative description of patient safety events. The Pruned Sets in conjunction with Naïve Bayes are the outperformed algorithm in one dataset, with an overall F score of 60.0 % and the highest F score of 96.0 % for identifying "Adverse Drug Reaction". The Classifier Chains in conjunction with Naïve Bayes are the outperformed algorithm in another dataset, with an overall F score of 43.2 % and the highest F score of 64.0 % for identifying "Medication". During the RCA, human experts applied the optimized presentation of information which showed advantages of identifying system vulnerabilities.

CONCLUSION

Our study demonstrated the feasibility of using multilabel text classification for identifying safety-related information from the narrative description of patient safety events. The extracted information when grouped by safety-related information can better aid human experts to conduct systems-centered RCA and disclose system vulnerabilities.

A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis

CONTEXT

Adverse events in healthcare are often collated in incident reports which contain unstructured free text. Learning from these events may improve patient safety. Natural language processing (NLP) uses computational techniques to interrogate free text, reducing the human workload associated with its analysis. There is growing interest in applying NLP to patient safety, but the evidence in the field has not been summarised and evaluated to date.

OBJECTIVE

To perform a systematic literature review and narrative synthesis to describe and evaluate NLP methods for classification of incident reports and adverse events in healthcare.

METHODS

Data sources included Medline, Embase, The Cochrane Library, CINAHL, MIDIRS, ISI Web of Science, SciELO, Google Scholar, PROSPERO, hand searching of key articles, and OpenGrey. Data items were manually abstracted to a standardised extraction form.

RESULTS

From 428 articles screened for eligibility, 35 met the inclusion criteria of using NLP to perform a classification task on incident reports, or with the aim of detecting adverse events. The majority of studies used free text from incident reporting systems or electronic health records. Models were typically designed to classify by type of incident, type of medication error, or harm severity. A broad range of NLP techniques are demonstrated to perform these classification tasks with favourable performance outcomes. There are methodological challenges in how these results can be interpreted in a broader context.

CONCLUSION

NLP can generate meaningful information from unstructured data in the specific domain of the classification of incident reports and adverse events. Understanding what or why incidents are occurring is important in adverse event analysis. If NLP enables these insights to be drawn from larger datasets it may improve the learning from adverse events in healthcare.

Automated classification of primary care patient safety incident report content and severity using supervised machine learning (ML) approaches

Learning from patient safety incident reports is a vital part of improving healthcare. However, the volume of reports and their largely free-text nature poses a major analytic challenge. The objective of this study was to test the capability of autonomous classifying of free text within patient safety incident reports to determine incident type and the severity of harm outcome. Primary care patient safety incident reports (n=31333) previously expert-categorised by clinicians (training data) were processed using J48, SVM and Naïve Bayes.The SVM classifier was the highest scoring classifier for incident type (AUROC, 0.891) and severity of harm (AUROC, 0.708). Incident reports containing deaths were most easily classified, correctly identifying 72.82% of reports. In conclusion, supervised ML can be used to classify patient safety incident report categories. The severity classifier, whilst not accurate enough to replace manual processing, could provide a valuable screening tool for this critical aspect of patient safety.

Current challenges in health information technology-related patient safety

We identify and describe nine key, short-term, challenges to help healthcare organizations, health information technology developers, researchers, policymakers, and funders focus their efforts on health information technology-related patient safety. Categorized according to the stage of the health information technology lifecycle where they appear, these challenges relate to (1) developing models, methods, and tools to enable risk assessment; (2) developing standard user interface design features and functions; (3) ensuring the safety of software in an interfaced, network-enabled clinical environment; (4) implementing a method for unambiguous patient identification (1-4 Design and Development stage); (5) developing and implementing decision support which improves safety; (6) identifying practices to safely manage information technology system transitions (5 and 6 Implementation and Use stage); (7) developing real-time methods to enable automated surveillance and monitoring of system performance and safety; (8) establishing the cultural and legal framework/safe harbor to allow sharing information about hazards and adverse events; and (9) developing models and methods for consumers/patients to improve health information technology safety (7-9 Monitoring, Evaluation, and Optimization stage). These challenges represent key "to-do's" that must be completed before we can expect to have safe, reliable, and efficient health information technology-based systems required to care for patients.

An automated pipeline for analyzing medication event reports in clinical settings

BACKGROUND

Medication events in clinical settings are significant threats to patient safety. Analyzing and learning from the medication event reports is an important way to prevent the recurrence of these events. Currently, the analysis of medication event reports is ineffective and requires heavy workloads for clinicians. An automated pipeline is proposed to help clinicians deal with the accumulated reports, extract valuable information and generate feedback from the reports. Thus, the strategy of medication event prevention can be further developed based on the lessons learned.

METHODS

In order to build the automated pipeline, four classic machine learning classifiers (i.e., support vector machine, Naïve Bayes, random forest, and multi-layer perceptron) were compared to identify the event originating stages, event types, and event causes from the medication event reports. The precision, recall and F-1 measure were calculated to assess the performance of the classifiers. Further, a strategy to measure the similarity of medication event reports in our pipeline was established and evaluated by human subjects through a questionnaire.

RESULTS

We developed three classifiers to identify the medication event originating stages, event types and causes, respectively. For the event originating stages, a support vector machine classifier obtains the best performance with an F-1 measure of 0.792. For the event types, a support vector machine classifier exhibits the best performance with an F-1 measure of 0.758. And for the event causes, a random forest classifier reaches an F-1 measure of 0.925. The questionnaire results show that the similarity measurement is consistent with the domain experts in the task of identifying similar reports.

CONCLUSION

We developed and evaluated an automated pipeline that could identify three attributes from the medication event reports and calculate the similarity scores between the reports based on the attributes. The pipeline is expected to improve the efficiency of analyzing the medication event reports and to learn from the reports in a timely manner.

Analyzing Medication Error Reports in Clinical Settings: An Automated Pipeline Approach

Medication error is a severe patient safety event in the United States. Medication error reports collected by Patient Safety Organizations provide an opportunity to analyze and learn from previous errors. However, the current workflow of analyzing the error reports is labor-intensive and time-consuming. To reduce the workloads for clinicians and save time, we developed a pipeline for medication error report pre-analysis by applying automated text classification techniques. The pipeline was proven functional in two tasks, i.e., identifying the error originated stages, error types and error causes from the medication error reports, and calculating the similarity scores between the error reports for re-organization. The proposed pipeline holds promise in helping clinicians understand the nature of medication error in an error report, and better manage the error reports, which could further facilitate the prevention of medication errors in healthcare settings.

What Can We Learn about Fall Risk Factors from EHR Nursing Notes? A Text Mining Study

INTRODUCTION

Hospital falls are a continuing clinical concern, with over one million falls occurring each year in the United States. Annually, hospital-acquired falls result in an estimated $34 billion in direct medical costs. Falls are considered largely preventable and, as a result, the Centers for Medicare and Medicaid Services have announced that fall-related injuries are no longer a reimbursable hospital cost. While policies and practices have been implemented to reduce falls, little sustained reduction has been achieved. Little empirical evidence supports the validity of published fall risk factors. While chart abstraction has been used to operationalize risk factors, few studies have examined registered nurses' (RNs') narrative notes as a source of actionable data. Therefore, the purpose of our study was to explore whether there is meaningful fall risk and prevention information in RNs' electronic narrative notes.

METHODS

This study utilized a natural language processing design. Data for this study were extracted from the publicly available Medical Information Mart for Intensive Care (MIMIC-III) database. The date comprises deidentified EHR data associated with patients who stayed in critical care units between 2001 and 2012. Text mining procedures were performed on RN's narrative notes following the traditional steps of knowledge discovery.

RESULTS

The corpus of data extracted from MIMIC-III database was comprised of 1,046,053 RNs' notes from 36,583 unique patients. We identified 3,972 notes (0.4 percent) representing 1,789 (5 percent) patients with explicit documentation related to fall risk/prevention. Around 10 percent of the notes (103,685) from 23,025 patients mentioned intrinsic (patient-related) factors that have been theoretically associated with risk of falling. An additional 1,322 notes (0.1 percent) from 692 patients (2 percent) mentioned extrinsic risk factors, related to organizational design and environment. Moreover, 7672 notes (0.7 percent) from 2,571 patients (7 percent) included information on interventions that could theoretically impact patient falls.

CONCLUSIONS

This exploratory study using a NLP approach revealed that meaningful information related to fall risk and prevention may be found in RNs' narrative notes. In particular, RNs' notes can contain information about clinical as well as environmental and organizational factors that could affect fall risk but are not explicitly recorded by the provider as a fall risk factors. In our study, potential fall risk factors were documented for more than half of the sample. Further research is needed to determine the predictive value of these factors.

IMPLICATIONS FOR POLICY OR PRACTICE

This study highlights a potentially rich but understudied source of actionable fall risk data. Furthermore, the application of novel methods to identify quality and safety measures in RNs' notes can facilitate inclusion of RNs' voices in patient outcomes and health services research.

10,000 Good Catches: Increasing Safety Event Reporting In A Pediatric Health Care System

Background:

In 2014, Children’s National Health System’s executive leadership team challenged the organization to double the number of voluntary safety event reports submitted over a 3-year period; the intent was to increase reliability and promote our safety culture by hardwiring employee event reporting.

Methods:

Following a Donabedian quality improvement framework of structure, process, and outcomes, a multidisciplinary team was formed and areas for improvement were identified. The multidisciplinary team focused on 3 major areas: the perceived ease of reporting (ie, how difficult is it to report an event?); the perceived safety of reporting (ie, will I get in trouble for reporting?); and the perceived impact of reporting (ie, does my report make a difference?) technology, making it safe to report, and how reporting makes a difference. The team developed a key driver diagram and implemented interventions designed to impact the key drivers and to increase reporting.

Results:

Children’s National increased the number of safety event reports from 4,668 in fiscal year 2014 to 10,971 safety event reports in fiscal year 2017. Median event report submission time was decreased by nearly 30%, anonymous reporting decreased by 69%, the number of submitting departments increased by 94%, and the number of reports submitted as “other” decreased from a baseline of 6% to 2%.

Conclusions:

Children’s National Health System’s focus on increasing safety event reporting resulted in increased organizational engagement and attention. This initiative served as a tangible step to improve organizational reliability and the culture of safety and is readily generalizable to other hospitals.

A Systematic Machine Learning Based Approach for the Diagnosis of Non-Alcoholic Fatty Liver Disease Risk and Progression

Prevention and diagnosis of NAFLD is an ongoing area of interest in the healthcare community. Screening is complicated by the fact that the accuracy of noninvasive testing lacks specificity and sensitivity to make and stage the diagnosis. Currently no non-invasive ATP III criteria based prediction method is available to diagnose NAFLD risk. Firstly, the objective of this research is to develop machine learning based method in order to identify individuals at an increased risk of developing NAFLD using risk factors of ATP III clinical criteria updated in 2005 for Metabolic Syndrome (MetS). Secondly, to validate the relative ability of quantitative score defined by Italian Association for the Study of the Liver (IASF) and guideline explicitly defined for the Canadian population based on triglyceride thresholds to predict NAFLD risk. We proposed a Decision Tree based method to evaluate the risk of developing NAFLD and its progression in the Canadian population, using Electronic Medical Records (EMRs) by exploring novel risk factors for NAFLD. Our results show proposed method could potentially help physicians make more informed choices about their management of patients with NAFLD. Employing the proposed application in ordinary medical checkup is expected to lessen healthcare expenditures compared with administering additional complicated test.