Automated detection of adverse events using natural language processing of discharge summaries

The Impact of Artificial Intelligence on Allergy Diagnosis and Treatment

PURPOSE OF REVIEW

Artificial intelligence (AI), be it neuronal networks, machine learning or deep learning, has numerous beneficial effects on healthcare systems; however, its potential applications and diagnostic capabilities for immunologic diseases have yet to be explored. Understanding AI systems can help healthcare workers better assimilate artificial intelligence into their practice and unravel its potential in diagnostics, clinical research, and disease management.

RECENT FINDINGS

We reviewed recent advancements in AI systems and their integration in healthcare systems, along with their potential benefits in the diagnosis and management of diseases. We explored machine learning as employed in allergy diagnosis and its learning patterns from patient datasets, as well as the possible advantages of using AI in the field of research related to allergic reactions and even remote monitoring. Considering the ethical challenges and privacy concerns raised by clinicians and patients with regard to integrating AI in healthcare, we explored the new guidelines adapted by regulatory bodies. Despite these challenges, AI appears to have been successfully incorporated into various healthcare systems and is providing patient-centered solutions while simultaneously assisting healthcare workers. Artificial intelligence offers new hope in the field of immunologic disease diagnosis, monitoring, and management and thus has the potential to revolutionize healthcare systems.

Clinical Applications of Machine Learning

Objective

This review introduces interpretable predictive machine learning approaches, natural language processing, image recognition, and reinforcement learning methodologies to familiarize end users.

Background

As machine learning, artificial intelligence, and generative artificial intelligence become increasingly utilized in clinical medicine, it is imperative that end users understand the underlying methodologies.

Methods

This review describes publicly available datasets that can be used with interpretable predictive approaches, natural language processing, image recognition, and reinforcement learning models, outlines result interpretation, and provides references for in-depth information about each analytical framework.

Results

This review introduces interpretable predictive machine learning models, natural language processing, image recognition, and reinforcement learning methodologies.

Conclusions

Interpretable predictive machine learning models, natural language processing, image recognition, and reinforcement learning are core machine learning methodologies that underlie many of the artificial intelligence methodologies that will drive the future of clinical medicine and surgery. End users must be well versed in the strengths and weaknesses of these tools as they are applied to patient care now and in the future.

Using Large Language Models to Support Content Analysis: A Case Study of ChatGPT for Adverse Event Detection

This study explores the potential of using large language models to assist content analysis by conducting a case study to identify adverse events (AEs) in social media posts. The case study compares ChatGPT's performance with human annotators' in detecting AEs associated with delta-8-tetrahydrocannabinol, a cannabis-derived product. Using the identical instructions given to human annotators, ChatGPT closely approximated human results, with a high degree of agreement noted: 94.4% (9436/10,000) for any AE detection (Fleiss κ=0.95) and 99.3% (9931/10,000) for serious AEs (κ=0.96). These findings suggest that ChatGPT has the potential to replicate human annotation accurately and efficiently. The study recognizes possible limitations, including concerns about the generalizability due to ChatGPT's training data, and prompts further research with different models, data sources, and content analysis tasks. The study highlights the promise of large language models for enhancing the efficiency of biomedical research.

Real-World Treatment Patterns and Clinical Outcomes among Patients Receiving CDK4/6 Inhibitors for Metastatic Breast Cancer in a Canadian Setting Using AI-Extracted Data

Cyclin-dependent kinase 4/6 inhibitors (CDK4/6i) are widely used in patients with hormone receptor-positive (HR+)/human epidermal growth factor receptor 2 negative (HER2-) advanced/metastatic breast cancer (ABC/MBC) in first line (1L), but little is known about their real-world use and clinical outcomes long-term, in Canada. This study used Pentavere's previously validated artificial intelligence (AI) to extract real-world data on the treatment patterns and outcomes of patients receiving CDK4/6i+endocrine therapy (ET) for HR+/HER2- ABC/MBC at Sinai Health in Toronto, Canada. Between 1 January 2016 and 1 July 2021, 48 patients were diagnosed with HR+/HER2- ABC/MBC and received CDK4/6i + ET. A total of 38 out of 48 patients received CDK4/6i + ET in 1L, of which 34 of the 38 (89.5%) received palbociclib + ET. In 2L, 12 of the 21 (57.1%) patients received CDK4/6i + ET, of which 58.3% received abemaciclib. In 3L, most patients received chemotherapy (10/12, 83.3%). For the patients receiving CDK4/6i in 1L, the median (95% CI) time to the next treatment was 42.3 (41.2, NA) months. The median (95% CI) time to chemotherapy was 46.5 (41.4, NA) months. The two-year overall survival (95% CI) was 97.4% (92.4, 100.0), and the median (range) follow-up was 28.7 (3.4-67.6) months. Despite the limitations inherent in real-world studies and a limited number of patients, these AI-extracted data complement previous studies, demonstrating the effectiveness of CDK4/6i + ET in the Canadian real-world 1L, with most patients receiving palbociclib as CDK4/6i in 1L.

Potential Applications of Artificial Intelligence (AI) in Managing Polypharmacy in Saudi Arabia: A Narrative Review

Prescribing medications is a fundamental practice in the management of illnesses that necessitates in-depth knowledge of clinical pharmacology. Polypharmacy, or the concurrent use of multiple medications by individuals with complex health conditions, poses significant challenges, including an increased risk of drug interactions and adverse reactions. The Saudi Vision 2030 prioritises enhancing healthcare quality and safety, including addressing polypharmacy. Artificial intelligence (AI) offers promising tools to optimise medication plans, predict adverse drug reactions and ensure drug safety. This review explores AI's potential to revolutionise polypharmacy management in Saudi Arabia, highlighting practical applications, challenges and the path forward for the integration of AI solutions into healthcare practices.

Effectiveness of ChatGPT in clinical pharmacy and the role of artificial intelligence in medication therapy management

BACKGROUND

The use of artificial intelligence (AI) to optimize medication therapy management (MTM) in identifying drug interactions may potentially improve MTM efficiency. ChatGPT, an AI language model, may be applied to identify medication interventions by integrating patient and drug databases. ChatGPT has been shown to be effective in other areas of clinical medicine, from diagnosis to patient management. However, ChatGPT's ability to manage MTM related activities is little known.

OBJECTIVES

To evaluate the effectiveness of ChatGPT in MTM services in simple, complex, and very complex cases to understand AI contributions in MTM.

METHODS

Two clinical pharmacists rated and validated the difficulty of patient cases from simple, complex, and very complex. ChatGPT's response to the cases was assessed based on 3 criteria: the ability to identify drug interactions, precision in recommending alternatives, and appropriateness in devising management plans. Two clinical pharmacists validated the accuracy of ChatGPT's responses and compared them to actual answers for each complexity level.

RESULTS

ChatGPT 4.0 accurately solved 39 out of 39 (100 %) patient cases. ChatGPT successfully identified drug interactions, provided therapy recommendations and formulated general management plans, but it did not recommend specific dosages. Results suggest it can assist pharmacists in formulating MTM plans to improve overall efficiency.

CONCLUSION

The application of ChatGPT in MTM has the potential to enhance patient safety and involvement, lower healthcare costs, and assist healthcare providers in medication management and identifying drug interactions. Future pharmacists can utilize AI models such as ChatGPT to improve patient care. The future of the pharmacy profession will depend on how the field responds to the changing need for patient care optimized by AI and automation.

Natural language processing for identification of refractory status epilepticus in children

OBJECTIVE

Pediatric status epilepticus is one of the most frequent pediatric emergencies, with high mortality and morbidity. Utilizing electronic health records (EHRs) permits analysis of care approaches and disease outcomes at a lower cost than prospective research. However, reviewing EHR manually is time intensive. We aimed to compare refractory status epilepticus (rSE) cases identified by human EHR review with a natural language processing (NLP)-assisted rSE screen followed by a manual review.

METHODS

We used the NLP screening tool Document Review Tool (DrT) to generate regular expressions, trained a bag-of-words NLP classifier on EHRs from 2017 to 2019, and then tested our algorithm on data from February to December 2012. We compared results from manual review to NLP-assisted search followed by manual review.

RESULTS

Our algorithm identified 1528 notes in the test set. After removing notes pertaining to the same event by DrT, the user reviewed a total number of 400 notes to find patients with rSE. Within these 400 notes, we identified 31 rSE cases, including 12 new cases not found in manual review, and 19 of the 20 previously identified cases. The NLP-assisted model found 31 of 32 cases, with a sensitivity of 96.88% (95% CI = 82%-99.84%), whereas manual review identified 20 of 32 cases, with a sensitivity of 62.5% (95% CI = 43.75%-78.34%).

SIGNIFICANCE

DrT provided a highly sensitive model compared to human review and an increase in patient identification through EHRs. The use of DrT is a suitable application of NLP for identifying patients with a history of recent rSE, which ultimately contributes to the implementation of monitoring techniques and treatments in near real time.

A text mining approach to categorize patient safety event reports by medication error type

Patient safety reporting systems give healthcare provider staff the ability to report medication related safety events and errors; however, many of these reports go unanalyzed and safety hazards go undetected. The objective of this study is to examine whether natural language processing can be used to better categorize medication related patient safety event reports. 3,861 medication related patient safety event reports that were previously annotated using a consolidated medication error taxonomy were used to develop three models using the following algorithms: (1) logistic regression, (2) elastic net, and (3) XGBoost. After development, models were tested, and model performance was analyzed. We found the XGBoost model performed best across all medication error categories. 'Wrong Drug', 'Wrong Dosage Form or Technique or Route', and 'Improper Dose/Dose Omission' categories performed best across the three models. In addition, we identified five words most closely associated with each medication error category and which medication error categories were most likely to co-occur. Machine learning techniques offer a semi-automated method for identifying specific medication error types from the free text of patient safety event reports. These algorithms have the potential to improve the categorization of medication related patient safety event reports which may lead to better identification of important medication safety patterns and trends.

External Validation of Natural Language Processing Algorithms to Extract Common Data Elements in THA Operative Notes

BACKGROUND

Natural language processing (NLP) systems are distinctive in their ability to extract critical information from raw text in electronic health records (EHR). We previously developed three algorithms for total hip arthroplasty (THA) operative notes with rules aimed at capturing (1) operative approach, (2) fixation method, and (3) bearing surface using inputs from a single institution. The purpose of this study was to externally validate and improve these algorithms as a prerequisite for broader adoption in automated registry data curation.

METHODS

The previous NLP algorithms developed at Mayo Clinic were deployed and refined on EHRs from OrthoCarolina, evaluating 39 randomly selected primary THA operative reports from 2018 to 2021. Operative reports were available only in PDF format, requiring conversion to "readable" text with Adobe software. Accuracy statistics were calculated against manual chart review.

RESULTS

The operative approach, fixation technique, and bearing surface algorithms all demonstrated perfect accuracy of 100%. By comparison, validated performance at the developing center yielded an accuracy of 99.2% for operative approach, 90.7% for fixation technique, and 95.8% for bearing surface.

CONCLUSION

NLP algorithms applied to data from an external center demonstrated excellent accuracy in delineating common elements in THA operative notes. Notably, the algorithms had no functional problems evaluating scanned PDFs that were converted to "readable" text by common software. Taken together, these findings provide promise for NLP applied to scanned PDFs as a source to develop large registries by reliably extracting data of interest from very large unstructured data sets in an expeditious and cost-effective manner.

Toward alert triage: scalable qualitative coding framework for analyzing alert notes from the Telehealth Intervention Program for Seniors (TIPS)

Objective

Combined with mobile monitoring devices, telehealth generates overwhelming data, which could cause clinician burnout and overlooking critical patient status. Developing novel and efficient ways to correctly triage such data will be critical to a successful telehealth adoption. We aim to develop an automated classification framework of existing nurses' notes for each alert that will serve as a training dataset for a future alert triage system for telehealth programs.

Materials and Methods

We analyzed and developed a coding framework and a regular expression-based keyword match approach based on the information of 24 931 alert notes from a community-based telehealth program. We evaluated our automated alert triaging model for its scalability on a stratified sampling of 800 alert notes for precision and recall analysis.

Results

We found 22 717 out of 24 579 alert notes (92%) belonging to at least one of the 17 codes. The evaluation of the automated alert note analysis using the regular expression-based information extraction approach resulted in an average precision of 0.86 (SD = 0.13) and recall 0.90 (SD = 0.13).

Discussion

The high-performance results show the feasibility and the scalability potential of this approach in community-based, low-income older adult telehealth settings. The resulting coded alert notes can be combined with participants' health monitoring results to generate predictive models and to triage false alerts. The findings build steps toward developing an automated alert triaging model to improve the identification of alert types in remote health monitoring and telehealth systems.

Automated, machine learning-based alerts increase epilepsy surgery referrals: A randomized controlled trial

OBJECTIVE

To determine whether automated, electronic alerts increased referrals for epilepsy surgery.

METHODS

We conducted a prospective, randomized controlled trial of a natural language processing-based clinical decision support system embedded in the electronic health record (EHR) at 14 pediatric neurology outpatient clinic sites. Children with epilepsy and at least two prior neurology visits were screened by the system prior to their scheduled visit. Patients classified as a potential surgical candidate were randomized 2:1 for their provider to receive an alert or standard of care (no alert). The primary outcome was referral for a neurosurgical evaluation. The likelihood of referral was estimated using a Cox proportional hazards regression model.

RESULTS

Between April 2017 and April 2019, at total of 4858 children were screened by the system, and 284 (5.8%) were identified as potential surgical candidates. Two hundred four patients received an alert, and 96 patients received standard care. Median follow-up time was 24 months (range: 12-36 months). Compared to the control group, patients whose provider received an alert were more likely to be referred for a presurgical evaluation (3.1% vs 9.8%; adjusted hazard ratio [HR] = 3.21, 95% confidence interval [CI]: 0.95-10.8; one-sided p = .03). Nine patients (4.4%) in the alert group underwent epilepsy surgery, compared to none (0%) in the control group (one-sided p = .03).

SIGNIFICANCE

Machine learning-based automated alerts may improve the utilization of referrals for epilepsy surgery evaluations.

Developing an Inpatient Electronic Medical Record Phenotype for Hospital-Acquired Pressure Injuries: Case Study Using Natural Language Processing Models

BACKGROUND

Surveillance of hospital-acquired pressure injuries (HAPI) is often suboptimal when relying on administrative health data, as International Classification of Diseases (ICD) codes are known to have long delays and are undercoded. We leveraged natural language processing (NLP) applications on free-text notes, particularly the inpatient nursing notes, from electronic medical records (EMRs), to more accurately and timely identify HAPIs.

OBJECTIVE

This study aimed to show that EMR-based phenotyping algorithms are more fitted to detect HAPIs than ICD-10-CA algorithms alone, while the clinical logs are recorded with higher accuracy via NLP using nursing notes.

METHODS

Patients with HAPIs were identified from head-to-toe skin assessments in a local tertiary acute care hospital during a clinical trial that took place from 2015 to 2018 in Calgary, Alberta, Canada. Clinical notes documented during the trial were extracted from the EMR database after the linkage with the discharge abstract database. Different combinations of several types of clinical notes were processed by sequential forward selection during the model development. Text classification algorithms for HAPI detection were developed using random forest (RF), extreme gradient boosting (XGBoost), and deep learning models. The classification threshold was tuned to enable the model to achieve similar specificity to an ICD-based phenotyping study. Each model's performance was assessed, and comparisons were made between the metrics, including sensitivity, positive predictive value, negative predictive value, and F1-score.

RESULTS

Data from 280 eligible patients were used in this study, among whom 97 patients had HAPIs during the trial. RF was the optimal performing model with a sensitivity of 0.464 (95% CI 0.365-0.563), specificity of 0.984 (95% CI 0.965-1.000), and F1-score of 0.612 (95% CI of 0.473-0.751). The machine learning (ML) model reached higher sensitivity without sacrificing much specificity compared to the previously reported performance of ICD-based algorithms.

CONCLUSIONS

The EMR-based NLP phenotyping algorithms demonstrated improved performance in HAPI case detection over ICD-10-CA codes alone. Daily generated nursing notes in EMRs are a valuable data resource for ML models to accurately detect adverse events. The study contributes to enhancing automated health care quality and safety surveillance.

Practice-Based Learning and Improvement: Improving Morbidity and Mortality Review Using Natural Language Processing

INTRODUCTION

Practice-Based Learning and Improvement, a core competency identified by the Accreditation Council for Graduate Medical Education, carries importance throughout a physician's career. Practice-Based Learning and Improvement is cultivated by a critical review of complications, yet methods to accurately identify complications are inadequate. Machine-learning algorithms show promise in improving identification of complications. We compare a manual-supplemented natural language processing (ms-NLP) methodology against a validated electronic morbidity and mortality (MM) database, the Morbidity and Mortality Adverse Event Reporting System (MARS) to understand the utility of NLP in MM review.

METHODS

The number and severity of complications were compared between MARS and ms-NLP of surgical hospitalization discharge summaries among three academic medical centers. Clavien-Dindo (CD) scores were assigned to cases with identified complications and classified into minor (CD I-II) or major (CD III-IV) harm.

RESULTS

Of 7774 admissions, 987 cases were identified to have 1659 complications by MARS and 1296 by ms-NLP. MARS identified 611 (62%) cases, whereas ms-NLP identified 670 (68%) cases. Less than one-third of cases (299, 30.3%) were detected by both methods. MARS identified a greater number of complications with major harm (457, 46.30%) than did ms-NLP (P < 0.0001).

CONCLUSIONS

Both a prospectively maintained MM database and ms-NLP review of discharge summaries fail to identify a significant proportion of postoperative complications and overlap 1/3 of the time. ms-NLP more frequently identifies cases with minor complications, whereas prospective voluntary reporting more frequently identifies major complications. The educational benefit of reporting and analysis of complication data may be supplemented by ms-NLP but not replaced by it at this time.

Automated Detection of Patient Harm: Implementation and Prospective Evaluation of a Real-Time Broad-Spectrum Surveillance Application in a Hospital With Limited Resources

OBJECTIVES

This study aimed to prospectively validate an application that automates the detection of broad categories of hospital adverse events (AEs) extracted from a basic hospital information system, and to efficiently mobilize resources to reduce the level of acquired patient harm.

METHODS

Data were collected from an internally designed software, extracting results from 14 triggers indicative of patient harm, querying clinical and administrative databases including all inpatient admissions (n = 8760) from October 2019 to June 2020. Representative samples of the triggered cases were clinically validated using chart review by a consensus expert panel. The positive predictive value (PPV) of each trigger was evaluated, and the detection sensitivity of the surveillance system was estimated relative to incidence ranges in the literature.

RESULTS

The system identified 394 AEs among 946 triggered cases, associated with 291 patients, yielding an overall PPV of 42%. Variability was observed among the trigger PPVs and among the estimated detection sensitivities across the harm categories, the highest being for the healthcare-associated infections. The median length of stay of patients with an AE showed to be significantly higher than the median for the overall patient population.

CONCLUSIONS

This application was able to identify AEs across a broad spectrum of harm categories, in a real-time manner, while reducing the use of resources required by other harm detection methods. Such a system could serve as a promising patient safety tool for AE surveillance, allowing for timely, targeted, and resource-efficient interventions, even for hospitals with limited resources.

Developing a deep learning natural language processing algorithm for automated reporting of adverse drug reactions

The detection of adverse drug reactions (ADRs) is critical to our understanding of the safety and risk-benefit profile of medications. With an incidence that has not changed over the last 30 years, ADRs are a significant source of patient morbidity, responsible for 5%-10% of acute care hospital admissions worldwide. Spontaneous reporting of ADRs has long been the standard method of reporting, however this approach is known to have high rates of under-reporting, a problem that limits pharmacovigilance efforts. Automated ADR reporting presents an alternative pathway to increase reporting rates, although this may be limited by over-reporting of other drug-related adverse events. We developed a deep learning natural language processing algorithm to identify ADRs in discharge summaries at a single academic hospital centre. Our model was developed in two stages: first, a pre-trained model (DeBERTa) was further pre-trained on 1.1 million unlabelled clinical documents; secondly, this model was fine-tuned to detect ADR mentions in a corpus of 861 annotated discharge summaries. This model was compared to a version without the pre-training step, and a previously published RoBERTa model pretrained on MIMIC III, which has demonstrated strong performance on other pharmacovigilance tasks. To ensure that our algorithm could differentiate ADRs from other drug-related adverse events, the annotated corpus was enriched for both validated ADR reports and confounding drug-related adverse events using. The final model demonstrated good performance with a ROC-AUC of 0.955 (95% CI 0.933 - 0.978) for the task of identifying discharge summaries containing ADR mentions, significantly outperforming the two comparator models.

Implementation of Machine Learning Pipelines for Clinical Practice: Development and Validation Study

BACKGROUND

Artificial intelligence (AI) technologies, such as machine learning and natural language processing, have the potential to provide new insights into complex health data. Although powerful, these algorithms rarely move from experimental studies to direct clinical care implementation.

OBJECTIVE

We aimed to describe the key components for successful development and integration of two AI technology-based research pipelines for clinical practice.

METHODS

We summarized the approach, results, and key learnings from the implementation of the following two systems implemented at a large, tertiary care children's hospital: (1) epilepsy surgical candidate identification (or epilepsy ID) in an ambulatory neurology clinic; and (2) an automated clinical trial eligibility screener (ACTES) for the real-time identification of patients for research studies in a pediatric emergency department.

RESULTS

The epilepsy ID system performed as well as board-certified neurologists in identifying surgical candidates (with a sensitivity of 71% and positive predictive value of 77%). The ACTES system decreased coordinator screening time by 12.9%. The success of each project was largely dependent upon the collaboration between machine learning experts, research and operational information technology professionals, longitudinal support from clinical providers, and institutional leadership.

CONCLUSIONS

These projects showcase novel interactions between machine learning recommendations and providers during clinical care. Our deployment provides seamless, real-time integration of AI technology to provide decision support and improve patient care.

Electronic surveillance of patient safety events using natural language processing

Objective

We describe our approach to surveillance of reportable safety events captured in hospital data including free-text clinical notes. We hypothesize that a) some patient safety events are documented only in the clinical notes and not in any other accessible source; and b) large-scale abstraction of event data from clinical notes is feasible.

Materials and Methods

We use regular expressions to generate a training data set for a machine learning model and apply this model to the full set of clinical notes and conduct further review to identify safety events of interest. We demonstrate this approach on peripheral intravenous (PIV) infiltrations and extravasations (PIVIEs).

Results

During Phase 1, we collected 21,362 clinical notes, of which 2342 were reviewed. We identified 125 PIV events, of which 44 cases (35%) were not captured by other patient safety systems. During Phase 2, we collected 60,735 clinical notes and identified 440 infiltrate events. Our classifier demonstrated accuracy above 90%.

Conclusion

Our method to identify safety events from the free text of clinical documentation offers a feasible and scalable approach to enhance existing patient safety systems. Expert reviewers, using a machine learning model, can conduct routine surveillance of patient safety events.

Automated Ascertainment of Typhlitis From the Electronic Health Record

PURPOSE

Adverse events (AEs) on Children's Oncology Group (COG) trials are manually ascertained using Common Terminology Criteria for Adverse Events. Despite significant effort, we previously demonstrated that COG typhlitis reporting sensitivity was only 37% when compared with gold standard physician chart abstraction. This study tested an automated typhlitis identification algorithm using electronic health record data.

METHODS

Electronic health record data from children with leukemia age 0-22 years treated at a single institution from 2006 to 2019 were included. Patients were divided into derivation and validation cohorts. Rigorous chart abstraction of validation cohort patients established a gold standard AE data set. We created an automated algorithm to identify typhlitis matching Common Terminology Criteria for Adverse Events v5 that included antibiotics, neutropenia, and non-negated mention of typhlitis in a note. We iteratively refined the algorithm using the derivation cohort and then applied the algorithm to the validation cohort; performance was compared with the gold standard. For patients on trial AAML1031, COG AE report performance was compared with the gold standard.

RESULTS

The derivation cohort included 337 patients. The validation cohort included 270 patients (961 courses). Chart abstraction identified 16 courses with typhlitis. The algorithm identified 37 courses with typhlitis; 13 were true positives (sensitivity 81.3%, positive predictive value 35.1%). For patients on AAML1031, chart abstraction identified nine courses with typhlitis, and COG reporting correctly identified 4 (sensitivity 44.4%, positive predictive value 100.0%).

CONCLUSION

The automated algorithm identified true cases of typhlitis with higher sensitivity than COG reporting. The algorithm identified false positives but reduced the number of courses needing manual review by 96% (961 to 37) by detecting potential typhlitis. This algorithm could provide a useful screening tool to reduce manual effort required for typhlitis AE reporting.

Use of Diagnosis Codes to Find Blood Transfusion Adverse Events in Electronic Health Records

OBJECTIVE

Electronic health records (EHRs) and big data tools offer the opportunity for surveillance of adverse events (patient harm associated with medical care). We used International Classification of Diseases, Ninth Revision, codes in electronic records to identify known, and potentially novel, adverse reactions to blood transfusion.

METHODS

We used 49,331 adult admissions involving critical care at a major teaching hospital, 2001-2012, in the Medical Information Mart for Intensive Care III EHRs database. We formed a T (defined as packed red blood cells, platelets, or plasma) group of 21,443 admissions versus 25,468 comparison (C) admissions. The International Classification of Diseases, Ninth Revision, Clinical Modification , diagnosis codes were compared for T versus C, described, and tested with statistical tools.

RESULTS

Transfusion adverse events (TAEs) such as transfusion-associated circulatory overload (TACO; 12 T cases; rate ratio [RR], 15.61; 95% confidence interval [CI], 2.49-98) were found. There were also potential TAEs similar to TAEs, such as fluid overload disorder (361 T admissions; RR, 2.24; 95% CI, 1.88-2.65), similar to TACO. Some diagnoses could have been sequelae of TAEs, including nontraumatic compartment syndrome of abdomen (52 T cases; RR, 6.76; 95% CI, 3.40-14.9) possibly being a consequence of TACO.

CONCLUSIONS

Surveillance for diagnosis codes that could be TAE sequelae or unrecognized TAE might be useful supplements to existing medical product adverse event programs.

Development of a Quality Improvement Dental Chart Review Training Program

INTRODUCTION

Chart review is central to understanding adverse events (AEs) in medicine. In this article, we describe the process and results of educating chart reviewers assigned to evaluate dental AEs.

METHODS

We developed a Web-based training program, "Dental Patient Safety Training," which uses both independent and consensus-based curricula, for identifying AEs recorded in electronic health records in the dental setting. Training included (1) didactic education, (2) skills training using videos and guided walkthroughs, (3) quizzes with feedback, and (4) hands-on learning exercises. In addition, novice reviewers were coached weekly during consensus review discussions. TeamExpert was composed of 2 experienced reviewers, and TeamNovice included 2 chart reviewers in training. McNemar test, interrater reliability, sensitivity, specificity, positive predictive value, and negative predictive value were calculated to compare accuracy rates on the identification of charts containing AEs at the start of training and 7 months after consensus building discussions between the 2 teams.

RESULTS

TeamNovice completed independent and consensus development training. Initial chart reviews were conducted on a shared set of charts (n = 51) followed by additional training including consensus building discussions. There was a marked improvement in overall percent agreement, prevalence and bias-adjusted κ correlation, and diagnostic measures (sensitivity, specificity, positive predictive value, and negative predictive value) of reviewed charts between both teams from the phase I training program to phase II consensus building.

CONCLUSIONS

This study detailed the process of training new chart reviewers and evaluating their performance. Our results suggest that standardized training and continuous coaching improves calibration between experts and trained chart reviewers.

Multi-Center Validation of Natural Language Processing Algorithms for Detection of Common Data Elements in Operative Notes for Total Hip Arthroplasty (Preprint)

Background

Natural language processing (NLP) methods are powerful tools for extracting and analyzing critical information from free-text data. MedTaggerIE, an open-source NLP pipeline for information extraction based on text patterns, has been widely used in the annotation of clinical notes. A rule-based system, MedTagger-total hip arthroplasty (THA), developed based on MedTaggerIE, was previously shown to correctly identify the surgical approach, fixation, and bearing surface from the THA operative notes at Mayo Clinic.

Objective

This study aimed to assess the implementability, usability, and portability of MedTagger-THA at two external institutions, Michigan Medicine and the University of Iowa, and provide lessons learned for best practices.

Methods

We conducted iterative test-apply-refinement processes with three involved sites—the development site (Mayo Clinic) and two deployment sites (Michigan Medicine and the University of Iowa). Mayo Clinic was the primary NLP development site, with the THA registry as the gold standard. The activities at the two deployment sites included the extraction of the operative notes, gold standard development (Michigan: registry data; Iowa: manual chart review), the refinement of NLP algorithms on training data, and the evaluation of test data. Error analyses were conducted to understand language variations across sites. To further assess the model specificity for approach and fixation, we applied the refined MedTagger-THA to arthroscopic hip procedures and periacetabular osteotomy cases, as neither of these operative notes should contain any approach or fixation keywords.

Results

MedTagger-THA algorithms were implemented and refined independently for both sites. At Michigan, the study comprised THA-related notes for 2569 patient-date pairs. Before model refinement, MedTagger-THA algorithms demonstrated excellent accuracy for approach (96.6%, 95% CI 94.6%-97.9%) and fixation (95.7%, 95% CI 92.4%-97.6%). These results were comparable with internal accuracy at the development site (99.2% for approach and 90.7% for fixation). Model refinement improved accuracies slightly for both approach (99%, 95% CI 97.6%-99.6%) and fixation (98%, 95% CI 95.3%-99.3%). The specificity of approach identification was 88.9% for arthroscopy cases, and the specificity of fixation identification was 100% for both periacetabular osteotomy and arthroscopy cases. At the Iowa site, the study comprised an overall data set of 100 operative notes (50 training notes and 50 test notes). MedTagger-THA algorithms achieved moderate-high performance on the training data. After model refinement, the model achieved high performance for approach (100%, 95% CI 91.3%-100%), fixation (98%, 95% CI 88.3%-100%), and bearing surface (92%, 95% CI 80.5%-97.3%).

Conclusions

High performance across centers was achieved for the MedTagger-THA algorithms, demonstrating that they were sufficiently implementable, usable, and portable to different deployment sites. This study provided important lessons learned during the model deployment and validation processes, and it can serve as a reference for transferring rule-based electronic health record models.

Patient Safety Monitoring in Acute Care in a Decentralized National Health Care System: Conceptual Framework and Initial Set of Actionable Indicators

OBJECTIVES

Monitoring patient safety is critical for continuous quality improvement in acute care. We carried out a national project to identify a conceptual framework with core indicators that could be uniformly applied in the decentralized health system of Italy.

METHODS

We used key international references to identify a framework with a core list of indicators and data sources for calculation in 4 hospitals in the Lombardy region. Two different data processing methods were applied: (a) centralized analysis of national databases and (b) decentralized data extraction and calculation using different hospital data available in Lombardy.

RESULTS

Agreement was reached on a conceptual framework for patient safety monitoring in acute care, including structures, processes, and outcomes as vertical dimensions and health care needs as horizontal axes. We were able to compute 15 of 32 indicators through the application of a range of methods. The calculation of indicators using national databases was based on international standards. The consistency of the estimates obtained through the use of different methods and data sources seemed limited.

CONCLUSIONS

We successfully identified a conceptual framework for patient safety in acute care including actionable indicators that can be calculated routinely using different data sources at national, regional, and hospital levels. Further work is required to compare methods and understand whether a combination of strategies at national and local levels could be proven effective.

The Food and Drug Administration Biologics Effectiveness and Safety Initiative Facilitates Detection of Vaccine Administrations From Unstructured Data in Medical Records Through Natural Language Processing

Introduction: The Food and Drug Administration Center for Biologics Evaluation and Research conducts post-market surveillance of biologic products to ensure their safety and effectiveness. Studies have found that common vaccine exposures may be missing from structured data elements of electronic health records (EHRs), instead being captured in clinical notes. This impacts monitoring of adverse events following immunizations (AEFIs). For example, COVID-19 vaccines have been regularly administered outside of traditional medical settings. We developed a natural language processing (NLP) algorithm to mine unstructured clinical notes for vaccinations not captured in structured EHR data.

Methods: A random sample of 1,000 influenza vaccine administrations, representing 995 unique patients, was extracted from a large U.S. EHR database. NLP techniques were used to detect administrations from the clinical notes in the training dataset [80% (N = 797) of patients]. The algorithm was applied to the validation dataset [20% (N = 198) of patients] to assess performance. Full medical charts for 28 randomly selected administration events in the validation dataset were reviewed by clinicians. The NLP algorithm was then applied across the entire dataset (N = 995) to quantify the number of additional events identified.

Results: A total of 3,199 administrations were identified in the structured data and clinical notes combined. Of these, 2,740 (85.7%) were identified in the structured data, while the NLP algorithm identified 1,183 (37.0%) administrations in clinical notes; 459 were not also captured in the structured data. This represents a 16.8% increase in the identification of vaccine administrations compared to using structured data alone. The validation of 28 vaccine administrations confirmed 27 (96.4%) as “definite” vaccine administrations; 18 (64.3%) had evidence of a vaccination event in the structured data, while 10 (35.7%) were found solely in the unstructured notes.

Discussion: We demonstrated the utility of an NLP algorithm to identify vaccine administrations not captured in structured EHR data. NLP techniques have the potential to improve detection of vaccine administrations not otherwise reported without increasing the analysis burden on physicians or practitioners. Future applications could include refining estimates of vaccine coverage and detecting other exposures, population characteristics, and outcomes not reliably captured in structured EHR data.

A Keyword Approach to Identify Adverse Events Within Narrative Documents From 4 Italian Institutions

OBJECTIVES

Existing methods for measuring adverse events in hospitals intercept a restricted number of events. Text mining refers to a range of techniques to extract data from narrative sources. The goal of this study was to evaluate the performance of an automated approach for extracting adverse event keywords from within electronic health records.

METHODS

The study involved 4 medical centers in the Region of Lombardy. A starting set of keywords was trained in an iterative process to develop queries for 7 adverse events, including those used by the Agency for Healthcare Research and Quality as patient safety indicators. We calculated positive predictive values of the 7 queries and performed an error analysis to detect reasons for false-positive cases of pulmonary embolism, deep vein thrombosis, and urinary tract infection.

RESULTS

Overall, 397,233 records were collected (34,805 discharge summaries, 292,593 emergency department notes, and 69,835 operation reports). Positive predictive values were higher for postoperative wound dehiscence (83.83%) and urinary tract infection (73.07%), whereas they were lower for deep vein thrombosis (5.37%), pulmonary embolism (13.63%), and postoperative sepsis (12.28%). The most common reasons for false positives were reporting of past events (42.25%), negations (22.80%), and conditions suspected by physicians but not confirmed by a diagnostic test (11.25%).

CONCLUSIONS

The results of our study demonstrated the feasibility of using an automated approach to detect multiple adverse events in several data sources. More sophisticated techniques, such as natural language processing, should be tested to evaluate the feasibility of using text mining as a routine method for monitoring adverse events in hospitals.

Machine learning and natural language processing to identify falls in electronic patient care records from ambulance attendances

We derived machine learning models utilizing features generated by natural language processing (NLP) of free-text data from an ambulance services provider to identify fall cases. The data comprised samples of electronic patient care records care records (ePCRs) from St John Western Australia (WA), the sole ambulance services provider in most of WA. We manually labeled fall cases by reviewing the free-text summary. The models used features including case characteristics (e.g., age) and text frequency-inverse document frequency (tf-idf) of each word of the free-text generated by NLP. Support vector machine (SVM) and random forest were used as classifiers. We compared the performance of the models against the manual identification of falls by recall, precision, and F-measure. A total of 9,447 cases (1%) were randomly sampled, of which 1,648 (17%) were labeled as fall. The best model was an SVM model using case characteristics and tf-idf's of the first 100 words of free-text, with recall of 0.84, precision of 0.86, and F-measure of 0.85. This performance was better than an SVM model with only case characteristics. Machine-learning models incorporated with features generated by NLP improved the performance of classifying fall cases compared with models without such features. Scope remains for further improvement.

A weakly supervised model for the automated detection of adverse events using clinical notes

With clinical trials unable to detect all potential adverse reactions to drugs and medical devices prior to their release into the market, accurate post-market surveillance is critical to ensure their safety and efficacy. Electronic health records (EHR) contain rich observational patient data, making them a valuable source to actively monitor the safety of drugs and devices. While structured EHR data and spontaneous reporting systems often underreport the complexities of patient encounters and outcomes, free-text clinical notes offer greater detail about a patient's status. Previous studies have proposed machine learning methods to detect adverse events from clinical notes, but suffer from manually extracted features, reliance on costly hand-labeled data, and lack of validation on external datasets. To address these challenges, we develop a weakly-supervised machine learning framework for adverse event detection from unstructured clinical notes and evaluate it on insulin pump failure as a test case. Our model accurately detected cases of pump failure with 0.842 PR AUC on the holdout test set and 0.815 PR AUC when validated on an external dataset. Our approach allowed us to leverage a large dataset with far less hand-labeled data and can be easily transferred to additional adverse events for scalable post-market surveillance.

A Systematic Review of Methods for Medical Record Analysis to Detect Adverse Events in Hospitalized Patients

OBJECTIVE

In this systematic review, we evaluate 2 of the most used trigger tools according to the criteria of the World Health Organization for evaluating methods.

METHODS

We searched Embase, PubMed, and Cochrane databases for studies (2000-2017). Studies were included if medical record review (MRR) was performed with either the Global Trigger Tool or the Harvard Medical Practice Study in a hospital population. Quality assessment was performed in duplicate. Fifty studies were included, and results were reported for every criterion separately.

RESULTS

Medical record review reveals more adverse events (AEs) than any other method. However, at the same time, it detects different AEs. The costs of an AE were on average €4296. Considerable efforts have been made worldwide in health care to improve safety and to reduce errors. These have resulted in some positive effects. The literature showed that MRR is focused on several domains of quality of care and seems suitable for both small and large cohorts. Furthermore, we found a moderate to substantial agreement for the presence of a trigger and a moderate to good agreement for the presence of an AE.

CONCLUSIONS

Medical record review with a trigger tool is a reasonably well-researched method for the evaluation of the medical records for AEs. However, looking at the World Health Organization criteria, much research is still lacking or of moderate quality. Especially for the cost of detecting AEs, valuable information is missing. Moreover, knowledge of how MRR changes quality and safety of care should be evaluated.

Evolving Factors in Hospital Safety: A Systematic Review and Meta-Analysis of Hospital Adverse Events

OBJECTIVE

This study aimed to estimate the frequency of hospital adverse events (AEs) and explore the rate of AEs over time, and across and within hospital populations.

METHODS

Validated search terms were run in MEDLINE and EMBASE; gray literature and references of included studies were also searched. Studies of any design or language providing an estimate of AEs within the hospital were eligible. Studies were excluded if they only provided an estimate for a specific AE, a subgroup of hospital patients or children. Data were abstracted in duplicate using a standardized data abstraction form. Study quality was assessed using the Newcastle-Ottawa Scale. A random-effects meta-analysis estimated the occurrence of hospital AEs, and meta-regression explored the association between hospital AEs, and patient and hospital characteristics.

RESULTS

A total of 45,426 unique references were identified; 1,265 full-texts were reviewed and 94 studies representing 590 million admissions from 25 countries from 1961 to 2014 were included. The incidence of hospital AEs was 8.6 per 100 patient admissions (95% confidence interval [CI], 8.3 to 8.9; I2 = 100%, P < 0.001). Half of the AEs were preventable (52.6%), and a third resulted in moderate/significant harm (39.7%). The most evaluated AEs were surgical AEs, drug-related AEs, and nosocomial infections. The occurrence of AEs increased by year (95% CI, -0.05 to -0.04; P < 0.001) and patient age (95% CI = -0.15 to -0.14; P < 0.001), and varied by country income level and study characteristics. Patient sex, hospital type, hospital service, and geographical location were not associated with AEs.

CONCLUSIONS

Hospital AEs are common, and reported rates are increasing in the literature. Given the increase in AEs over time, hospitals should reinvest in improving hospital safety with a focus on interventions targeted toward the more than half of AEs that are preventable.

Patient Safety Incidents in Primary Care Dentistry in England and Wales: A Mixed-Methods Study

BACKGROUND

In recent decades, there has been considerable international attention aimed at improving the safety of hospital care, and more recently, this attention has broadened to include primary medical care. In contrast, the safety profile of primary care dentistry remains poorly characterized.

OBJECTIVES

We aimed to describe the types of primary care dental patient safety incidents reported within a national incident reporting database and understand their contributory factors and consequences.

METHODS

We undertook a cross-sectional mixed-methods study, which involved analysis of a weighted randomized sample of the most severe incident reports from primary care dentistry submitted to England and Wales' National Reporting and Learning System. Drawing on a conceptual literature-derived model of patient safety threats that we previously developed, we developed coding frameworks to describe and conduct thematic analysis of free text incident reports and determine the relationship between incident types, contributory factors, and outcomes.

RESULTS

Of 2000 reports sampled, 1456 were eligible for analysis. Sixty types of incidents were identified and organized across preoperative (40.3%, n = 587), intraoperative (56.1%, n = 817), and postoperative (3.6%, n = 52) stages. The main sources of unsafe care were delays in treatment (344/1456, 23.6%), procedural errors (excluding wrong-tooth extraction) (227/1456; 15.6%), medication-related adverse incidents (161/1456, 11.1%), equipment failure (90/1456, 6.2%) and x-ray related errors (87/1456, 6.0%). Of all incidents that resulted in a harmful outcome (n = 77, 5.3%), more than half were due to wrong tooth extractions (37/77, 48.1%) mainly resulting from distraction of the dentist. As a result of this type of incident, 34 of the 37 patients (91.9%) examined required further unnecessary procedures.

CONCLUSIONS

Flaws in administrative processes need improvement because they are the main cause for patients experiencing delays in receiving treatment. Checklists and standardization of clinical procedures have the potential to reduce procedural errors and avoid overuse of services. Wrong-tooth extractions should be addressed through focused research initiatives and encouraging policy development to mandate learning from serious dental errors like never events.

Developing an Analytical Pipeline to Classify Patient Safety Event Reports Using Optimized Predictive Algorithms

BACKGROUND

 Patient safety event reports provide valuable insight into systemic safety issues but deriving insights from these reports requires computational tools to efficiently parse through large volumes of qualitative data. Natural language processing (NLP) combined with predictive learning provides an automated approach to evaluating these data and supporting the work of patient safety analysts.

OBJECTIVES

 The objective of this study was to use NLP and machine learning techniques to develop a generalizable, scalable, and reliable approach to classifying event reports for the purpose of driving improvements in the safety and quality of patient care.

METHODS

 Datasets for 14 different labels (themes) were vectorized using a bag-of-words, tf-idf, or document embeddings approach and then applied to a series of classification algorithms via a hyperparameter grid search to derive an optimized model. Reports were also analyzed for terms strongly associated with each theme using an adjusted F-score calculation.

RESULTS

 F₁ score for each optimized model ranged from 0.951 ("Fall") to 0.544 ("Environment"). The bag-of-words approach proved optimal for 12 of 14 labels, and the naïve Bayes algorithm performed best for nine labels. Linear support vector machine was demonstrated as optimal for three labels and XGBoost for four of the 14 labels. Labels with more distinctly associated terms performed better than less distinct themes, as shown by a Pearson's correlation coefficient of 0.634.

CONCLUSIONS

 We were able to demonstrate an analytical pipeline that broadly applies NLP and predictive modeling to categorize patient safety reports from multiple facilities. This pipeline allows analysts to more rapidly identify and structure information contained in patient safety data, which can enhance the evaluation and the use of this information over time.

Evaluation of a Broad-Spectrum Partially Automated Adverse Event Surveillance System: A Potential Tool for Patient Safety Improvement in Hospitals With Limited Resources

OBJECTIVE

The aim of the study was to evaluate the sensitivity and resource efficiency of a partially automated adverse event (AE) surveillance system for routine patient safety efforts in hospitals with limited resources.

METHODS

Twenty-eight automated triggers from the hospital information system's clinical and administrative databases identified cases that were then filtered by exclusion criteria per trigger and then reviewed by an interdisciplinary team. The system, developed and implemented using in-house resources, was applied for 45 days of surveillance, for all hospital inpatient admissions (N = 1107). Each trigger was evaluated for its positive predictive value (PPV). Furthermore, the sensitivity of the surveillance system (overall and by AE category) was estimated relative to incidence ranges in the literature.

RESULTS

The surveillance system identified a total of 123 AEs among 283 reviewed medical records, yielding an overall PPV of 52%. The tool showed variable levels of sensitivity across and within AE categories when compared with the literature, with a relatively low overall sensitivity estimated between 21% and 44%. Adverse events were detected in 23 of the 36 AE categories defined by an established harm classification system. Furthermore, none of the detected AEs were voluntarily reported.

CONCLUSIONS

The surveillance system showed variable sensitivity levels across a broad range of AE categories with an acceptable PPV, overcoming certain limitations associated with other harm detection methods. The number of cases captured was substantial, and none had been previously detected or voluntarily reported. For hospitals with limited resources, this methodology provides valuable safety information from which interventions for quality improvement can be formulated.

Finding Potential Adverse Events in the Unstructured Text of Electronic Health Care Records: Development of the Shakespeare Method

BACKGROUND

Big data tools provide opportunities to monitor adverse events (patient harm associated with medical care) (AEs) in the unstructured text of electronic health care records (EHRs). Writers may explicitly state an apparent association between treatment and adverse outcome ("attributed") or state the simple treatment and outcome without an association ("unattributed"). Many methods for finding AEs in text rely on predefining possible AEs before searching for prespecified words and phrases or manual labeling (standardization) by investigators. We developed a method to identify possible AEs, even if unknown or unattributed, without any prespecifications or standardization of notes. Our method was inspired by word-frequency analysis methods used to uncover the true authorship of disputed works credited to William Shakespeare. We chose two use cases, "transfusion" and "time-based." Transfusion was chosen because new transfusion AE types were becoming recognized during the study data period; therefore, we anticipated an opportunity to find unattributed potential AEs (PAEs) in the notes. With the time-based case, we wanted to simulate near real-time surveillance. We chose time periods in the hope of detecting PAEs due to contaminated heparin from mid-2007 to mid-2008 that were announced in early 2008. We hypothesized that the prevalence of contaminated heparin may have been widespread enough to manifest in EHRs through symptoms related to heparin AEs, independent of clinicians' documentation of attributed AEs.

OBJECTIVE

We aimed to develop a new method to identify attributed and unattributed PAEs using the unstructured text of EHRs.

METHODS

We used EHRs for adult critical care admissions at a major teaching hospital (2001-2012). For each case, we formed a group of interest and a comparison group. We concatenated the text notes for each admission into one document sorted by date, and deleted replicate sentences and lists. We identified statistically significant words in the group of interest versus the comparison group. Documents in the group of interest were filtered to those words, followed by topic modeling on the filtered documents to produce topics. For each topic, the three documents with the maximum topic scores were manually reviewed to identify PAEs.

RESULTS

Topics centered around medical conditions that were unique to or more common in the group of interest, including PAEs. In each use case, most PAEs were unattributed in the notes. Among the transfusion PAEs was unattributed evidence of transfusion-associated cardiac overload and transfusion-related acute lung injury. Some of the PAEs from mid-2007 to mid-2008 were increased unattributed events consistent with AEs related to heparin contamination.

CONCLUSIONS

The Shakespeare method could be a useful supplement to AE reporting and surveillance of structured EHR data. Future improvements should include automation of the manual review process.

The incidence and nature of adverse events in rehabilitation inpatients with acquired brain injuries

INTRODUCTION

Patient safety is important in all healthcare settings. Few studies have examined the state of patient safety in rehabilitation and none have examined patient safety in the setting of acquired brain injury (ABI) rehabilitation.

OBJECTIVES

To determine the incidence, most common types, and severities of adverse events among inpatients undergoing ABI rehabilitation.

DESIGN

Retrospective case series descriptive study.

SETTING

The inpatient ABI rehabilitation program at an academic, tertiary rehabilitation hospital in Canada.

PARTICIPANTS

108 consecutive inpatients with acquired brain injuries.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURE(S)

Patient charts and incident reports from the hospital's voluntary reporting system were reviewed by three board-certified physiatrists to determine the incidence, type, severity and preventability of adverse events. Adverse events were identified and classified for severity and type using the WHO International Classification for Patient Safety. Preventability was rated on a 6-point Likert scale.

RESULTS

During the study period, the incidence of adverse events was 17.42 ± 3.86 per 1000 patient days. Adverse events affected 52.8% of patients. Most adverse events identified were mild in severity (98.51%) and the rest were of moderate severity. The two most common types of adverse events were 1) patient incidents (56.72%) such as falls, pressure ulcers and skin tears, and 2) patient behaviors such as missing patient, assault, or sexual behaviors (16.42%). Of the 80 adverse events identified in the study, 44.78% were preventable. The hospital's voluntary reporting system did not capture 57.9% of the adverse events identified.

CONCLUSIONS

Future efforts to improve patient safety in ABI rehab should focus on reducing falls, skin injuries and behaviors, and removing barriers to voluntary incident reporting. Detecting adverse events through chart reviews provide a more complete understanding of patient safety risks in ABI rehab than relying on incident reporting alone. This article is protected by copyright. All rights reserved.

Identification of Inpatient Falls Using Automated Review of Text-Based Medical Records

OBJECTIVES

Although falls are among the most common adverse event in hospitals, they are difficult to measure and often unreported. Mechanisms to track falls include incident reporting and medical records review. Because of limitations of each method, researchers suggest multimodal approaches. Although incident reporting is commonly used, medical records review is limited by the need to read a high volume of clinical notes. Natural language processing (NLP) is 1 potential mechanism to automate this process.

METHOD

We compared automated NLP to manual chart review and incident reporting as a method to detect falls among inpatients. First, we developed an NLP algorithm to identify inpatient progress notes describing falls. Second, we compared the NLP algorithm to manual records review in identifying inpatient progress notes that describe falls. Third, we compared the NLP algorithm to the incident reporting system in identifying falls.

RESULTS

When examining individual inpatient notes, our NLP algorithm was highly specific (0.97) but had low sensitivity (0.44) when compared with our manual records review. However, when considering groups of inpatient notes, all describing the same fall, our NLP algorithm had a large improvement in sensitivity (0.80) with some loss of specificity (0.65) compared with incident reporting.

CONCLUSIONS

National language processing represents a promising method to automate review of inpatient medical records to identify falls.

Case Report: Utilizing AI and NLP to Assist with Healthcare and Rehabilitation During the COVID-19 Pandemic

The COVID-19 pandemic has profoundly affected healthcare systems and healthcare delivery worldwide. Policy makers are utilizing social distancing and isolation policies to reduce the risk of transmission and spread of COVID-19, while the research, development, and testing of antiviral treatments and vaccines are ongoing. As part of these isolation policies, in-person healthcare delivery has been reduced, or eliminated, to avoid the risk of COVID-19 infection in high-risk and vulnerable populations, particularly those with comorbidities. Clinicians, occupational therapists, and physiotherapists have traditionally relied on in-person diagnosis and treatment of acute and chronic musculoskeletal (MSK) and neurological conditions and illnesses. The assessment and rehabilitation of persons with acute and chronic conditions has, therefore, been particularly impacted during the pandemic. This article presents a perspective on how Artificial Intelligence and Machine Learning (AI/ML) technologies, such as Natural Language Processing (NLP), can be used to assist with assessment and rehabilitation for acute and chronic conditions.

Technology Can Augment, but Not Replace, Critical Human Skills Needed for Patient Care

The practice of medicine is changing rapidly as a consequence of electronic health record adoption, new technologies for patient care, disruptive innovations that breakdown professional hierarchies, and evolving societal norms. Collectively, these have resulted in the modification of the physician's role as the gatekeeper for health care, increased shift-based care, and amplified interprofessional team-based care. Technological innovations present opportunities as well as challenges. Artificial intelligence, which has great potential, has already transformed some tasks, particularly those involving image interpretation. Ubiquitous access to information via the Internet by physicians and patients alike presents benefits as well as drawbacks: patients and providers have ready access to virtually all of human knowledge, but some websites are contaminated with misinformation and many people have difficulty differentiating between solid, evidence-based data and untruths. The role of the future physician will shift as complexity in health care increases and as artificial intelligence and other technologies advance. These technological advances demand new skills of physicians; memory and knowledge accumulation will diminish in importance while information management skills will become more important. In parallel, medical educators must enhance their teaching and assessment of critical human skills (e.g., clear communication, empathy) in the delivery of patient care. The authors emphasize the enduring role of critical human skills in safe and effective patient care even as medical practice is increasingly guided by artificial intelligence and related technology, and they suggest new and longitudinal ways of assessing essential noncognitive skills to meet the demands of the future. The authors envision practical and achievable benefits accruing to patients and providers if practitioners leverage technological advancements to facilitate the development of their critical human skills.

Application of Augmented Intelligence for Pharmacovigilance Case Seriousness Determination

INTRODUCTION

Identification of adverse events and determination of their seriousness ensures timely detection of potential patient safety concerns. Adverse event seriousness is a key factor in defining reporting timelines and is often performed manually by pharmacovigilance experts. The dramatic increase in the volume of safety reports necessitates exploration of scalable solutions that also meet reporting timeline requirements.

OBJECTIVE

The aim of this study was to develop an augmented intelligence methodology for automatically identifying adverse event seriousness in spontaneous, solicited, and medical literature safety reports. Deep learning models were evaluated for accuracy and/or the F1 score against a ground truth labeled by pharmacovigilance experts.

METHODS

Using a stratified random sample of safety reports received by Celgene, we developed three neural networks for addressing identification of adverse event seriousness: (1) a binary adverse-event level seriousness classifier; (2) a classifier for determining seriousness categorization at the adverse-event level; and (3) an annotator for identifying seriousness criteria terms to provide supporting evidence at the document level.

RESULTS

The seriousness classifier achieved an accuracy of 83.0% in post-marketing reports, 92.9% in solicited reports, and 86.3% in medical literature reports. F1 scores for seriousness categorization were 77.7 for death, 78.9 for hospitalization, and 75.5 for important medical events. The seriousness annotator achieved an F1 score of 89.9 in solicited reports, and 75.2 in medical literature reports.

CONCLUSIONS

The results of this study indicate that a neural network approach can provide an accurate and scalable solution for potentially augmenting pharmacovigilance practitioner determination of adverse event seriousness in spontaneous, solicited, and medical literature reports.

Automated Fall Detection Algorithm With Global Trigger Tool, Incident Reports, Manual Chart Review, and Patient-Reported Falls: Algorithm Development and Validation With a Retrospective Diagnostic Accuracy Study

BACKGROUND

Falls are common adverse events in hospitals, frequently leading to additional health costs due to prolonged stays and extra care. Therefore, reliable fall detection is vital to develop and test fall prevention strategies. However, conventional methods-voluntary incident reports and manual chart reviews-are error-prone and time consuming, respectively. Using a search algorithm to examine patients' electronic health record data and flag fall indicators offers an inexpensive, sensitive, cost-effective alternative.

OBJECTIVE

This study's purpose was to develop a fall detection algorithm for use with electronic health record data, then to evaluate it alongside the Global Trigger Tool, incident reports, a manual chart review, and patient-reported falls.

METHODS

Conducted on 2 campuses of a large hospital system in Switzerland, this retrospective diagnostic accuracy study consisted of 2 substudies: the first, targeting 240 patients, for algorithm development and the second, targeting 298 patients, for validation. In the development study, we compared the new algorithm's in-hospital fall rates with those indicated by the Global Trigger Tool and incident reports; in the validation study, we compared the algorithm's in-hospital fall rates with those from patient-reported falls and manual chart review. We compared the various methods by calculating sensitivity, specificity, and predictive values.

RESULTS

Twenty in-hospital falls were discovered in the development study sample. Of these, the algorithm detected 19 (sensitivity 95%), the Global Trigger Tool detected 18 (90%), and incident reports detected 14 (67%). Of the 15 falls found in the validation sample, the algorithm identified all 15 (100%), the manual chart review identified 14 (93%), and the patient-reported fall measure identified 5 (33%). Owing to relatively high numbers of false positives based on falls present on admission, the algorithm's positive predictive values were 50% (development sample) and 47% (validation sample). Instead of requiring 10 minutes per case for a full manual review or 20 minutes to apply the Global Trigger Tool, the algorithm requires only a few seconds, after which only the positive results (roughly 11% of the full case number) require review.

CONCLUSIONS

The newly developed electronic health record algorithm demonstrated very high sensitivity for fall detection. Applied in near real time, the algorithm can record in-hospital falls events effectively and help to develop and test fall prevention measures.

Operational measurement of diagnostic safety: state of the science

Reducing the incidence of diagnostic errors is increasingly a priority for government, professional, and philanthropic organizations. Several obstacles to measurement of diagnostic safety have hampered progress toward this goal. Although a coordinated national strategy to measure diagnostic safety remains an aspirational goal, recent research has yielded practical guidance for healthcare organizations to start using measurement to enhance diagnostic safety. This paper, concurrently published as an Issue Brief by the Agency for Healthcare Research and Quality, issues a "call to action" for healthcare organizations to begin measurement efforts using data sources currently available to them. Our aims are to outline the state of the science and provide practical recommendations for organizations to start identifying and learning from diagnostic errors. Whether by strategically leveraging current resources or building additional capacity for data gathering, nearly all organizations can begin their journeys to measure and reduce preventable diagnostic harm.

Measuring Exposure to Incarceration Using the Electronic Health Record

BACKGROUND

Electronic health records (EHRs) are a rich source of health information; however social determinants of health, including incarceration, and how they impact health and health care disparities can be hard to extract.

OBJECTIVE

The main objective of this study was to compare sensitivity and specificity of patient self-report with various methods of identifying incarceration exposure using the EHR.

RESEARCH DESIGN

Validation study using multiple data sources and types.

SUBJECTS

Participants of the Veterans Aging Cohort Study (VACS), a national observational cohort based on data from the Veterans Health Administration (VHA) EHR that includes all human immunodeficiency virus-infected patients in care (47,805) and uninfected patients (99,060) matched on region, age, race/ethnicity, and sex.

MEASURES AND DATA SOURCES

Self-reported incarceration history compared with: (1) linked VHA EHR data to administrative data from a state Department of Correction (DOC), (2) linked VHA EHR data to administrative data on incarceration from Centers for Medicare and Medicaid Services (CMS), (3) VHA EHR-specific identifier codes indicative of receipt of VHA incarceration reentry services, and (4) natural language processing (NLP) in unstructured text in VHA EHR.

RESULTS

Linking the EHR to DOC data: sensitivity 2.5%, specificity 100%; linking the EHR to CMS data: sensitivity 7.9%, specificity 99.3%; VHA EHR-specific identifier for receipt of reentry services: sensitivity 7.3%, specificity 98.9%; and NLP, sensitivity 63.5%, specificity 95.9%.

CONCLUSIONS

NLP tools hold promise as a feasible and valid method to identify individuals with exposure to incarceration in EHR. Future work should expand this approach using a larger body of documents and refinement of the methods, which may further improve operating characteristics of this method.

Finding warning markers: Leveraging natural language processing and machine learning technologies to detect risk of school violence

INTRODUCTION

School violence has a far-reaching effect, impacting the entire school population including staff, students and their families. Among youth attending the most violent schools, studies have reported higher dropout rates, poor school attendance, and poor scholastic achievement. It was noted that the largest crime-prevention results occurred when youth at elevated risk were given an individualized prevention program. However, much work is needed to establish an effective approach to identify at-risk subjects.

OBJECTIVE

In our earlier research, we developed a risk assessment program to interview subjects, identify risk and protective factors, and evaluate risk for school violence. This study focused on developing natural language processing (NLP) and machine learning technologies to automate the risk assessment process.

MATERIAL AND METHODS

We prospectively recruited 131 students with or without behavioral concerns from 89 schools between 05/01/2015 and 04/30/2018. The subjects were interviewed with two risk assessment scales and a questionnaire, and their risk of violence were determined by pediatric psychiatrists based on clinical judgment. Using NLP technologies, different types of linguistic features were extracted from the interview content. Machine learning classifiers were then applied to predict risk of school violence for individual subjects. A two-stage feature selection was implemented to identify violence-related predictors. The performance was validated on the psychiatrist-generated reference standard of risk levels, where positive predictive value (PPV), sensitivity (SEN), negative predictive value (NPV), specificity (SPEC) and area under the ROC curve (AUC) were assessed.

RESULTS

Compared to subjects' sociodemographic information, use of linguistic features significantly improved classifiers' predictive performance (P < 0.01). The best-performing classifier with n-gram features achieved 86.5 %/86.5 %/85.7 %/85.7 %/94.0 % (PPV/SEN/NPV/SPEC/AUC) on the cross-validation set and 83.3 %/93.8 %/91.7 %/78.6 %/94.6 % (PPV/SEN/NPV/SPEC/AUC) on the test data. The feature selection process identified a set of predictors covering the discussion of subjects' thoughts, perspectives, behaviors, individual characteristics, peers and family dynamics, and protective factors.

CONCLUSIONS

By analyzing the content from subject interviews, the NLP and machine learning algorithms showed good capacity for detecting risk of school violence. The feature selection uncovered multiple warning markers that could deliver useful clinical insights to assist personalizing intervention. Consequently, the developed approach offered the promise of an accurate and scalable computerized screening service for preventing school violence.

Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?

There is little known about how academic medical centers (AMCs) in the US develop, implement, and maintain predictive modeling and machine learning (PM and ML) models. We conducted semi-structured interviews with leaders from AMCs to assess their use of PM and ML in clinical care, understand associated challenges, and determine recommended best practices. Each transcribed interview was iteratively coded and reconciled by a minimum of 2 investigators to identify key barriers to and facilitators of PM and ML adoption and implementation in clinical care. Interviews were conducted with 33 individuals from 19 AMCs nationally. AMCs varied greatly in the use of PM and ML within clinical care, from some just beginning to explore their utility to others with multiple models integrated into clinical care. Informants identified 5 key barriers to the adoption and implementation of PM and ML in clinical care: (1) culture and personnel, (2) clinical utility of the PM and ML tool, (3) financing, (4) technology, and (5) data. Recommendation to the informatics community to overcome these barriers included: (1) development of robust evaluation methodologies, (2) partnership with vendors, and (3) development and dissemination of best practices. For institutions developing clinical PM and ML applications, they are advised to: (1) develop appropriate governance, (2) strengthen data access, integrity, and provenance, and (3) adhere to the 5 rights of clinical decision support. This article highlights key challenges of implementing PM and ML in clinical care at AMCs and suggests best practices for development, implementation, and maintenance at these institutions.

Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction

Background

The rapid adoption of electronic health records (EHRs) holds great promise for advancing medicine through practice-based knowledge discovery. However, the validity of EHR-based clinical research is questionable due to poor research reproducibility caused by the heterogeneity and complexity of healthcare institutions and EHR systems, the cross-disciplinary nature of the research team, and the lack of standard processes and best practices for conducting EHR-based clinical research.

Method

We developed a data abstraction framework to standardize the process for multi-site EHR-based clinical studies aiming to enhance research reproducibility. The framework was implemented for a multi-site EHR-based research project, the ESPRESSO project, with the goal to identify individuals with silent brain infarctions (SBI) at Tufts Medical Center (TMC) and Mayo Clinic. The heterogeneity of healthcare institutions, EHR systems, documentation, and process variation in case identification was assessed quantitatively and qualitatively.

Result

We discovered a significant variation in the patient populations, neuroimaging reporting, EHR systems, and abstraction processes across the two sites. The prevalence of SBI for patients over age 50 for TMC and Mayo is 7.4 and 12.5% respectively. There is a variation regarding neuroimaging reporting where TMC are lengthy, standardized and descriptive while Mayo’s reports are short and definitive with more textual variations. Furthermore, differences in the EHR system, technology infrastructure, and data collection process were identified.

Conclusion

The implementation of the framework identified the institutional and process variations and the heterogeneity of EHRs across the sites participating in the case study. The experiment demonstrates the necessity to have a standardized process for data abstraction when conducting EHR-based clinical studies.

Automatically Identifying Financial Stress Information from Clinical Notes for Patients with Prostate Cancer

Background

Financial stress, one of the social determinants, is common among cancer patients because of high out-ofpocket costs for treatment, as well as indirect costs. The National Academy of Medicine (NAM) has advised providers to recognize and discuss cost concerns with patients in order to enhance shared decision-making for treatment and exploration of financial assistant programs. However, financial stress is rarely assessed in clinical practice or research, thus, under-coded and under-documented in clinical practice. Natural language processing (NLP) offers great potential that can automatically extract and process data on financial stress from clinical free text existing in the patient electronic health record (EHR).

Methods

We developed and evaluated an NLP approach to identify financial stress from clinical narratives for patients with prostate cancer. Of 4,195 eligible prostate cancer patients, we randomly sampled 3,138 patients (75%) as a training dataset (150,990 documents) to develop a financial stress lexicon and NLP algorithms iteratively. The remaining 1,057 patients (25%) were used as a test dataset (55,516 documents) to evaluate the NLP algorithm performance. The common terms representing financial stress were "financial concerns," "unable to afford," "insurance issue," "unemployed," and "financial assistance." Negations were used to exclude false mentions of financial stress.

Results

Applying both pre- and post-negation, the NLP algorithm identified 209 patients (6.0%) from the training sample and 66 patients (6.2%) with 161 notes from the test sample as having documented financial stress. Two independent domain experts manually reviewed all 161 notes with NLP identified positives and randomly selected 161 notes with NLP-identified negatives, the NLP algorithm yielded 0.86 for precision, 1 for recall, and 0.9.2 for F-score.

Conclusions

Financial stress information is not commonly documented in the EHR, neither in structured format nor in clinical narratives. However, natural language processing can accurately extract financial stress information from clinical notes when such narrative information is available.