1
|
Mertes PM, Morgand C, Barach P, Jurkolow G, Assmann KE, Dufetelle E, Susplugas V, Alauddin B, Yavordios PG, Tourres J, Dumeix JM, Capdevila X. Validation of a natural language processing algorithm using national reporting data to improve identification of anesthesia-related ADVerse evENTs: The "ADVENTURE" study. Anaesth Crit Care Pain Med 2024; 43:101390. [PMID: 38718923 DOI: 10.1016/j.accpm.2024.101390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2023] [Revised: 04/02/2024] [Accepted: 04/22/2024] [Indexed: 06/13/2024]
Abstract
BACKGROUND Reporting and analysis of adverse events (AE) is associated with improved health system learning, quality outcomes, and patient safety. Manual text analysis is time-consuming, costly, and prone to human errors. We aimed to demonstrate the feasibility of novel machine learning and natural language processing (NLP) approaches for early predictions of adverse events and provide input to direct quality improvement and patient safety initiatives. METHODS We used machine learning to analyze 9559 continuously reported AE by clinicians and healthcare systems to the French National Health accreditor (HAS) between January 1, 2009, and December 31, 2020 . We validated the labeling of 135,000 unique de-identified AE reports and determined the associations between different system's root causes and patient consequences. The model was validated by independent expert anesthesiologists. RESULTS The machine learning (ML) and Artificial Intelligence (AI) model trained on 9559 AE datasets accurately categorized 8800 (88%) of reported AE. The three most frequent AE types were "difficult orotracheal intubation" (16.9% of AE reports), "medication error" (10.5%), and "post-induction hypotension" (6.9%). The accuracy of the AI model reached 70.9% sensitivity, 96.6% specificity for "difficult intubation", 43.2% sensitivity, and 98.9% specificity for "medication error." CONCLUSIONS This unsupervised ML method provides an accurate, automated, AI-supported search algorithm that ranks and helps to understand complex risk patterns and has greater speed, precision, and clarity when compared to manual human data extraction. Machine learning and Natural language processing (NLP) models can effectively be used to process natural language AE reports and augment expert clinician input. This model can support clinical applications and methodological standards and used to better inform and enhance decision-making for improved risk management and patient safety. TRIAL REGISTRATION The study was approved by the ethics committee of the French Society of Anesthesiology (IRB 00010254-2020-20) and the CNIL (CNIL: 118 58 95) and the study was registered with ClinicalTrials.gov (NCT: NCT05185479).
Collapse
Affiliation(s)
- Paul M Mertes
- Department of Anesthesia and Intensive Care, Hôpitaux Universitaires de Strasbourg, Nouvel Hôpital Civil, EA 3072, FMTS de Strasbourg, Strasbourg, France; CFAR - Collège Français des Anesthésistes-Réanimateurs, 75016 Paris, France
| | - Claire Morgand
- Evaluation Department and Tools for Quality and Safety of Care, French national authority for health (Haute Autorité de Santé - EvOQSS), Saint Denis, France
| | - Paul Barach
- Thomas Jefferson School of Medicine, Philadelphia, USA; Sigmund Freud University, Vienna, Austria
| | - Geoffrey Jurkolow
- CFAR - Collège Français des Anesthésistes-Réanimateurs, 75016 Paris, France.
| | - Karen E Assmann
- Evaluation Department and Tools for Quality and Safety of Care, French national authority for health (Haute Autorité de Santé - EvOQSS), Saint Denis, France
| | | | | | - Bilal Alauddin
- Collective Thinking, 23 rue Yves Toudic, 75010 Paris, France
| | | | - Jean Tourres
- CFAR - Collège Français des Anesthésistes-Réanimateurs, 75016 Paris, France
| | - Jean-Marc Dumeix
- CFAR - Collège Français des Anesthésistes-Réanimateurs, 75016 Paris, France
| | - Xavier Capdevila
- Department of Anesthesiology and Critical Care Medicine, Lapeyronie University Hospital, 34295 Montpellier Cedex 5, France; Inserm Unit 1298 Montpellier NeuroSciences Institute, Montpellier University, 34295 Montpellier Cedex 5, France
| |
Collapse
|
2
|
Sarwal D, Wang L, Gandhi S, Sagheb Hossein Pour E, Janssens LP, Delgado AM, Doering KA, Mishra AK, Greenwood JD, Liu H, Majumder S. Identification of pancreatic cancer risk factors from clinical notes using natural language processing. Pancreatology 2024; 24:572-578. [PMID: 38693040 DOI: 10.1016/j.pan.2024.03.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/04/2024] [Revised: 03/20/2024] [Accepted: 03/23/2024] [Indexed: 05/03/2024]
Abstract
OBJECTIVES Screening for pancreatic ductal adenocarcinoma (PDAC) is considered in high-risk individuals (HRIs) with established PDAC risk factors, such as family history and germline mutations in PDAC susceptibility genes. Accurate assessment of risk factor status is provider knowledge-dependent and requires extensive manual chart review by experts. Natural Language Processing (NLP) has shown promise in automated data extraction from the electronic health record (EHR). We aimed to use NLP for automated extraction of PDAC risk factors from unstructured clinical notes in the EHR. METHODS We first developed rule-based NLP algorithms to extract PDAC risk factors at the document-level, using an annotated corpus of 2091 clinical notes. Next, we further improved the NLP algorithms using a cohort of 1138 patients through patient-level training, validation, and testing, with comparison against a pre-specified reference standard. To minimize false-negative results we prioritized algorithm recall. RESULTS In the test set (n = 807), the NLP algorithms achieved a recall of 0.933, precision of 0.790, and F1-score of 0.856 for family history of PDAC. For germline genetic mutations, the algorithm had a high recall of 0.851, while precision and F1-score were lower at 0.350 and 0.496 respectively. Most false positives for germline mutations resulted from erroneous recognition of tissue mutations. CONCLUSIONS Rule-based NLP algorithms applied to unstructured clinical notes are highly sensitive for automated identification of PDAC risk factors. Further validation in a large primary-care patient population is warranted to assess real-world utility in identifying HRIs for pancreatic cancer screening.
Collapse
Affiliation(s)
- Dhruv Sarwal
- Department of Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN, USA
| | - Liwei Wang
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | - Sonal Gandhi
- Department of Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN, USA
| | | | - Laurens P Janssens
- Department of Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN, USA
| | - Adriana M Delgado
- Department of Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN, USA
| | - Karen A Doering
- Department of Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN, USA
| | - Anup Kumar Mishra
- Department of Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN, USA
| | - Jason D Greenwood
- Department of Family Medicine, Mayo Clinic, Rochester, MN, USA; Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Hongfang Liu
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | - Shounak Majumder
- Department of Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN, USA.
| |
Collapse
|
3
|
Wi S, Goldhoff PE, Fuller LA, Grewal K, Wentzensen N, Clarke MA, Lorey TS. Using Natural Language Processing to Improve Discrete Data Capture From Interpretive Cervical Biopsy Diagnoses at a Large Health Care Organization. Arch Pathol Lab Med 2023; 147:222-226. [PMID: 35390126 DOI: 10.5858/arpa.2021-0410-oa] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/15/2021] [Indexed: 02/05/2023]
Abstract
CONTEXT.— The terminology used by pathologists to describe and grade dysplasia and premalignant changes of the cervical epithelium has evolved over time. Unfortunately, coexistence of different classification systems combined with nonstandardized interpretive text has created multiple layers of interpretive ambiguity. OBJECTIVE.— To use natural language processing (NLP) to automate and expedite translation of interpretive text to a single most severe, and thus actionable, cervical intraepithelial neoplasia (CIN) diagnosis. DESIGN.— We developed and applied NLP algorithms to 35 847 unstructured cervical pathology reports and assessed NLP performance in identifying the most severe diagnosis, compared to expert manual review. NLP performance was determined by calculating precision, recall, and F score. RESULTS.— The NLP algorithms yielded a precision of 0.957, a recall of 0.925, and an F score of 0.94. Additionally, we estimated that the time to evaluate each monthly biopsy file was significantly reduced, from 30 hours to 0.5 hours. CONCLUSIONS.— A set of validated NLP algorithms applied to pathology reports can rapidly and efficiently assign a discrete, actionable diagnosis using CIN classification to assist with clinical management of cervical pathology and disease. Moreover, discrete diagnostic data encoded as CIN terminology can enhance the efficiency of clinical research.
Collapse
Affiliation(s)
- Soora Wi
- From Kaiser Permanente, TPMG Regional Laboratories, Berkeley, California (Wi, Goldhoff, Fuller, Grewal, Lorey)
| | - Patricia E Goldhoff
- From Kaiser Permanente, TPMG Regional Laboratories, Berkeley, California (Wi, Goldhoff, Fuller, Grewal, Lorey)
| | - Laurie A Fuller
- From Kaiser Permanente, TPMG Regional Laboratories, Berkeley, California (Wi, Goldhoff, Fuller, Grewal, Lorey)
| | - Kiranjit Grewal
- From Kaiser Permanente, TPMG Regional Laboratories, Berkeley, California (Wi, Goldhoff, Fuller, Grewal, Lorey)
| | - Nicolas Wentzensen
- From the Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, Maryland (Wentzensen, Clarke)
| | - Megan A Clarke
- From the Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, Maryland (Wentzensen, Clarke)
| | - Thomas S Lorey
- From Kaiser Permanente, TPMG Regional Laboratories, Berkeley, California (Wi, Goldhoff, Fuller, Grewal, Lorey)
| |
Collapse
|
4
|
Saha A, Burns L, Kulkarni AM. A scoping review of natural language processing of radiology reports in breast cancer. Front Oncol 2023; 13:1160167. [PMID: 37124523 PMCID: PMC10130381 DOI: 10.3389/fonc.2023.1160167] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Accepted: 03/28/2023] [Indexed: 05/02/2023] Open
Abstract
Various natural language processing (NLP) algorithms have been applied in the literature to analyze radiology reports pertaining to the diagnosis and subsequent care of cancer patients. Applications of this technology include cohort selection for clinical trials, population of large-scale data registries, and quality improvement in radiology workflows including mammography screening. This scoping review is the first to examine such applications in the specific context of breast cancer. Out of 210 identified articles initially, 44 met our inclusion criteria for this review. Extracted data elements included both clinical and technical details of studies that developed or evaluated NLP algorithms applied to free-text radiology reports of breast cancer. Our review illustrates an emphasis on applications in diagnostic and screening processes over treatment or therapeutic applications and describes growth in deep learning and transfer learning approaches in recent years, although rule-based approaches continue to be useful. Furthermore, we observe increased efforts in code and software sharing but not with data sharing.
Collapse
Affiliation(s)
- Ashirbani Saha
- Department of Oncology, McMaster University, Hamilton, ON, Canada
- Hamilton Health Sciences and McMaster University, Escarpment Cancer Research Institute, Hamilton, ON, Canada
- *Correspondence: Ashirbani Saha,
| | - Levi Burns
- Michael G. DeGroote School of Medicine, McMaster University, Hamilton, ON, Canada
| | | |
Collapse
|
5
|
Wang L, Fu S, Wen A, Ruan X, He H, Liu S, Moon S, Mai M, Riaz IB, Wang N, Yang P, Xu H, Warner JL, Liu H. Assessment of Electronic Health Record for Cancer Research and Patient Care Through a Scoping Review of Cancer Natural Language Processing. JCO Clin Cancer Inform 2022; 6:e2200006. [PMID: 35917480 PMCID: PMC9470142 DOI: 10.1200/cci.22.00006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2022] [Revised: 03/18/2022] [Accepted: 06/15/2022] [Indexed: 11/20/2022] Open
Abstract
PURPOSE The advancement of natural language processing (NLP) has promoted the use of detailed textual data in electronic health records (EHRs) to support cancer research and to facilitate patient care. In this review, we aim to assess EHR for cancer research and patient care by using the Minimal Common Oncology Data Elements (mCODE), which is a community-driven effort to define a minimal set of data elements for cancer research and practice. Specifically, we aim to assess the alignment of NLP-extracted data elements with mCODE and review existing NLP methodologies for extracting said data elements. METHODS Published literature studies were searched to retrieve cancer-related NLP articles that were written in English and published between January 2010 and September 2020 from main literature databases. After the retrieval, articles with EHRs as the data source were manually identified. A charting form was developed for relevant study analysis and used to categorize data including four main topics: metadata, EHR data and targeted cancer types, NLP methodology, and oncology data elements and standards. RESULTS A total of 123 publications were selected finally and included in our analysis. We found that cancer research and patient care require some data elements beyond mCODE as expected. Transparency and reproductivity are not sufficient in NLP methods, and inconsistency in NLP evaluation exists. CONCLUSION We conducted a comprehensive review of cancer NLP for research and patient care using EHRs data. Issues and barriers for wide adoption of cancer NLP were identified and discussed.
Collapse
Affiliation(s)
- Liwei Wang
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Sunyang Fu
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Andrew Wen
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Xiaoyang Ruan
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Huan He
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Sijia Liu
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Sungrim Moon
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Michelle Mai
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| | - Irbaz B. Riaz
- Department of Hematology/Oncology, Mayo Clinic, Scottsdale, AZ
| | - Nan Wang
- Department of Computer Science and Engineering, College of Science and Engineering, University of Minnesota, Minneapolis, MN
| | - Ping Yang
- Department of Quantitative Health Sciences, Mayo Clinic, Scottsdale, AZ
| | - Hua Xu
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX
| | - Jeremy L. Warner
- Departments of Medicine (Hematology/Oncology), Vanderbilt University, Nashville, TN
- Department Biomedical Informatics, Vanderbilt University, Nashville, TN
| | - Hongfang Liu
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN
| |
Collapse
|
6
|
Diamond CJ, Laurentiev J, Yang J, Wint A, Harris KA, Dang TH, Mecker A, Carpenter EB, Tosteson AN, Wright A, Haas JS, Atlas SJ, Zhou L. Natural Language Processing to Identify Abnormal Breast, Lung, and Cervical Cancer Screening Test Results from Unstructured Reports to Support Timely Follow-up. Stud Health Technol Inform 2022; 290:433-437. [PMID: 35673051 PMCID: PMC10792587 DOI: 10.3233/shti220112] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Cancer screening and timely follow-up of abnormal results can reduce mortality. One barrier to follow-up is the failure to identify abnormal results. While EHRs have coded results for certain tests, cancer screening results are often stored in free-text reports, which limit capabilities for automated decision support. As part of the multilevel Follow-up of Cancer Screening (mFOCUS) trial, we developed and implemented a natural language processing (NLP) tool to assist with real-time detection of abnormal cancer screening test results (including mammograms, low-dose chest CT scans, and Pap smears) and identification of gynecological follow-up for higher risk abnormalities (i.e. colposcopy) from free-text reports. We demonstrate the integration and implementation of NLP, within the mFOCUS system, to improve the follow-up of abnormal cancer screening results in a large integrated healthcare system. The NLP pipelines have detected scenarios when guideline-recommended care was not delivered, in part because the provider mis-identified the text-based result reports.
Collapse
Affiliation(s)
- Courtney J. Diamond
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - John Laurentiev
- Department of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, United States
| | - Jie Yang
- Department of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, United States
| | - Amy Wint
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Kimberly A. Harris
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Tin H. Dang
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Amrita Mecker
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Emily B. Carpenter
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Anna N. Tosteson
- The Dartmouth Institute for Health Policy and Clinical Practice, Dartmouth College, Lebanon, NH, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Jennifer S. Haas
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Steven J. Atlas
- Department of General Internal Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Li Zhou
- Department of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, United States
| |
Collapse
|
7
|
The Role of Artificial Intelligence in Early Cancer Diagnosis. Cancers (Basel) 2022; 14:cancers14061524. [PMID: 35326674 PMCID: PMC8946688 DOI: 10.3390/cancers14061524] [Citation(s) in RCA: 54] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2022] [Revised: 03/08/2022] [Accepted: 03/10/2022] [Indexed: 02/01/2023] Open
Abstract
Improving the proportion of patients diagnosed with early-stage cancer is a key priority of the World Health Organisation. In many tumour groups, screening programmes have led to improvements in survival, but patient selection and risk stratification are key challenges. In addition, there are concerns about limited diagnostic workforces, particularly in light of the COVID-19 pandemic, placing a strain on pathology and radiology services. In this review, we discuss how artificial intelligence algorithms could assist clinicians in (1) screening asymptomatic patients at risk of cancer, (2) investigating and triaging symptomatic patients, and (3) more effectively diagnosing cancer recurrence. We provide an overview of the main artificial intelligence approaches, including historical models such as logistic regression, as well as deep learning and neural networks, and highlight their early diagnosis applications. Many data types are suitable for computational analysis, including electronic healthcare records, diagnostic images, pathology slides and peripheral blood, and we provide examples of how these data can be utilised to diagnose cancer. We also discuss the potential clinical implications for artificial intelligence algorithms, including an overview of models currently used in clinical practice. Finally, we discuss the potential limitations and pitfalls, including ethical concerns, resource demands, data security and reporting standards.
Collapse
|
8
|
Asan O, Choudhury A. Research Trends in Artificial Intelligence Applications in Human Factors Health Care: Mapping Review. JMIR Hum Factors 2021; 8:e28236. [PMID: 34142968 PMCID: PMC8277302 DOI: 10.2196/28236] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Revised: 04/14/2021] [Accepted: 05/03/2021] [Indexed: 01/17/2023] Open
Abstract
BACKGROUND Despite advancements in artificial intelligence (AI) to develop prediction and classification models, little research has been devoted to real-world translations with a user-centered design approach. AI development studies in the health care context have often ignored two critical factors of ecological validity and human cognition, creating challenges at the interface with clinicians and the clinical environment. OBJECTIVE The aim of this literature review was to investigate the contributions made by major human factors communities in health care AI applications. This review also discusses emerging research gaps, and provides future research directions to facilitate a safer and user-centered integration of AI into the clinical workflow. METHODS We performed an extensive mapping review to capture all relevant articles published within the last 10 years in the major human factors journals and conference proceedings listed in the "Human Factors and Ergonomics" category of the Scopus Master List. In each published volume, we searched for studies reporting qualitative or quantitative findings in the context of AI in health care. Studies are discussed based on the key principles such as evaluating workload, usability, trust in technology, perception, and user-centered design. RESULTS Forty-eight articles were included in the final review. Most of the studies emphasized user perception, the usability of AI-based devices or technologies, cognitive workload, and user's trust in AI. The review revealed a nascent but growing body of literature focusing on augmenting health care AI; however, little effort has been made to ensure ecological validity with user-centered design approaches. Moreover, few studies (n=5 against clinical/baseline standards, n=5 against clinicians) compared their AI models against a standard measure. CONCLUSIONS Human factors researchers should actively be part of efforts in AI design and implementation, as well as dynamic assessments of AI systems' effects on interaction, workflow, and patient outcomes. An AI system is part of a greater sociotechnical system. Investigators with human factors and ergonomics expertise are essential when defining the dynamic interaction of AI within each element, process, and result of the work system.
Collapse
Affiliation(s)
- Onur Asan
- School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, United States
| | - Avishek Choudhury
- School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, United States
| |
Collapse
|
9
|
Young IJB, Luz S, Lone N. A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis. Int J Med Inform 2019; 132:103971. [PMID: 31630063 DOI: 10.1016/j.ijmedinf.2019.103971] [Citation(s) in RCA: 55] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2019] [Revised: 08/06/2019] [Accepted: 09/14/2019] [Indexed: 12/26/2022]
Abstract
CONTEXT Adverse events in healthcare are often collated in incident reports which contain unstructured free text. Learning from these events may improve patient safety. Natural language processing (NLP) uses computational techniques to interrogate free text, reducing the human workload associated with its analysis. There is growing interest in applying NLP to patient safety, but the evidence in the field has not been summarised and evaluated to date. OBJECTIVE To perform a systematic literature review and narrative synthesis to describe and evaluate NLP methods for classification of incident reports and adverse events in healthcare. METHODS Data sources included Medline, Embase, The Cochrane Library, CINAHL, MIDIRS, ISI Web of Science, SciELO, Google Scholar, PROSPERO, hand searching of key articles, and OpenGrey. Data items were manually abstracted to a standardised extraction form. RESULTS From 428 articles screened for eligibility, 35 met the inclusion criteria of using NLP to perform a classification task on incident reports, or with the aim of detecting adverse events. The majority of studies used free text from incident reporting systems or electronic health records. Models were typically designed to classify by type of incident, type of medication error, or harm severity. A broad range of NLP techniques are demonstrated to perform these classification tasks with favourable performance outcomes. There are methodological challenges in how these results can be interpreted in a broader context. CONCLUSION NLP can generate meaningful information from unstructured data in the specific domain of the classification of incident reports and adverse events. Understanding what or why incidents are occurring is important in adverse event analysis. If NLP enables these insights to be drawn from larger datasets it may improve the learning from adverse events in healthcare.
Collapse
Affiliation(s)
- Ian James Bruce Young
- Department of Anaesthesia, Critical Care and Pain Medicine, Edinburgh Royal Infirmary, 51 Little France Crescent, Edinburgh, Scotland, EH16 4SA, United Kingdom.
| | - Saturnino Luz
- Usher Institute of Population Health Sciences & Informatics, The University of Edinburgh, 9 Little France Rd, Edinburgh, Scotland EH16 4UX, United Kingdom.
| | - Nazir Lone
- Usher Institute of Population Health Sciences and Informatics, The University of Edinburgh, Teviot Place, Edinburgh, EH8 9AG, United Kingdom.
| |
Collapse
|
10
|
A frame semantic overview of NLP-based information extraction for cancer-related EHR notes. J Biomed Inform 2019; 100:103301. [PMID: 31589927 DOI: 10.1016/j.jbi.2019.103301] [Citation(s) in RCA: 35] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2019] [Revised: 09/04/2019] [Accepted: 10/03/2019] [Indexed: 02/07/2023]
Abstract
OBJECTIVE There is a lot of information about cancer in Electronic Health Record (EHR) notes that can be useful for biomedical research provided natural language processing (NLP) methods are available to extract and structure this information. In this paper, we present a scoping review of existing clinical NLP literature for cancer. METHODS We identified studies describing an NLP method to extract specific cancer-related information from EHR sources from PubMed, Google Scholar, ACL Anthology, and existing reviews. Two exclusion criteria were used in this study. We excluded articles where the extraction techniques used were too broad to be represented as frames (e.g., document classification) and also where very low-level extraction methods were used (e.g. simply identifying clinical concepts). 78 articles were included in the final review. We organized this information according to frame semantic principles to help identify common areas of overlap and potential gaps. RESULTS Frames were created from the reviewed articles pertaining to cancer information such as cancer diagnosis, tumor description, cancer procedure, breast cancer diagnosis, prostate cancer diagnosis and pain in prostate cancer patients. These frames included both a definition as well as specific frame elements (i.e. extractable attributes). We found that cancer diagnosis was the most common frame among the reviewed papers (36 out of 78), with recent work focusing on extracting information related to treatment and breast cancer diagnosis. CONCLUSION The list of common frames described in this paper identifies important cancer-related information extracted by existing NLP techniques and serves as a useful resource for future researchers requiring cancer information extracted from EHR notes. We also argue, due to the heavy duplication of cancer NLP systems, that a general purpose resource of annotated cancer frames and corresponding NLP tools would be valuable.
Collapse
|
11
|
Roman-Belmonte JM, De la Corte-Rodriguez H, Rodriguez-Merchan EC. Comparative analysis of two methods of data entry into electronic medical records: A randomized clinical trial (research letter). J Eval Clin Pract 2017; 23:1478-1481. [PMID: 28948670 DOI: 10.1111/jep.12835] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/25/2017] [Accepted: 08/28/2017] [Indexed: 11/29/2022]
Affiliation(s)
- Juan M Roman-Belmonte
- Physical Medicine and Rehabilitation Physician, Department of Physical Medicine and Rehabilitation, "Cruz Roja San José y Santa Adela" University Hospital, Madrid, Spain
| | - Hortensia De la Corte-Rodriguez
- Physical Medicine and Rehabilitation Physician, Department of Physical Medicine and Rehabilitation, "La Paz" University Hospital-IdiPaz, Madrid, Spain
| | - E Carlos Rodriguez-Merchan
- Orthopedic Surgeon, Department of Orthopedic Surgery, "La Paz" University Hospital-IdiPaz, Madrid, Spain
| |
Collapse
|
12
|
Ferraro JP, Ye Y, Gesteland PH, Haug PJ, Tsui FR, Cooper GF, Van Bree R, Ginter T, Nowalk AJ, Wagner M. The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance. Appl Clin Inform 2017; 8:560-580. [PMID: 28561130 PMCID: PMC6241736 DOI: 10.4338/aci-2016-12-ra-0211] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2016] [Accepted: 03/11/2017] [Indexed: 11/23/2022] Open
Abstract
OBJECTIVES This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance. METHODS We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) 'other' diagnosis. RESULTS On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser). CONCLUSION In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.
Collapse
Affiliation(s)
- Jeffrey P Ferraro
- Jeffrey P. Ferraro, Homer Warner Center | Intermountain Healthcare, 5171 South Cottonwood St, Suite 220, Murray, Utah 84107, , Tel: 801-244-6570
| | | | | | | | | | | | | | | | | | | |
Collapse
|
13
|
Gao H, Aiello Bowles EJ, Carrell D, Buist DSM. Using natural language processing to extract mammographic findings. J Biomed Inform 2015; 54:77-84. [PMID: 25661260 DOI: 10.1016/j.jbi.2015.01.010] [Citation(s) in RCA: 22] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2014] [Revised: 01/21/2015] [Accepted: 01/25/2015] [Indexed: 11/28/2022]
Abstract
OBJECTIVE Structured data on mammographic findings are difficult to obtain without manual review. We developed and evaluated a rule-based natural language processing (NLP) system to extract mammographic findings from free-text mammography reports. MATERIALS AND METHODS The NLP system extracted four mammographic findings: mass, calcification, asymmetry, and architectural distortion, using a dictionary look-up method on 93,705 mammography reports from Group Health. Status annotations and anatomical location annotation were associated to each NLP detected finding through association rules. After excluding negated, uncertain, and historical findings, affirmative mentions of detected findings were summarized. Confidence flags were developed to denote reports with highly confident NLP results and reports with possible NLP errors. A random sample of 100 reports was manually abstracted to evaluate the accuracy of the system. RESULTS The NLP system correctly coded 96-99 out of our sample of 100 reports depending on findings. Measures of sensitivity, specificity and negative predictive values exceeded 0.92 for all findings. Positive predictive values were relatively low for some findings due to their low prevalence. DISCUSSION Our NLP system was implemented entirely in SAS Base, which makes it portable and easy to implement. It performed reasonably well with multiple applications, such as using confidence flags as a filter to improve the efficiency of manual review. Refinements of library and association rules, and testing on more diverse samples may further improve its performance. CONCLUSION Our NLP system successfully extracts clinically useful information from mammography reports. Moreover, SAS is a feasible platform for implementing NLP algorithms.
Collapse
Affiliation(s)
- Hongyuan Gao
- Group Health Research Institute, Seattle, WA, USA.
| | | | | | | |
Collapse
|