1
|
Shah C, Nachand D, Wald C, Chen PH. Keeping Patient Data Secure in the Age of Radiology Artificial Intelligence: Cybersecurity Considerations and Future Directions. J Am Coll Radiol 2023; 20:828-835. [PMID: 37488026 DOI: 10.1016/j.jacr.2023.06.023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 06/14/2023] [Indexed: 07/26/2023]
Abstract
Artificial intelligence (AI)-based solutions are increasingly being incorporated into radiology workflows. Implementation of AI comes along with cybersecurity risks and challenges that practices should be aware of and mitigate for a successful and secure deployment. In this article, these cybersecurity issues are examined through the lens of the "CIA" triad framework-confidentiality, integrity, and availability. We discuss the implications of implementation configurations and development approaches on data security and confidentiality and the potential impact that the insertion of AI can have on the truthfulness of data, access to data, and the cybersecurity attack surface. Finally, we provide a checklist to address important security considerations before deployment of an AI application, and discuss future advances in AI addressing some of these security concerns.
Collapse
Affiliation(s)
- Chintan Shah
- Associate Staff, Section of Neuroradiology and Section of Imaging Informatics, Safety, Improvement, Quality and Experience Officer-Neuroradiology, Department of Radiology, Imaging Institute, Cleveland Clinic, Cleveland, Ohio.
| | - Douglas Nachand
- Staff, Section of Abdominal Imaging and Section of Imaging Informatics, Cleveland Clinic, Cleveland, Ohio
| | - Christoph Wald
- Professor of Radiology, Tufts University Medical School, Lahey Health, Boston, Massachusetts; Chair, Lahey Radiology, Chair, ACR Informatics Commission. https://twitter.com/waldchristoph
| | - Po-Hao Chen
- Chief Informatics Officer, Imaging Institute, Medical Director for Enterprise Radiology, IT Division, Staff, Section of Musculoskeletal Imaging, Cleveland Clinic, Cleveland, Ohio Chair, Informatics Advisory Council, ACR; Co-Chair, 2023 Data Science Summit, ACR. https://twitter.com/howardpchen
| |
Collapse
|
2
|
Fu T, Plecha D, Sommer J. System-driven longitudinal follow-up of incidental imaging findings. Br J Radiol 2023; 96:20220573. [PMID: 36063362 PMCID: PMC9975520 DOI: 10.1259/bjr.20220573] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/27/2023] Open
Abstract
Increasing utilization of cross-sectional imaging has resulted in more clinically significant incidental findings being discovered. However, the current approach for handling these findings is commonly inconsistent and relies greatly on the efforts of individual clinicians. Making sure every actionable incidental finding is handled in a consistent and reliable manner can be difficult, especially for a large health system. We propose an approach to handling incidental findings aimed at improving patient follow-up rates, which involves implementing system-level processes that standardize the reporting of incidental findings, notification of clinicians and the patient, and centralized monitoring of longitudinal patient follow-up. We will lay out a general framework for standardized reporting of incidental findings by the radiologist using software integrated into the daily workflow. This should enable simultaneous notification of the ordering clinician, the patient's primary-care provider, and an incidental findings navigator. The navigator will "close the loop" by working with clinicians to notify the patient of the finding, coordinate patient follow-up, and document the finding and long-term follow-up. We hope this can serve as a basic framework to help large health systems design an incidental findings workflow to improve follow-up rates and reduce patient harm.
Collapse
Affiliation(s)
- Tianyuan Fu
- University Hospitals Cleveland Medical Center, Case Western Reserve University, Cleveland, Ohio
| | - Donna Plecha
- University Hospitals Cleveland Medical Center, Case Western Reserve University, Cleveland, Ohio
| | - Jennifer Sommer
- University Hospitals Cleveland Medical Center, Case Western Reserve University, Cleveland, Ohio
| |
Collapse
|
3
|
Banerjee I, Davis MA, Vey BL, Mazaheri S, Khan F, Zavaletta V, Gerard R, Gichoya JW, Patel B. Natural Language Processing Model for Identifying Critical Findings-A Multi-Institutional Study. J Digit Imaging 2023; 36:105-113. [PMID: 36344632 PMCID: PMC9984612 DOI: 10.1007/s10278-022-00712-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2022] [Revised: 09/02/2022] [Accepted: 10/03/2022] [Indexed: 11/09/2022] Open
Abstract
Improving detection and follow-up of recommendations made in radiology reports is a critical unmet need. The long and unstructured nature of radiology reports limits the ability of clinicians to assimilate the full report and identify all the pertinent information for prioritizing the critical cases. We developed an automated NLP pipeline using a transformer-based ClinicalBERT++ model which was fine-tuned on 3 M radiology reports and compared against the traditional BERT model. We validated the models on both internal hold-out ED cases from EUH as well as external cases from Mayo Clinic. We also evaluated the model by combining different sections of the radiology reports. On the internal test set of 3819 reports, the ClinicalBERT++ model achieved 0.96 f1-score while the BERT also achieved the same performance using the reason for exam and impression sections. However, ClinicalBERT++ outperformed BERT on the external test dataset of 2039 reports and achieved the highest performance for classifying critical finding reports (0.81 precision and 0.54 recall). The ClinicalBERT++ model has been successfully applied to large-scale radiology reports from 5 different sites. Automated NLP system that can analyze free-text radiology reports, along with the reason for the exam, to identify critical radiology findings and recommendations could enable automated alert notifications to clinicians about the need for clinical follow-up. The clinical significance of our proposed model is that it could be used as an additional layer of safeguard to clinical practice and reduce the chance of important findings reported in a radiology report is not overlooked by clinicians as well as provide a way to retrospectively track large hospital databases for evaluating the documentation of the critical findings.
Collapse
Affiliation(s)
- Imon Banerjee
- Department of Radiology, Mayo Clinic, 5777 E Mayo Blvd, Phoenix, AZ, 85054, USA.
- Arizona State University, SCAI, 6161 E Mayo Blvd, Phoenix, AZ, 85054, USA.
| | - Melissa A Davis
- School of Medicine, Emory University, 1364 Clifton Road NE, 30322, Atlanta, USA
| | - Brianna L Vey
- School of Medicine, Emory University, 1364 Clifton Road NE, 30322, Atlanta, USA
| | - Sina Mazaheri
- School of Medicine, Emory University, 1364 Clifton Road NE, 30322, Atlanta, USA
| | - Fiza Khan
- School of Medicine, Emory University, 1364 Clifton Road NE, 30322, Atlanta, USA
| | - Vaz Zavaletta
- School of Medicine, Emory University, 1364 Clifton Road NE, 30322, Atlanta, USA
| | - Roger Gerard
- School of Medicine, Emory University, 1364 Clifton Road NE, 30322, Atlanta, USA
| | - Judy Wawira Gichoya
- School of Medicine, Emory University, 1364 Clifton Road NE, 30322, Atlanta, USA
| | - Bhavik Patel
- Department of Radiology, Mayo Clinic, 5777 E Mayo Blvd, Phoenix, AZ, 85054, USA
- Arizona State University, SCAI, 6161 E Mayo Blvd, Phoenix, AZ, 85054, USA
| |
Collapse
|
4
|
Binsfeld Gonçalves L, Nesic I, Obradovic M, Stieltjes B, Weikert T, Bremerich J. Natural Language Processing and Graph Theory: Making Sense of Imaging Records in a Novel Representation Frame. JMIR Med Inform 2022; 10:e40534. [PMID: 36542426 PMCID: PMC9813822 DOI: 10.2196/40534] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Revised: 09/13/2022] [Accepted: 11/30/2022] [Indexed: 12/24/2022] Open
Abstract
BACKGROUND A concise visualization framework of related reports would increase readability and improve patient management. To this end, temporal referrals to prior comparative exams are an essential connection to previous exams in written reports. Due to unstructured narrative texts' variable structure and content, their extraction is hampered by poor computer readability. Natural language processing (NLP) permits the extraction of structured information from unstructured texts automatically and can serve as an essential input for such a novel visualization framework. OBJECTIVE This study proposes and evaluates an NLP-based algorithm capable of extracting the temporal referrals in written radiology reports, applies it to all the radiology reports generated for 10 years, introduces a graphical representation of imaging reports, and investigates its benefits for clinical and research purposes. METHODS In this single-center, university hospital, retrospective study, we developed a convolutional neural network capable of extracting the date of referrals from imaging reports. The model's performance was assessed by calculating precision, recall, and F1-score using an independent test set of 149 reports. Next, the algorithm was applied to our department's radiology reports generated from 2011 to 2021. Finally, the reports and their metadata were represented in a modulable graph. RESULTS For extracting the date of referrals, the named-entity recognition (NER) model had a high precision of 0.93, a recall of 0.95, and an F1-score of 0.94. A total of 1,684,635 reports were included in the analysis. Temporal reference was mentioned in 53.3% (656,852/1,684,635), explicitly stated as not available in 21.0% (258,386/1,684,635), and omitted in 25.7% (317,059/1,684,635) of the reports. Imaging records can be visualized in a directed and modulable graph, in which the referring links represent the connecting arrows. CONCLUSIONS Automatically extracting the date of referrals from unstructured radiology reports using deep learning NLP algorithms is feasible. Graphs refined the selection of distinct pathology pathways, facilitated the revelation of missing comparisons, and enabled the query of specific referring exam sequences. Further work is needed to evaluate its benefits in clinics, research, and resource planning.
Collapse
Affiliation(s)
- Laurent Binsfeld Gonçalves
- Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Ivan Nesic
- Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Marko Obradovic
- Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Bram Stieltjes
- Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Thomas Weikert
- Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Jens Bremerich
- Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Basel, Switzerland
| |
Collapse
|
5
|
Tulk Jesso S, Kelliher A, Sanghavi H, Martin T, Henrickson Parker S. Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review. Front Psychol 2022; 13:830345. [PMID: 35465567 PMCID: PMC9022040 DOI: 10.3389/fpsyg.2022.830345] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Accepted: 02/09/2022] [Indexed: 12/11/2022] Open
Abstract
The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.
Collapse
Affiliation(s)
- Stephanie Tulk Jesso
- Fralin Biomedical Research Institute, Virginia Tech, Roanoke, VA, United States.,Institute for Creativity, Arts, and Technology, Blacksburg, VA, United States
| | - Aisling Kelliher
- Department of Computer Science, College of Engineering, Virginia Tech, Blacksburg, VA, United States
| | | | - Thomas Martin
- Institute for Creativity, Arts, and Technology, Blacksburg, VA, United States.,Department of Electrical and Computer Engineering, College of Engineering, Virginia Tech, Blacksburg, VA, United States
| | - Sarah Henrickson Parker
- Fralin Biomedical Research Institute, Virginia Tech, Roanoke, VA, United States.,Department of Health Systems and Implementation Science, Virginia Tech Carilion School of Medicine, Roanoke, VA, United States
| |
Collapse
|
6
|
Tadavarthi Y, Makeeva V, Wagstaff W, Zhan H, Podlasek A, Bhatia N, Heilbrun M, Krupinski E, Safdar N, Banerjee I, Gichoya J, Trivedi H. Overview of Noninterpretive Artificial Intelligence Models for Safety, Quality, Workflow, and Education Applications in Radiology Practice. Radiol Artif Intell 2022; 4:e210114. [PMID: 35391770 DOI: 10.1148/ryai.210114] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2021] [Revised: 12/17/2021] [Accepted: 01/11/2022] [Indexed: 12/17/2022]
Abstract
Artificial intelligence has become a ubiquitous term in radiology over the past several years, and much attention has been given to applications that aid radiologists in the detection of abnormalities and diagnosis of diseases. However, there are many potential applications related to radiologic image quality, safety, and workflow improvements that present equal, if not greater, value propositions to radiology practices, insurance companies, and hospital systems. This review focuses on six major categories for artificial intelligence applications: study selection and protocoling, image acquisition, worklist prioritization, study reporting, business applications, and resident education. All of these categories can substantially affect different aspects of radiology practices and workflows. Each of these categories has different value propositions in terms of whether they could be used to increase efficiency, improve patient safety, increase revenue, or save costs. Each application is covered in depth in the context of both current and future areas of work. Keywords: Use of AI in Education, Application Domain, Supervised Learning, Safety © RSNA, 2022.
Collapse
Affiliation(s)
- Yasasvi Tadavarthi
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Valeria Makeeva
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - William Wagstaff
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Henry Zhan
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Anna Podlasek
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Neil Bhatia
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Marta Heilbrun
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Elizabeth Krupinski
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Nabile Safdar
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Imon Banerjee
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Judy Gichoya
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| | - Hari Trivedi
- Department of Medicine, Medical College of Georgia, Augusta, Ga (Y.T.); Department of Radiology and Imaging Sciences (V.M., W.W., H.Z., M.H., E.K., N.S., J.G., H.T.), School of Medicine (N.B.), and Department of Biomedical Informatics (I.B.), Emory University, 1364 E Clifton Rd NE, Atlanta, GA 30322; and Southend University Hospital NHS Foundation Trust, Westcliff-on-Sea, UK (A.P.)
| |
Collapse
|
7
|
Dedhia PH, Chen K, Song Y, LaRose E, Imbus JR, Peissig PL, Mendonca EA, Schneider DF. Ambiguous and Incomplete: Natural Language Processing Reveals Problematic Reporting Styles in Thyroid Ultrasound Reports. Methods Inf Med 2022; 61:11-18. [PMID: 34991173 DOI: 10.1055/s-0041-1740493] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
OBJECTIVE Natural language processing (NLP) systems convert unstructured text into analyzable data. Here, we describe the performance measures of NLP to capture granular details on nodules from thyroid ultrasound (US) reports and reveal critical issues with reporting language. METHODS We iteratively developed NLP tools using clinical Text Analysis and Knowledge Extraction System (cTAKES) and thyroid US reports from 2007 to 2013. We incorporated nine nodule features for NLP extraction. Next, we evaluated the precision, recall, and accuracy of our NLP tools using a separate set of US reports from an academic medical center (A) and a regional health care system (B) during the same period. Two physicians manually annotated each test-set report. A third physician then adjudicated discrepancies. The adjudicated "gold standard" was then used to evaluate NLP performance on the test-set. RESULTS A total of 243 thyroid US reports contained 6,405 data elements. Inter-annotator agreement for all elements was 91.3%. Compared with the gold standard, overall recall of the NLP tool was 90%. NLP recall for thyroid lobe or isthmus characteristics was: laterality 96% and size 95%. NLP accuracy for nodule characteristics was: laterality 92%, size 92%, calcifications 76%, vascularity 65%, echogenicity 62%, contents 76%, and borders 40%. NLP recall for presence or absence of lymphadenopathy was 61%. Reporting style accounted for 18% errors. For example, the word "heterogeneous" interchangeably referred to nodule contents or echogenicity. While nodule dimensions and laterality were often described, US reports only described contents, echogenicity, vascularity, calcifications, borders, and lymphadenopathy, 46, 41, 17, 15, 9, and 41% of the time, respectively. Most nodule characteristics were equally likely to be described at hospital A compared with hospital B. CONCLUSIONS NLP can automate extraction of critical information from thyroid US reports. However, ambiguous and incomplete reporting language hinders performance of NLP systems regardless of institutional setting. Standardized or synoptic thyroid US reports could improve NLP performance.
Collapse
Affiliation(s)
- Priya H Dedhia
- Department of Surgery, Division of Surgical Oncology, Ohio State University Comprehensive Cancer Center and Ohio State University Wexner Medical Center, Columbus, Ohio, United States
| | - Kallie Chen
- Department of Surgery, Division of Endocrine Surgery, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Yiqiang Song
- Department of Biostatistics and Medical Informatics, Department of Pediatrics, University of Wisconsin-Madison, Madison, Wisconsin, United States
| | - Eric LaRose
- Center for Precision Medicine Research, Marshfield Clinic Research Institute, Marshfield Clinic Health System, Marshfield, Wisconsin, United States
| | - Joseph R Imbus
- Department of Surgery, Division of Endocrine Surgery, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Peggy L Peissig
- Center for Precision Medicine Research, Marshfield Clinic Research Institute, Marshfield Clinic Health System, Marshfield, Wisconsin, United States
| | - Eneida A Mendonca
- Department of Biostatistics and Medical Informatics, Department of Pediatrics, University of Wisconsin-Madison, Madison, Wisconsin, United States.,Department of Pediatrics, Department of Biostatistics and Health Data Sciences, Indiana University, Indianapolis, Indiana, United States
| | - David F Schneider
- Department of Surgery, Division of Endocrine Surgery, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States
| |
Collapse
|
8
|
Different Data Mining Approaches Based Medical Text Data. JOURNAL OF HEALTHCARE ENGINEERING 2021; 2021:1285167. [PMID: 34912530 PMCID: PMC8668297 DOI: 10.1155/2021/1285167] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/02/2021] [Accepted: 11/18/2021] [Indexed: 12/15/2022]
Abstract
The amount of medical text data is increasing dramatically. Medical text data record the progress of medicine and imply a large amount of medical knowledge. As a natural language, they are characterized by semistructured, high-dimensional, high data volume semantics and cannot participate in arithmetic operations. Therefore, how to extract useful knowledge or information from the total available data is very important task. Using various techniques of data mining can extract valuable knowledge or information from data. In the current study, we reviewed different approaches to apply for medical text data mining. The advantages and shortcomings for each technique compared to different processes of medical text data were analyzed. We also explored the applications of algorithms for providing insights to the users and enabling them to use the resources for the specific challenges in medical text data. Further, the main challenges in medical text data mining were discussed. Findings of this paper are benefit for helping the researchers to choose the reasonable techniques for mining medical text data and presenting the main challenges to them in medical text data mining.
Collapse
|
9
|
Steinkamp J, Cook TS. Basic Artificial Intelligence Techniques: Natural Language Processing of Radiology Reports. Radiol Clin North Am 2021; 59:919-931. [PMID: 34689877 DOI: 10.1016/j.rcl.2021.06.003] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
Natural language processing (NLP) is a subfield of computer science and linguistics that can be applied to extract meaningful information from radiology reports. Symbolic NLP is rule based and well suited to problems that can be explicitly defined by a set of rules. Statistical NLP is better situated to problems that cannot be well defined and requires annotated or labeled examples from which machine learning algorithms can infer the rules. Both symbolic and statistical NLP have found success in a variety of radiology use cases. More recently, deep learning approaches, including transformers, have gained traction and demonstrated good performance.
Collapse
Affiliation(s)
- Jackson Steinkamp
- Department of Medicine, Hospital of the University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA 19104, USA
| | - Tessa S Cook
- Perelman School of Medicine at the University of Pennsylvania, 3400 Spruce Street, 1 Silverstein Radiology, Philadelphia, PA 19104, USA.
| |
Collapse
|
10
|
Yuan Z, Finan S, Warner J, Savova G, Hochheiser H. Interactive Exploration of Longitudinal Cancer Patient Histories Extracted From Clinical Text. JCO Clin Cancer Inform 2021; 4:412-420. [PMID: 32383981 PMCID: PMC7265796 DOI: 10.1200/cci.19.00115] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
PURPOSE Retrospective cancer research requires identification of patients matching both categorical and temporal inclusion criteria, often on the basis of factors exclusively available in clinical notes. Although natural language processing approaches for inferring higher-level concepts have shown promise for bringing structure to clinical texts, interpreting results is often challenging, involving the need to move between abstracted representations and constituent text elements. Our goal was to build interactive visual tools to support the process of interpreting rich representations of histories of patients with cancer. METHODS Qualitative inquiry into user tasks and goals, a structured data model, and an innovative natural language processing pipeline were used to guide design. RESULTS The resulting information visualization tool provides cohort- and patient-level views with linked interactions between components. CONCLUSION Interactive tools hold promise for facilitating the interpretation of patient summaries and identification of cohorts for retrospective research.
Collapse
Affiliation(s)
- Zhou Yuan
- University of Pittsburgh, Pittsburgh, PA
| | | | | | | | | |
Collapse
|
11
|
Shah M, Shu D, Prasath VBS, Ni Y, Schapiro AH, Dufendach KR. Machine Learning for Detection of Correct Peripherally Inserted Central Catheter Tip Position from Radiology Reports in Infants. Appl Clin Inform 2021; 12:856-863. [PMID: 34496420 PMCID: PMC8426077 DOI: 10.1055/s-0041-1735178] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023] Open
Abstract
BACKGROUND In critically ill infants, the position of a peripherally inserted central catheter (PICC) must be confirmed frequently, as the tip may move from its original position and run the risk of hyperosmolar vascular damage or extravasation into surrounding spaces. Automated detection of PICC tip position holds great promise for alerting bedside clinicians to noncentral PICCs. OBJECTIVES This research seeks to use natural language processing (NLP) and supervised machine learning (ML) techniques to predict PICC tip position based primarily on text analysis of radiograph reports from infants with an upper extremity PICC. METHODS Radiographs, containing a PICC line in infants under 6 months of age, were manually classified into 12 anatomical locations based on the radiologist's textual report of the PICC line's tip. After categorization, we performed a 70/30 train/test split and benchmarked the performance of seven different (neural network, support vector machine, the naïve Bayes, decision tree, random forest, AdaBoost, and K-nearest neighbors) supervised ML algorithms. After optimization, we calculated accuracy, precision, and recall of each algorithm's ability to correctly categorize the stated location of the PICC tip. RESULTS A total of 17,337 radiographs met criteria for inclusion and were labeled manually. Interrater agreement was 99.1%. Support vector machines and neural networks yielded accuracies as high as 98% in identifying PICC tips in central versus noncentral position (binary outcome) and accuracies as high as 95% when attempting to categorize the individual anatomical location (12-category outcome). CONCLUSION Our study shows that ML classifiers can automatically extract the anatomical location of PICC tips from radiology reports. Two ML classifiers, support vector machine (SVM) and a neural network, obtained top accuracies in both binary and multiple category predictions. Implementing these algorithms in a neonatal intensive care unit as a clinical decision support system may help clinicians address PICC line position.
Collapse
Affiliation(s)
- Manan Shah
- Division of Neonatal and Pulmonary Biology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
- Address for correspondence Manan Shah, MD Division of Neonatal and Pulmonary Biology, Cincinnati Children's Hospital Medical Center3333 Burnet Avenue MLC 7009, Cincinnati, OH 45229United States
| | - Derek Shu
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
| | - V. B. Surya Prasath
- Division of Bioinformatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
| | - Yizhao Ni
- Division of Bioinformatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
- Department of Biomedical Informatics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
| | - Andrew H. Schapiro
- Department of Radiology and Medical Imaging, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
- Department of Radiology, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
| | - Kevin R. Dufendach
- Division of Neonatal and Pulmonary Biology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
- Division of Bioinformatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
| |
Collapse
|
12
|
Canales L, Menke S, Marchesseau S, D'Agostino A, Del Rio-Bermudez C, Taberna M, Tello J. Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology. JMIR Med Inform 2021; 9:e20492. [PMID: 34297002 PMCID: PMC8367121 DOI: 10.2196/20492] [Citation(s) in RCA: 32] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Key Words] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2020] [Revised: 07/31/2020] [Accepted: 06/17/2021] [Indexed: 12/22/2022] Open
Abstract
Background Clinical natural language processing (cNLP) systems are of crucial importance due to their increasing capability in extracting clinically important information from free text contained in electronic health records (EHRs). The conversion of a nonstructured representation of a patient’s clinical history into a structured format enables medical doctors to generate clinical knowledge at a level that was not possible before. Finally, the interpretation of the insights gained provided by cNLP systems has a great potential in driving decisions about clinical practice. However, carrying out robust evaluations of those cNLP systems is a complex task that is hindered by a lack of standard guidance on how to systematically approach them. Objective Our objective was to offer natural language processing (NLP) experts a methodology for the evaluation of cNLP systems to assist them in carrying out this task. By following the proposed phases, the robustness and representativeness of the performance metrics of their own cNLP systems can be assured. Methods The proposed evaluation methodology comprised five phases: (1) the definition of the target population, (2) the statistical document collection, (3) the design of the annotation guidelines and annotation project, (4) the external annotations, and (5) the cNLP system performance evaluation. We presented the application of all phases to evaluate the performance of a cNLP system called “EHRead Technology” (developed by Savana, an international medical company), applied in a study on patients with asthma. As part of the evaluation methodology, we introduced the Sample Size Calculator for Evaluations (SLiCE), a software tool that calculates the number of documents needed to achieve a statistically useful and resourceful gold standard. Results The application of the proposed evaluation methodology on a real use-case study of patients with asthma revealed the benefit of the different phases for cNLP system evaluations. By using SLiCE to adjust the number of documents needed, a meaningful and resourceful gold standard was created. In the presented use-case, using as little as 519 EHRs, it was possible to evaluate the performance of the cNLP system and obtain performance metrics for the primary variable within the expected CIs. Conclusions We showed that our evaluation methodology can offer guidance to NLP experts on how to approach the evaluation of their cNLP systems. By following the five phases, NLP experts can assure the robustness of their evaluation and avoid unnecessary investment of human and financial resources. Besides the theoretical guidance, we offer SLiCE as an easy-to-use, open-source Python library.
Collapse
Affiliation(s)
- Lea Canales
- Department of Software and Computing System, University of Alicante, Alicante, Spain
| | | | | | | | | | | | | |
Collapse
|
13
|
Newman-Griffis D, Lehman JF, Rosé C, Hochheiser H. Translational NLP: A New Paradigm and General Principles for Natural Language Processing Research. PROCEEDINGS OF THE CONFERENCE. ASSOCIATION FOR COMPUTATIONAL LINGUISTICS. NORTH AMERICAN CHAPTER. MEETING 2021; 2021:4125-4138. [PMID: 34179899 PMCID: PMC8223521] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Natural language processing (NLP) research combines the study of universal principles, through basic science, with applied science targeting specific use cases and settings. However, the process of exchange between basic NLP and applications is often assumed to emerge naturally, resulting in many innovations going unapplied and many important questions left unstudied. We describe a new paradigm of Translational NLP, which aims to structure and facilitate the processes by which basic and applied NLP research inform one another. Translational NLP thus presents a third research paradigm, focused on understanding the challenges posed by application needs and how these challenges can drive innovation in basic science and technology design. We show that many significant advances in NLP research have emerged from the intersection of basic principles with application needs, and present a conceptual framework outlining the stakeholders and key questions in translational research. Our framework provides a roadmap for developing Translational NLP as a dedicated research area, and identifies general translational principles to facilitate exchange between basic and applied research.
Collapse
Affiliation(s)
| | - Jill Fain Lehman
- Human-Computer Interaction Institute, Carnegie Mellon University, USA
| | - Carolyn Rosé
- Language Technologies Institute, Carnegie Mellon University, USA
| | - Harry Hochheiser
- Department of Biomedical Informatics, University of Pittsburgh, USA
| |
Collapse
|
14
|
Text Messaging-Based Medical Diagnosis Using Natural Language Processing and Fuzzy Logic. JOURNAL OF HEALTHCARE ENGINEERING 2020. [DOI: 10.1155/2020/8839524] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
The use of natural language processing (NLP) methods and their application to developing conversational systems for health diagnosis increases patients’ access to medical knowledge. In this study, a chatbot service was developed for the Covenant University Doctor (CUDoctor) telehealth system based on fuzzy logic rules and fuzzy inference. The service focuses on assessing the symptoms of tropical diseases in Nigeria. Telegram Bot Application Programming Interface (API) was used to create the interconnection between the chatbot and the system, while Twilio API was used for interconnectivity between the system and a short messaging service (SMS) subscriber. The service uses the knowledge base consisting of known facts on diseases and symptoms acquired from medical ontologies. A fuzzy support vector machine (SVM) is used to effectively predict the disease based on the symptoms inputted. The inputs of the users are recognized by NLP and are forwarded to the CUDoctor for decision support. Finally, a notification message displaying the end of the diagnosis process is sent to the user. The result is a medical diagnosis system which provides a personalized diagnosis utilizing self-input from users to effectively diagnose diseases. The usability of the developed system was evaluated using the system usability scale (SUS), yielding a mean SUS score of 80.4, which indicates the overall positive evaluation.
Collapse
|
15
|
Bala W, Steinkamp J, Feeney T, Gupta A, Sharma A, Kantrowitz J, Cordella N, Moses J, Drake FT. A Web Application for Adrenal Incidentaloma Identification, Tracking, and Management Using Machine Learning. Appl Clin Inform 2020; 11:606-616. [PMID: 32937677 DOI: 10.1055/s-0040-1715892] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023] Open
Abstract
BACKGROUND Incidental radiographic findings, such as adrenal nodules, are commonly identified in imaging studies and documented in radiology reports. However, patients with such findings frequently do not receive appropriate follow-up, partially due to the lack of tools for the management of such findings and the time required to maintain up-to-date lists. Natural language processing (NLP) is capable of extracting information from free-text clinical documents and could provide the basis for software solutions that do not require changes to clinical workflows. OBJECTIVES In this manuscript we present (1) a machine learning algorithm we trained to identify radiology reports documenting the presence of a newly discovered adrenal incidentaloma, and (2) the web application and results database we developed to manage these clinical findings. METHODS We manually annotated a training corpus of 4,090 radiology reports from across our institution with a binary label indicating whether or not a report contains a newly discovered adrenal incidentaloma. We trained a convolutional neural network to perform this text classification task. Over the NLP backbone we built a web application that allows users to coordinate clinical management of adrenal incidentalomas in real time. RESULTS The annotated dataset included 404 positive (9.9%) and 3,686 (90.1%) negative reports. Our model achieved a sensitivity of 92.9% (95% confidence interval: 80.9-97.5%), a positive predictive value of 83.0% (69.9-91.1)%, a specificity of 97.8% (95.8-98.9)%, and an F1 score of 87.6%. We developed a front-end web application based on the model's output. CONCLUSION Developing an NLP-enabled custom web application for tracking and management of high-risk adrenal incidentalomas is feasible in a resource constrained, safety net hospital. Such applications can be used by an institution's quality department or its primary care providers and can easily be generalized to other types of clinical findings.
Collapse
Affiliation(s)
- Wasif Bala
- Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
| | - Jackson Steinkamp
- Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
| | - Timothy Feeney
- Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
| | - Avneesh Gupta
- Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
| | - Abhinav Sharma
- Department of Family Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, Ontario, Canada
| | - Jake Kantrowitz
- Department of Internal Medicine, Kent Hospital, Brown University Alpert Medical School, Warwick, Rhode Island, United States
| | - Nicholas Cordella
- Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
| | - James Moses
- Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
| | | |
Collapse
|