1
|
Hu D, Liu B, Zhu X, Lu X, Wu N. Zero-shot information extraction from radiological reports using ChatGPT. Int J Med Inform 2024; 183:105321. [PMID: 38157785 DOI: 10.1016/j.ijmedinf.2023.105321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2023] [Revised: 12/04/2023] [Accepted: 12/16/2023] [Indexed: 01/03/2024]
Abstract
INTRODUCTION Electronic health records contain an enormous amount of valuable information recorded in free text. Information extraction is the strategy to transform free text into structured data, but some of its components require annotated data to tune, which has become a bottleneck. Large language models achieve good performances on various downstream NLP tasks without parameter tuning, becoming a possible way to extract information in a zero-shot manner. METHODS In this study, we aim to explore whether the most popular large language model, ChatGPT, can extract information from the radiological reports. We first design the prompt template for the interested information in the CT reports. Then, we generate the prompts by combining the prompt template with the CT reports as the inputs of ChatGPT to obtain the responses. A post-processing module is developed to transform the responses into structured extraction results. Besides, we add prior medical knowledge to the prompt template to reduce wrong extraction results. We also explore the consistency of the extraction results. RESULTS We conducted the experiments with 847 real CT reports. The experimental results indicate that ChatGPT can achieve competitive performances for some extraction tasks like tumor location, tumor long and short diameters compared with the baseline information extraction system. By adding some prior medical knowledge to the prompt template, extraction tasks about tumor spiculations and lobulations obtain significant improvements but tasks about tumor density and lymph node status do not achieve better performances. CONCLUSION ChatGPT can achieve competitive information extraction for radiological reports in a zero-shot manner. Adding prior medical knowledge as instructions can further improve performances for some extraction tasks but may lead to worse performances for some complex extraction tasks.
Collapse
Affiliation(s)
- Danqing Hu
- Zhejiang Lab, Hangzhou, 311121, Zhejiang, China.
| | - Bing Liu
- Department of Thoracic Surgery II, Peking University Cancer Hospital and Institute, Beijing, 100142, China
| | - Xiaofeng Zhu
- Zhejiang Lab, Hangzhou, 311121, Zhejiang, China.
| | - Xudong Lu
- College of Biomedical Engineering and Instrumental Science, Zhejiang University, Hangzhou, 310027, Zhejiang, China
| | - Nan Wu
- Department of Thoracic Surgery II, Peking University Cancer Hospital and Institute, Beijing, 100142, China.
| |
Collapse
|
2
|
Ahumada R, Dunstan J, Rojas M, Peñafiel S, Paredes I, Báez P. Automatic Detection of Distant Metastasis Mentions in Radiology Reports in Spanish. JCO Clin Cancer Inform 2024; 8:e2300130. [PMID: 38194615 PMCID: PMC10793975 DOI: 10.1200/cci.23.00130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 10/12/2023] [Accepted: 11/08/2023] [Indexed: 01/11/2024] Open
Abstract
PURPOSE A critical task in oncology is extracting information related to cancer metastasis from electronic health records. Metastasis-related information is crucial for planning treatment, evaluating patient prognoses, and cancer research. However, the unstructured way in which findings of distant metastasis are often written in radiology reports makes it difficult to extract information automatically. The main aim of this study was to extract distant metastasis findings from free-text imaging and nuclear medicine reports to classify the patient status according to the presence or absence of distant metastasis. MATERIALS AND METHODS We created a distant metastasis annotated corpus using positron emission tomography-computed tomography and computed tomography reports of patients with prostate, colorectal, and breast cancers. Entities were labeled M1 or M0 according to affirmative or negative metastasis descriptions. We used a named entity recognition model on the basis of a bidirectional long short-term memory model and conditional random fields to identify entities. Mentions were subsequently used to classify whole reports into M1 or M0. RESULTS The model detected distant metastasis mentions with a weighted average F1 score performance of 0.84. Whole reports were classified with an F1 score of 0.92 for M0 documents and 0.90 for M1 documents. CONCLUSION These results show the usefulness of the model in detecting distant metastasis findings in three different types of cancer and the consequent classification of reports. The relevance of this study is to generate structured distant metastasis information from free-text imaging reports in Spanish. In addition, the manually annotated corpus, annotation guidelines, and code are freely released to the research community.
Collapse
Affiliation(s)
- Ricardo Ahumada
- Center of Medical Informatics and Telemedicine, Faculty of Medicine, University of Chile, Santiago, Chile
| | - Jocelyn Dunstan
- Department of Computer Science & the Institute for Mathematical Computing, Pontificia Universidad Católica de Chile, Santiago, Chile
| | - Matías Rojas
- Center for Mathematical Modeling—CNRS IRL 2807, Faculty of Physical and Mathematical Sciences, University of Chile, Santiago, Chile
| | - Sergio Peñafiel
- Unidad de Informática Médica y Data Science, Departamento de Investigación del Cáncer, Instituto Oncológico Fundación Arturo López Pérez, Santiago, Chile
| | - Inti Paredes
- Unidad de Informática Médica y Data Science, Departamento de Investigación del Cáncer, Instituto Oncológico Fundación Arturo López Pérez, Santiago, Chile
| | - Pablo Báez
- Center of Medical Informatics and Telemedicine, Faculty of Medicine, University of Chile, Santiago, Chile
| |
Collapse
|
3
|
Lee K, Liu Z, Chandran U, Kalsekar I, Laxmanan B, Higashi MK, Jun T, Ma M, Li M, Mai Y, Gilman C, Wang T, Ai L, Aggarwal P, Pan Q, Oh W, Stolovitzky G, Schadt E, Wang X. Detecting Ground Glass Opacity Features in Patients With Lung Cancer: Automated Extraction and Longitudinal Analysis via Deep Learning-Based Natural Language Processing. JMIR AI 2023; 2:e44537. [PMID: 38875565 PMCID: PMC11041451 DOI: 10.2196/44537] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/23/2022] [Revised: 01/30/2023] [Accepted: 03/31/2023] [Indexed: 06/16/2024]
Abstract
BACKGROUND Ground-glass opacities (GGOs) appearing in computed tomography (CT) scans may indicate potential lung malignancy. Proper management of GGOs based on their features can prevent the development of lung cancer. Electronic health records are rich sources of information on GGO nodules and their granular features, but most of the valuable information is embedded in unstructured clinical notes. OBJECTIVE We aimed to develop, test, and validate a deep learning-based natural language processing (NLP) tool that automatically extracts GGO features to inform the longitudinal trajectory of GGO status from large-scale radiology notes. METHODS We developed a bidirectional long short-term memory with a conditional random field-based deep-learning NLP pipeline to extract GGO and granular features of GGO retrospectively from radiology notes of 13,216 lung cancer patients. We evaluated the pipeline with quality assessments and analyzed cohort characterization of the distribution of nodule features longitudinally to assess changes in size and solidity over time. RESULTS Our NLP pipeline built on the GGO ontology we developed achieved between 95% and 100% precision, 89% and 100% recall, and 92% and 100% F1-scores on different GGO features. We deployed this GGO NLP model to extract and structure comprehensive characteristics of GGOs from 29,496 radiology notes of 4521 lung cancer patients. Longitudinal analysis revealed that size increased in 16.8% (240/1424) of patients, decreased in 14.6% (208/1424), and remained unchanged in 68.5% (976/1424) in their last note compared to the first note. Among 1127 patients who had longitudinal radiology notes of GGO status, 815 (72.3%) were reported to have stable status, and 259 (23%) had increased/progressed status in the subsequent notes. CONCLUSIONS Our deep learning-based NLP pipeline can automatically extract granular GGO features at scale from electronic health records when this information is documented in radiology notes and help inform the natural history of GGO. This will open the way for a new paradigm in lung cancer prevention and early detection.
Collapse
Affiliation(s)
| | | | - Urmila Chandran
- Lung Cancer Initiative, Johnson & Johnson, New Brunswick, NJ, United States
| | - Iftekhar Kalsekar
- Lung Cancer Initiative, Johnson & Johnson, New Brunswick, NJ, United States
| | - Balaji Laxmanan
- Lung Cancer Initiative, Johnson & Johnson, New Brunswick, NJ, United States
| | | | - Tomi Jun
- Sema4, Stamford, CT, United States
| | - Meng Ma
- Sema4, Stamford, CT, United States
| | | | - Yun Mai
- Sema4, Stamford, CT, United States
| | | | | | - Lei Ai
- Sema4, Stamford, CT, United States
| | | | - Qi Pan
- Sema4, Stamford, CT, United States
| | - William Oh
- Icahn School of Medicine at Mount Sinai, New York, NY, United States
| | | | - Eric Schadt
- Icahn School of Medicine at Mount Sinai, New York, NY, United States
| | | |
Collapse
|
4
|
Mu J, Kuang K, Ao M, Li W, Dai H, Ouyang Z, Li J, Huang J, Guo S, Yang J, Yang L. Deep learning predicts malignancy and metastasis of solid pulmonary nodules from CT scans. Front Med (Lausanne) 2023; 10:1145846. [PMID: 37275359 PMCID: PMC10235703 DOI: 10.3389/fmed.2023.1145846] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2023] [Accepted: 04/10/2023] [Indexed: 06/07/2023] Open
Abstract
In the clinic, it is difficult to distinguish the malignancy and aggressiveness of solid pulmonary nodules (PNs). Incorrect assessments may lead to delayed diagnosis and an increased risk of complications. We developed and validated a deep learning-based model for the prediction of malignancy as well as local or distant metastasis in solid PNs based on CT images of primary lesions during initial diagnosis. In this study, we reviewed the data from multiple patients with solid PNs at our institution from 1 January 2019 to 30 April 2022. The patients were divided into three groups: benign, Ia-stage lung cancer, and T1-stage lung cancer with metastasis. Each cohort was further split into training and testing groups. The deep learning system predicted the malignancy and metastasis status of solid PNs based on CT images, and then we compared the malignancy prediction results among four different levels of clinicians. Experiments confirmed that human-computer collaboration can further enhance diagnostic accuracy. We made a held-out testing set of 134 cases, with 689 cases in total. Our convolutional neural network model reached an area under the ROC (AUC) of 80.37% for malignancy prediction and an AUC of 86.44% for metastasis prediction. In observer studies involving four clinicians, the proposed deep learning method outperformed a junior respiratory clinician and a 5-year respiratory clinician by considerable margins; it was on par with a senior respiratory clinician and was only slightly inferior to a senior radiologist. Our human-computer collaboration experiment showed that by simply adding binary human diagnosis into model prediction probabilities, model AUC scores improved to 81.80-88.70% when combined with three out of four clinicians. In summary, the deep learning method can accurately diagnose the malignancy of solid PNs, improve its performance when collaborating with human experts, predict local or distant metastasis in patients with T1-stage lung cancer, and facilitate the application of precision medicine.
Collapse
Affiliation(s)
- Junhao Mu
- Department of Respiratory and Critical Care Medicine, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Kaiming Kuang
- Dianei Technology, Shanghai, China
- University of California, San Diego, San Diego, CA, United States
| | - Min Ao
- Department of Respiratory and Critical Care Medicine, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Weiyi Li
- Department of Respiratory and Critical Care Medicine, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Haiyun Dai
- Department of Respiratory and Critical Care Medicine, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Zubin Ouyang
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Jingyu Li
- Dianei Technology, Shanghai, China
- School of Computer Science, Wuhan University, Wuhan, China
| | - Jing Huang
- Department of Respiratory and Critical Care Medicine, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Shuliang Guo
- Department of Respiratory and Critical Care Medicine, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Jiancheng Yang
- Dianei Technology, Shanghai, China
- Shanghai Jiao Tong University, Shanghai, China
- École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Li Yang
- Department of Respiratory and Critical Care Medicine, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| |
Collapse
|
5
|
Afshar M, Adelaine S, Resnik F, Mundt MP, Long J, Leaf M, Ampian T, Wills GJ, Schnapp B, Chao M, Brown R, Joyce C, Sharma B, Dligach D, Burnside ES, Mahoney J, Churpek MM, Patterson BW, Liao F. Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults. JMIR Med Inform 2023; 11:e44977. [PMID: 37079367 PMCID: PMC10160938 DOI: 10.2196/44977] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2022] [Revised: 02/01/2023] [Accepted: 03/26/2023] [Indexed: 04/21/2023] Open
Abstract
BACKGROUND The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. OBJECTIVE We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. METHODS The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. RESULTS The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. CONCLUSIONS The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence-driven CDS. TRIAL REGISTRATION ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480.
Collapse
Affiliation(s)
- Majid Afshar
- University of Wisconsin - Madison, Madison, WI, United States
| | | | - Felice Resnik
- University of Wisconsin - Madison, Madison, WI, United States
| | - Marlon P Mundt
- University of Wisconsin - Madison, Madison, WI, United States
| | - John Long
- University of Wisconsin - Madison, Madison, WI, United States
| | - Margaret Leaf
- University of Wisconsin - Madison, Madison, WI, United States
| | - Theodore Ampian
- University of Wisconsin - Madison, Madison, WI, United States
| | - Graham J Wills
- University of Wisconsin - Madison, Madison, WI, United States
| | | | - Michael Chao
- University of Wisconsin - Madison, Madison, WI, United States
| | - Randy Brown
- University of Wisconsin - Madison, Madison, WI, United States
| | - Cara Joyce
- Loyola University Chicago, Chicago, IL, United States
| | - Brihat Sharma
- University of Wisconsin - Madison, Madison, WI, United States
| | | | | | - Jane Mahoney
- University of Wisconsin - Madison, Madison, WI, United States
| | | | | | - Frank Liao
- University of Wisconsin - Madison, Madison, WI, United States
| |
Collapse
|