1
|
Javed M. The role of ChatGPT in medicine. Am J Med Sci 2024; 368:411. [PMID: 38925427 DOI: 10.1016/j.amjms.2024.06.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2024] [Accepted: 06/19/2024] [Indexed: 06/28/2024]
Affiliation(s)
- Momina Javed
- Institute: Watim Medical College, Department of Pathology, Islamabad, Pakistan.
| |
Collapse
|
2
|
López-Úbeda P, Martín-Noguerol T, Díaz-Angulo C, Luna A. Evaluation of large language models performance against humans for summarizing MRI knee radiology reports: A feasibility study. Int J Med Inform 2024; 187:105443. [PMID: 38615509 DOI: 10.1016/j.ijmedinf.2024.105443] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 03/20/2024] [Accepted: 03/29/2024] [Indexed: 04/16/2024]
Abstract
OBJECTIVES This study addresses the critical need for accurate summarization in radiology by comparing various Large Language Model (LLM)-based approaches for automatic summary generation. With the increasing volume of patient information, accurately and concisely conveying radiological findings becomes crucial for effective clinical decision-making. Minor inaccuracies in summaries can lead to significant consequences, highlighting the need for reliable automated summarization tools. METHODS We employed two language models - Text-to-Text Transfer Transformer (T5) and Bidirectional and Auto-Regressive Transformers (BART) - in both fine-tuned and zero-shot learning scenarios and compared them with a Recurrent Neural Network (RNN). Additionally, we conducted a comparative analysis of 100 MRI report summaries, using expert human judgment and criteria such as coherence, relevance, fluency, and consistency, to evaluate the models against the original radiologist summaries. To facilitate this, we compiled a dataset of 15,508 retrospective knee Magnetic Resonance Imaging (MRI) reports from our Radiology Information System (RIS), focusing on the findings section to predict the radiologist's summary. RESULTS The fine-tuned models outperform the neural network and show superior performance in the zero-shot variant. Specifically, the T5 model achieved a Rouge-L score of 0.638. Based on the radiologist readers' study, the summaries produced by this model were found to be very similar to those produced by a radiologist, with about 70% similarity in fluency and consistency between the T5-generated summaries and the original ones. CONCLUSIONS Technological advances, especially in NLP and LLM, hold great promise for improving and streamlining the summarization of radiological findings, thus providing valuable assistance to radiologists in their work.
Collapse
Affiliation(s)
| | | | | | - Antonio Luna
- MRI Unit, Radiology Department, Health Time, Jaén, Spain.
| |
Collapse
|
3
|
Nakaura T, Ito R, Ueda D, Nozaki T, Fushimi Y, Matsui Y, Yanagawa M, Yamada A, Tsuboyama T, Fujima N, Tatsugami F, Hirata K, Fujita S, Kamagata K, Fujioka T, Kawamura M, Naganawa S. The impact of large language models on radiology: a guide for radiologists on the latest innovations in AI. Jpn J Radiol 2024; 42:685-696. [PMID: 38551772 PMCID: PMC11217134 DOI: 10.1007/s11604-024-01552-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Accepted: 02/21/2024] [Indexed: 07/03/2024]
Abstract
The advent of Deep Learning (DL) has significantly propelled the field of diagnostic radiology forward by enhancing image analysis and interpretation. The introduction of the Transformer architecture, followed by the development of Large Language Models (LLMs), has further revolutionized this domain. LLMs now possess the potential to automate and refine the radiology workflow, extending from report generation to assistance in diagnostics and patient care. The integration of multimodal technology with LLMs could potentially leapfrog these applications to unprecedented levels.However, LLMs come with unresolved challenges such as information hallucinations and biases, which can affect clinical reliability. Despite these issues, the legislative and guideline frameworks have yet to catch up with technological advancements. Radiologists must acquire a thorough understanding of these technologies to leverage LLMs' potential to the fullest while maintaining medical safety and ethics. This review aims to aid in that endeavor.
Collapse
Affiliation(s)
- Takeshi Nakaura
- Department of Central Radiology, Kumamoto University Hospital, Honjo 1-1-1, Kumamoto, 860-8556, Japan.
| | - Rintaro Ito
- Department of Radiology, Nagoya University Graduate School of Medicine, Nagoya, Aichi, Japan
| | - Daiju Ueda
- Department of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, 1‑4‑3 Asahi‑Machi, Abeno‑ku, Osaka, 545‑8585, Japan
| | - Taiki Nozaki
- Department of Radiology, Keio University School of Medicine, Shinjuku‑ku, Tokyo, Japan
| | - Yasutaka Fushimi
- Department of Diagnostic Imaging and Nuclear Medicine, Kyoto University Graduate School of Medicine, Sakyoku, Kyoto, Japan
| | - Yusuke Matsui
- Department of Radiology, Faculty of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University, Kita‑ku, Okayama, Japan
| | - Masahiro Yanagawa
- Department of Radiology, Osaka University Graduate School of Medicine, Suita City, Osaka, Japan
| | - Akira Yamada
- Department of Radiology, Shinshu University School of Medicine, Matsumoto, Nagano, Japan
| | - Takahiro Tsuboyama
- Department of Radiology, Osaka University Graduate School of Medicine, Suita City, Osaka, Japan
| | - Noriyuki Fujima
- Department of Diagnostic and Interventional Radiology, Hokkaido University Hospital, Sapporo, Japan
| | - Fuminari Tatsugami
- Department of Diagnostic Radiology, Hiroshima University, Minami‑ku, Hiroshima, Japan
| | - Kenji Hirata
- Department of Diagnostic Imaging, Graduate School of Medicine, Hokkaido University, Kita‑ku, Sapporo, Hokkaido, Japan
| | - Shohei Fujita
- Department of Radiology, University of Tokyo, Bunkyo‑ku, Tokyo, Japan
| | - Koji Kamagata
- Department of Radiology, Juntendo University Graduate School of Medicine, Bunkyo‑ku, Tokyo, Japan
| | - Tomoyuki Fujioka
- Department of Diagnostic Radiology, Tokyo Medical and Dental University, Bunkyo‑ku, Tokyo, Japan
| | - Mariko Kawamura
- Department of Radiology, Nagoya University Graduate School of Medicine, Nagoya, Aichi, Japan
| | - Shinji Naganawa
- Department of Radiology, Nagoya University Graduate School of Medicine, Nagoya, Aichi, Japan
| |
Collapse
|
4
|
Nakai H, Suman G, Adamo DA, Navin PJ, Bookwalter CA, LeGout JD, Chen FK, Wellnitz CV, Silva AC, Thomas JV, Kawashima A, Fan JW, Froemming AT, Lomas DJ, Humphreys MR, Dora C, Korfiatis P, Takahashi N. Natural language processing pipeline to extract prostate cancer-related information from clinical notes. Eur Radiol 2024:10.1007/s00330-024-10812-6. [PMID: 38842692 DOI: 10.1007/s00330-024-10812-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2024] [Revised: 03/28/2024] [Accepted: 04/10/2024] [Indexed: 06/07/2024]
Abstract
OBJECTIVES To develop an automated pipeline for extracting prostate cancer-related information from clinical notes. MATERIALS AND METHODS This retrospective study included 23,225 patients who underwent prostate MRI between 2017 and 2022. Cancer risk factors (family history of cancer and digital rectal exam findings), pre-MRI prostate pathology, and treatment history of prostate cancer were extracted from free-text clinical notes in English as binary or multi-class classification tasks. Any sentence containing pre-defined keywords was extracted from clinical notes within one year before the MRI. After manually creating sentence-level datasets with ground truth, Bidirectional Encoder Representations from Transformers (BERT)-based sentence-level models were fine-tuned using the extracted sentence as input and the category as output. The patient-level output was determined by compilation of multiple sentence-level outputs using tree-based models. Sentence-level classification performance was evaluated using the area under the receiver operating characteristic curve (AUC) on 15% of the sentence-level dataset (sentence-level test set). The patient-level classification performance was evaluated on the patient-level test set created by radiologists by reviewing the clinical notes of 603 patients. Accuracy and sensitivity were compared between the pipeline and radiologists. RESULTS Sentence-level AUCs were ≥ 0.94. The pipeline showed higher patient-level sensitivity for extracting cancer risk factors (e.g., family history of prostate cancer, 96.5% vs. 77.9%, p < 0.001), but lower accuracy in classifying pre-MRI prostate pathology (92.5% vs. 95.9%, p = 0.002) and treatment history of prostate cancer (95.5% vs. 97.7%, p = 0.03) than radiologists, respectively. CONCLUSION The proposed pipeline showed promising performance, especially for extracting cancer risk factors from patient's clinical notes. CLINICAL RELEVANCE STATEMENT The natural language processing pipeline showed a higher sensitivity for extracting prostate cancer risk factors than radiologists and may help efficiently gather relevant text information when interpreting prostate MRI. KEY POINTS When interpreting prostate MRI, it is necessary to extract prostate cancer-related information from clinical notes. This pipeline extracted the presence of prostate cancer risk factors with higher sensitivity than radiologists. Natural language processing may help radiologists efficiently gather relevant prostate cancer-related text information.
Collapse
Affiliation(s)
| | - Garima Suman
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | - Daniel A Adamo
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | | | | | | | - Frank K Chen
- Department of Radiology, Mayo Clinic, Jacksonville, FL, USA
| | | | - Alvin C Silva
- Department of Radiology, Mayo Clinic, Scottsdale, AZ, USA
| | - John V Thomas
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | | | - Jungwei W Fan
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | | | - Derek J Lomas
- Department of Urology, Mayo Clinic, Rochester, MN, USA
| | | | - Chandler Dora
- Department of Urology, Mayo Clinic, Jacksonville, FL, USA
| | | | | |
Collapse
|
5
|
Codipilly DC, Faghani S, Hagan C, Lewis J, Erickson BJ, Iyer PG. The Evolving Role of Artificial Intelligence in Gastrointestinal Histopathology: An Update. Clin Gastroenterol Hepatol 2024; 22:1170-1180. [PMID: 38154727 DOI: 10.1016/j.cgh.2023.11.044] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Revised: 11/20/2023] [Accepted: 11/21/2023] [Indexed: 12/30/2023]
Abstract
Significant advances in artificial intelligence (AI) over the past decade potentially may lead to dramatic effects on clinical practice. Digitized histology represents an area ripe for AI implementation. We describe several current needs within the world of gastrointestinal histopathology, and outline, using currently studied models, how AI potentially can address them. We also highlight pitfalls as AI makes inroads into clinical practice.
Collapse
Affiliation(s)
- D Chamil Codipilly
- Barrett's Esophagus Unit, Division of Gastroenterology and Hepatology, Mayo Clinic Rochester, Rochester, Minnesota
| | - Shahriar Faghani
- Mayo Artificial Intelligence Laboratory, Department of Radiology, Mayo Clinic, Rochester, Minnesota
| | - Catherine Hagan
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, Minnesota
| | - Jason Lewis
- Department of Pathology, Mayo Clinic, Jacksonville, Florida
| | - Bradley J Erickson
- Mayo Artificial Intelligence Laboratory, Department of Radiology, Mayo Clinic, Rochester, Minnesota
| | - Prasad G Iyer
- Barrett's Esophagus Unit, Division of Gastroenterology and Hepatology, Mayo Clinic Rochester, Rochester, Minnesota.
| |
Collapse
|
6
|
Cavallo JJ, Davis MA. Establishing robust governance of clinical artificial intelligence software - Why radiologists should lead. Clin Imaging 2024; 110:110163. [PMID: 38678765 DOI: 10.1016/j.clinimag.2024.110163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2024] [Revised: 04/12/2024] [Accepted: 04/15/2024] [Indexed: 05/01/2024]
Affiliation(s)
- Joseph J Cavallo
- Yale Department of Radiology, Yale New Haven Hospital, 330 Cedar Street, TE 2-214, New Haven, CT 06520, United States of America.
| | - Melissa A Davis
- Yale Department of Radiology, Yale New Haven Hospital, 330 Cedar Street, TE 2-214, New Haven, CT 06520, United States of America.
| |
Collapse
|
7
|
López-Úbeda P, Martín-Noguerol T, Escartín J, Luna A. Automatic generation of conclusions from neuroradiology MRI reports through natural language processing. Neuroradiology 2024; 66:477-485. [PMID: 38381144 DOI: 10.1007/s00234-024-03312-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2023] [Accepted: 02/08/2024] [Indexed: 02/22/2024]
Abstract
PURPOSE The conclusion section of a radiology report is crucial for summarizing the primary radiological findings in natural language and essential for communicating results to clinicians. However, creating these summaries is time-consuming, repetitive, and prone to variability and errors among different radiologists. To address these issues, we evaluated a fine-tuned Text-To-Text Transfer Transformer (T5) model for abstractive summarization to automatically generate conclusions for neuroradiology MRI reports in a low-resource language. METHODS We retrospectively applied our method to a dataset of 232,425 neuroradiology MRI reports in Spanish. We compared various pre-trained T5 models, including multilingual T5 and those newly adapted for Spanish. For precise evaluation, we employed BLEU, METEOR, ROUGE-L, CIDEr, and cosine similarity metrics alongside expert radiologist assessments. RESULTS The findings are promising, with the models specifically fine-tuned for neuroradiology MRI achieving scores of 0.46, 0.28, 0.52, 2.45, and 0.87 in the BLEU-1, METEOR, ROUGE-L, CIDEr, and cosine similarity metrics, respectively. In the radiological experts' evaluation, they found that in 75% of the cases evaluated, the conclusions generated by the system were as good as or even better than the manually generated conclusions. CONCLUSION The methods demonstrate the potential and effectiveness of customizing state-of-the-art pre-trained models for neuroradiology, yielding automatic MRI report conclusions that nearly match expert quality. Furthermore, these results underscore the importance of designing and pre-training a dedicated language model for radiology report summarization.
Collapse
Affiliation(s)
| | | | - Jorge Escartín
- Diagnostic and Interventional Neuroradiology, HT Medica, Paseo de La Victoria S/N, 14004, Córdoba, Spain
| | - Antonio Luna
- Radiology Department, MRI Unit, HT Medica, Carmelo Torres 2, 23007, Jaén, Spain
| |
Collapse
|
8
|
Martín-Noguerol T, López-Úbeda P, Luna A. Imagine there is no paperwork… it's easy if you try. Br J Radiol 2024; 97:744-746. [PMID: 38335929 PMCID: PMC11027242 DOI: 10.1093/bjr/tqae035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2023] [Revised: 01/11/2024] [Accepted: 02/05/2024] [Indexed: 02/12/2024] Open
Abstract
Artificial Intelligence (AI) applied to radiology is so vast that it provides applications ranging from becoming a complete replacement for radiologists (a potential threat) to an efficient paperwork-saving time assistant (an evident strength). Nowadays, there are AI applications developed to facilitate the diagnostic process of radiologists without directly influencing (or replacing) the proper diagnostic decision step. These tools may help to reduce administrative workload, in different scenarios ranging from assisting in scheduling, study prioritization, or report communication, to helping with patient follow-up, including recommending additional exams. These are just a few of the highly time-consuming tasks that radiologists have to deal with every day in their routine workflow. These tasks hinder the time that radiologists should spend evaluating images and caring for patients, which will have a direct and negative impact on the quality of reports and patient attention, increasing the delay and waiting list of studies pending to be performed and reported. These types of AI applications should help to partially face this worldwide shortage of radiologists.
Collapse
Affiliation(s)
| | | | - Antonio Luna
- MRI Unit, Radiology Department, HT medica, Jaén 23007, Spain
| |
Collapse
|
9
|
Martín-Noguerol T, López-Úbeda P, Pons-Escoda A, Luna A. Natural language processing deep learning models for the differential between high-grade gliomas and metastasis: what if the key is how we report them? Eur Radiol 2024; 34:2113-2120. [PMID: 37665389 DOI: 10.1007/s00330-023-10202-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Revised: 07/10/2023] [Accepted: 07/20/2023] [Indexed: 09/05/2023]
Abstract
OBJECTIVES The differential between high-grade glioma (HGG) and metastasis remains challenging in common radiological practice. We compare different natural language processing (NLP)-based deep learning models to assist radiologists based on data contained in radiology reports. METHODS This retrospective study included 185 MRI reports between 2010 and 2022 from two different institutions. A total of 117 reports were used for the training and 21 were reserved for the validation set, while the rest were used as a test set. A comparison of the performance of different deep learning models for HGG and metastasis classification has been carried out. Specifically, Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory (BiLSTM), a hybrid version of BiLSTM and CNN, and a radiology-specific Bidirectional Encoder Representations from Transformers (RadBERT) model were used. RESULTS For the classification of MRI reports, the CNN network provided the best results among all tested, showing a macro-avg precision of 87.32%, a sensitivity of 87.45%, and an F1 score of 87.23%. In addition, our NLP algorithm detected keywords such as tumor, temporal, and lobe to positively classify a radiological report as HGG or metastasis group. CONCLUSIONS A deep learning model based on CNN enables radiologists to discriminate between HGG and metastasis based on MRI reports with high-precision values. This approach should be considered an additional tool in diagnosing these central nervous system lesions. CLINICAL RELEVANCE STATEMENT The use of our NLP model enables radiologists to differentiate between patients with high-grade glioma and metastasis based on their MRI reports and can be used as an additional tool to the conventional image-based approach for this challenging task. KEY POINTS • Differential between high-grade glioma and metastasis is still challenging in common radiological practice. • Natural language processing (NLP)-based deep learning models can assist radiologists based on data contained in radiology reports. • We have developed and tested a natural language processing model for discriminating between high-grade glioma and metastasis based on MRI reports that show high precision for this task.
Collapse
Affiliation(s)
| | | | - Albert Pons-Escoda
- Radiology Department, Hospital Universitari de Bellvitge, Barcelona, Spain
| | - Antonio Luna
- Radiology Department, MRI Unit, HT Medica, Carmelo Torres 2, 23007, Jaén, Spain
| |
Collapse
|
10
|
Ye Z. The power of the radiologist's last word: can deep learning models accurately differentiate between high-grade gliomas and metastasis through natural language processing on radiology reports? Eur Radiol 2024; 34:2110-2112. [PMID: 37740088 DOI: 10.1007/s00330-023-10245-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2023] [Revised: 08/21/2023] [Accepted: 08/23/2023] [Indexed: 09/24/2023]
Affiliation(s)
- Zezhong Ye
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, USA.
- Department of Radiation Oncology, Dana-Farber Cancer Institute, Brigham and Women's Hospital, Harvard Medical School, Boston, USA.
| |
Collapse
|
11
|
López-Úbeda P, Martín-Noguerol T, Luna A. Automatic classification and prioritisation of actionable BI-RADS categories using natural language processing models. Clin Radiol 2024; 79:e1-e7. [PMID: 37838546 DOI: 10.1016/j.crad.2023.09.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Revised: 08/31/2023] [Accepted: 09/03/2023] [Indexed: 10/16/2023]
Abstract
AIM To facilitate the routine tasks performed by radiologists in their evaluation of breast radiology reports, by enhancing the communication of relevant results to referring physicians via a natural language processing (NLP)-based system to classify and prioritise Breast Imaging Reporting Data System (BI-RADS). MATERIALS AND METHODS A NLP-based system was developed to classify and prioritise BI-RADS categories from breast ultrasound and mammogram reports, with the potential to streamline and speed up the standard procedures that radiologists must follow to evaluate and categorise breast imaging results. BI-RADS category extraction was divided into two specific tasks: (1) multi-label classification of BI-RADS categories (0-6) and (2) classification of high-priority (BI-RADS 0, 3, 4 and 5) and low priority (BI-RADS 1, 2, and 6) reports according to the previous BI-RADS assessment. RESULTS To develop the NLP tool, three different Bidirectional Encoder Representations from Transformers (BERT)-based models (XLM-RoBERTa, BETO, and Bio-BERT-Spanish) were trained and tested on three distinct corpora (containing only breast ultrasound reports, only mammogram reports, or both), and achieved an accuracy of 74.29-77.5% in detecting BI-RADS categories and 88.52-91.02% in prioritising reports. CONCLUSION The system designed can effectively classify all BI-RADS categories present in a single radiology report. In the clinical setting, such an automated tool can assist radiologists in evaluating breast radiology reports and decision-making tasks and enhance the speed of communicating priority BI-RADS reports to referring physicians.
Collapse
Affiliation(s)
- P López-Úbeda
- NLP Department, HT Médica, C. Carmelo Torres 2, 23007 Jaén, Spain.
| | - T Martín-Noguerol
- MRI Unit, Radiology Department, HT Médica, C. Carmelo Torres 2, 23007 Jaén, Spain
| | - A Luna
- MRI Unit, Radiology Department, HT Médica, C. Carmelo Torres 2, 23007 Jaén, Spain
| |
Collapse
|
12
|
Scharp D, Hobensack M, Davoudi A, Topaz M. Natural Language Processing Applied to Clinical Documentation in Post-acute Care Settings: A Scoping Review. J Am Med Dir Assoc 2024; 25:69-83. [PMID: 37838000 PMCID: PMC10792659 DOI: 10.1016/j.jamda.2023.09.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 09/05/2023] [Accepted: 09/07/2023] [Indexed: 10/16/2023]
Abstract
OBJECTIVES To determine the scope of the application of natural language processing to free-text clinical notes in post-acute care and provide a foundation for future natural language processing-based research in these settings. DESIGN Scoping review; reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. SETTING AND PARTICIPANTS Post-acute care (ie, home health care, long-term care, skilled nursing facilities, and inpatient rehabilitation facilities). METHODS PubMed, Cumulative Index of Nursing and Allied Health Literature, and Embase were searched in February 2023. Eligible studies had quantitative designs that used natural language processing applied to clinical documentation in post-acute care settings. The quality of each study was appraised. RESULTS Twenty-one studies were included. Almost all studies were conducted in home health care settings. Most studies extracted data from electronic health records to examine the risk for negative outcomes, including acute care utilization, medication errors, and suicide mortality. About half of the studies did not report age, sex, race, or ethnicity data or use standardized terminologies. Only 8 studies included variables from socio-behavioral domains. Most studies fulfilled all quality appraisal indicators. CONCLUSIONS AND IMPLICATIONS The application of natural language processing is nascent in post-acute care settings. Future research should apply natural language processing using standardized terminologies to leverage free-text clinical notes in post-acute care to promote timely, comprehensive, and equitable care. Natural language processing could be integrated with predictive models to help identify patients who are at risk of negative outcomes. Future research should incorporate socio-behavioral determinants and diverse samples to improve health equity in informatics tools.
Collapse
Affiliation(s)
| | | | - Anahita Davoudi
- VNS Health, Center for Home Care Policy & Research, New York, NY, USA
| | - Maxim Topaz
- Columbia University School of Nursing, New York, NY, USA
| |
Collapse
|
13
|
Chafjiri FMA, Reece L, Voke L, Landschaft A, Clark J, Kimia AA, Loddenkemper T. Natural language processing for identification of refractory status epilepticus in children. Epilepsia 2023; 64:3227-3237. [PMID: 37804085 DOI: 10.1111/epi.17789] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 10/03/2023] [Accepted: 10/03/2023] [Indexed: 10/08/2023]
Abstract
OBJECTIVE Pediatric status epilepticus is one of the most frequent pediatric emergencies, with high mortality and morbidity. Utilizing electronic health records (EHRs) permits analysis of care approaches and disease outcomes at a lower cost than prospective research. However, reviewing EHR manually is time intensive. We aimed to compare refractory status epilepticus (rSE) cases identified by human EHR review with a natural language processing (NLP)-assisted rSE screen followed by a manual review. METHODS We used the NLP screening tool Document Review Tool (DrT) to generate regular expressions, trained a bag-of-words NLP classifier on EHRs from 2017 to 2019, and then tested our algorithm on data from February to December 2012. We compared results from manual review to NLP-assisted search followed by manual review. RESULTS Our algorithm identified 1528 notes in the test set. After removing notes pertaining to the same event by DrT, the user reviewed a total number of 400 notes to find patients with rSE. Within these 400 notes, we identified 31 rSE cases, including 12 new cases not found in manual review, and 19 of the 20 previously identified cases. The NLP-assisted model found 31 of 32 cases, with a sensitivity of 96.88% (95% CI = 82%-99.84%), whereas manual review identified 20 of 32 cases, with a sensitivity of 62.5% (95% CI = 43.75%-78.34%). SIGNIFICANCE DrT provided a highly sensitive model compared to human review and an increase in patient identification through EHRs. The use of DrT is a suitable application of NLP for identifying patients with a history of recent rSE, which ultimately contributes to the implementation of monitoring techniques and treatments in near real time.
Collapse
Affiliation(s)
- Fatemeh Mohammad Alizadeh Chafjiri
- Department of Neurology, Division of Epilepsy and Clinical Neurophysiology, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Latania Reece
- Department of Neurology, Division of Epilepsy and Clinical Neurophysiology, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
- Nexamp, Boston, Massachusetts, USA
| | - Lillian Voke
- Department of Neurology, Division of Epilepsy and Clinical Neurophysiology, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | | | - Justice Clark
- Department of Neurology, Division of Epilepsy and Clinical Neurophysiology, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Amir A Kimia
- Department of Medicine, Division of Emergency Medicine, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
- Connecticut Children's Hospital, Hartford, Connecticut, USA
| | - Tobias Loddenkemper
- Department of Neurology, Division of Epilepsy and Clinical Neurophysiology, Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
14
|
Oh JH, Tannenbaum A, Deasy JO. Improved prediction of drug-induced liver injury literature using natural language processing and machine learning methods. Front Genet 2023; 14:1161047. [PMID: 37529777 PMCID: PMC10390074 DOI: 10.3389/fgene.2023.1161047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 06/29/2023] [Indexed: 08/03/2023] Open
Abstract
Drug-induced liver injury (DILI) is an adverse hepatic drug reaction that can potentially lead to life-threatening liver failure. Previously published work in the scientific literature on DILI has provided valuable insights for the understanding of hepatotoxicity as well as drug development. However, the manual search of scientific literature in PubMed is laborious and time-consuming. Natural language processing (NLP) techniques along with artificial intelligence/machine learning approaches may allow for automatic processing in identifying DILI-related literature, but useful methods are yet to be demonstrated. To address this issue, we have developed an integrated NLP/machine learning classification model to identify DILI-related literature using only paper titles and abstracts. For prediction modeling, we used 14,203 publications provided by the Critical Assessment of Massive Data Analysis (CAMDA) challenge, employing word vectorization techniques in NLP in conjunction with machine learning methods. Classification modeling was performed using 2/3 of the data for training and the remainder for test in internal validation. The best performance was achieved using a linear support vector machine (SVM) model on the combined vectors derived from term frequency-inverse document frequency (TF-IDF) and Word2Vec, resulting in an accuracy of 95.0% and an F1-score of 95.0%. The final SVM model constructed from all 14,203 publications was tested on independent datasets, resulting in accuracies of 92.5%, 96.3%, and 98.3%, and F1-scores of 93.5%, 86.1%, and 75.6% for three test sets (T1-T3). Furthermore, the SVM model was tested on four external validation sets (V1-V4), resulting in accuracies of 92.0%, 96.2%, 98.3%, and 93.1%, and F1-scores of 92.4%, 82.9%, 75.0%, and 93.3%.
Collapse
Affiliation(s)
- Jung Hun Oh
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, United States
| | - Allen Tannenbaum
- Department of Computer Science, Stony Brook University, Stony Brook, NY, United States
- Department of Applied Mathematics and Statistics, Stony Brook University, Stony Brook, NY, United States
| | - Joseph O. Deasy
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, United States
| |
Collapse
|
15
|
Martín-Noguerol T, López-Úbeda P, Luna A. Artificial Intelligence in Radiology: A Fast-Food Versus Slow-Food Question? J Am Coll Radiol 2023:S1546-1440(23)00494-5. [PMID: 37453599 DOI: 10.1016/j.jacr.2023.04.023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2023] [Accepted: 04/06/2023] [Indexed: 07/18/2023]
Affiliation(s)
| | | | - Antonio Luna
- MRI Unit, Radiology Department, HT Medica, Jaén, Spain
| |
Collapse
|
16
|
Sorin V, Klang E. Large language models and the emergence phenomena. Eur J Radiol Open 2023; 10:100494. [PMID: 37325497 PMCID: PMC10265451 DOI: 10.1016/j.ejro.2023.100494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Revised: 05/15/2023] [Accepted: 05/31/2023] [Indexed: 06/17/2023] Open
Abstract
This perspective explores the potential of emergence phenomena in large language models (LLMs) to transform data management and analysis in radiology. We provide a concise explanation of LLMs, define the concept of emergence in machine learning, offer examples of potential applications within the radiology field, and discuss risks and limitations. Our goal is to encourage radiologists to recognize and prepare for the impact this technology may have on radiology and medicine in the near future.
Collapse
Affiliation(s)
- Vera Sorin
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, affiliated to the Sackler School of Medicine, Tel-Aviv University, Israel
- DeepVision Lab, Sheba Medical Center, Tel Hashomer, Israel
| | - Eyal Klang
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, affiliated to the Sackler School of Medicine, Tel-Aviv University, Israel
- DeepVision Lab, Sheba Medical Center, Tel Hashomer, Israel
- Sami Sagol AI Hub, ARC, Sheba Medical Center, Israel
| |
Collapse
|
17
|
Zimmerman A. A Ghostwriter for the Masses: ChatGPT and the Future of Writing. Ann Surg Oncol 2023; 30:3170-3173. [PMID: 37029868 DOI: 10.1245/s10434-023-13436-0] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Accepted: 02/12/2023] [Indexed: 04/09/2023]
|
18
|
Artificial Intelligence Applied to Improve Scientific Reviews: The Antibacterial Activity of Cistus Plants as Proof of Concept. Antibiotics (Basel) 2023; 12:antibiotics12020327. [PMID: 36830239 PMCID: PMC9952093 DOI: 10.3390/antibiotics12020327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Revised: 01/29/2023] [Accepted: 02/02/2023] [Indexed: 02/08/2023] Open
Abstract
Reviews have traditionally been based on extensive searches of the available bibliography on the topic of interest. However, this approach is frequently influenced by the authors' background, leading to possible selection bias. Artificial intelligence applied to natural language processing (NLP) is a powerful tool that can be used for systematic reviews by speeding up the process and providing more objective results, but its use in scientific literature reviews is still scarce. This manuscript addresses this challenge by developing a reproducible tool that can be used to develop objective reviews on almost every topic. This tool has been used to review the antibacterial activity of Cistus genus plant extracts as proof of concept, providing a comprehensive and objective state of the art on this topic based on the analysis of 1601 research manuscripts and 136 patents. Data were processed using a publicly available Jupyter Notebook in Google Collaboratory here. NLP, when applied to the study of antibacterial activity of Cistus plants, is able to recover the main scientific manuscripts and patents related to the topic, avoiding any biases. The NLP-assisted literature review reveals that C. creticus and C. monspeliensis are the first and second most studied Cistus species respectively. Leaves and fruits are the most commonly used plant parts and methanol, followed by butanol and water, the most widely used solvents to prepare plant extracts. Furthermore, Staphylococcus. aureus followed by Bacillus. cereus are the most studied bacterial species, which are also the most susceptible bacteria in all studied assays. This new tool aims to change the actual paradigm of the review of scientific literature to make the process more efficient, reliable, and reproducible, according to Open Science standards.
Collapse
|