1
|
Nerella S, Bandyopadhyay S, Zhang J, Contreras M, Siegel S, Bumin A, Silva B, Sena J, Shickel B, Bihorac A, Khezeli K, Rashidi P. Transformers and large language models in healthcare: A review. Artif Intell Med 2024; 154:102900. [PMID: 38878555 DOI: 10.1016/j.artmed.2024.102900] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Revised: 05/28/2024] [Accepted: 05/30/2024] [Indexed: 08/09/2024]
Abstract
With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning architecture initially developed to solve general-purpose Natural Language Processing (NLP) tasks and has subsequently been adapted in many fields, including healthcare. In this survey paper, we provide an overview of how this architecture has been adopted to analyze various forms of healthcare data, including clinical NLP, medical imaging, structured Electronic Health Records (EHR), social media, bio-physiological signals, biomolecular sequences. Furthermore, which have also include the articles that used the transformer architecture for generating surgical instructions and predicting adverse outcomes after surgeries under the umbrella of critical care. Under diverse settings, these models have been used for clinical diagnosis, report generation, data reconstruction, and drug/protein synthesis. Finally, we also discuss the benefits and limitations of using transformers in healthcare and examine issues such as computational cost, model interpretability, fairness, alignment with human values, ethical implications, and environmental impact.
Collapse
Affiliation(s)
- Subhash Nerella
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | | | - Jiaqing Zhang
- Department of Electrical and Computer Engineering, University of Florida, Gainesville, United States
| | - Miguel Contreras
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | - Scott Siegel
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | - Aysegul Bumin
- Department of Computer and Information Science and Engineering, University of Florida, Gainesville, United States
| | - Brandon Silva
- Department of Computer and Information Science and Engineering, University of Florida, Gainesville, United States
| | - Jessica Sena
- Department Of Computer Science, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil
| | - Benjamin Shickel
- Department of Medicine, University of Florida, Gainesville, United States
| | - Azra Bihorac
- Department of Medicine, University of Florida, Gainesville, United States
| | - Kia Khezeli
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | - Parisa Rashidi
- Department of Biomedical Engineering, University of Florida, Gainesville, United States.
| |
Collapse
|
2
|
Beattie J, Neufeld S, Yang D, Chukwuma C, Gul A, Desai N, Jiang S, Dohopolski M. Utilizing Large Language Models for Enhanced Clinical Trial Matching: A Study on Automation in Patient Screening. Cureus 2024; 16:e60044. [PMID: 38854210 PMCID: PMC11162699 DOI: 10.7759/cureus.60044] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/10/2024] [Indexed: 06/11/2024] Open
Abstract
Background Clinical trial matching, essential for advancing medical research, involves detailed screening of potential participants to ensure alignment with specific trial requirements. Research staff face challenges due to the high volume of eligible patients and the complexity of varying eligibility criteria. The traditional manual process, both time-consuming and error-prone, often leads to missed opportunities. Recently, large language models (LLMs), specifically generative pre-trained transformers (GPTs), have become impressive and impactful tools. Utilizing such tools from artificial intelligence (AI) and natural language processing (NLP) may enhance the accuracy and efficiency of this process through automated patient screening against established criteria. Methods Utilizing data from the National NLP Clinical Challenges (n2c2) 2018 Challenge, we utilized 202 longitudinal patient records. These records were annotated by medical professionals and evaluated against 13 selection criteria encompassing various health assessments. Our approach involved embedding medical documents into a vector database to determine relevant document sections and then using an LLM (OpenAI's GPT-3.5 Turbo and GPT-4) in tandem with structured and chain-of-thought prompting techniques for systematic document assessment against the criteria. Misclassified criteria were also examined to identify classification challenges. Results This study achieved an accuracy of 0.81, sensitivity of 0.80, specificity of 0.82, and a micro F1 score of 0.79 using GPT-3.5 Turbo, and an accuracy of 0.87, sensitivity of 0.85, specificity of 0.89, and micro F1 score of 0.86 using GPT-4. Notably, some criteria in the ground truth appeared mislabeled, an issue we couldn't explore further due to insufficient label generation guidelines on the website. Conclusion Our findings underscore the potential of AI and NLP technologies, including LLMs, in the clinical trial matching process. The study demonstrated strong capabilities in identifying eligible patients and minimizing false inclusions. Such automated systems promise to alleviate the workload of research staff and improve clinical trial enrollment, thus accelerating the process and enhancing the overall feasibility of clinical research. Further work is needed to determine the potential of this approach when implemented on real clinical data.
Collapse
Affiliation(s)
- Jacob Beattie
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| | - Sarah Neufeld
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| | - Daniel Yang
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| | - Christian Chukwuma
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| | - Ahmed Gul
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| | - Neil Desai
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| | - Steve Jiang
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| | - Michael Dohopolski
- Department of Radiation Oncology, University of Texas (UT) Southwestern Medical Center, Dallas, USA
| |
Collapse
|
3
|
Jin Q, Wang Z, Floudas CS, Chen F, Gong C, Bracken-Clarke D, Xue E, Yang Y, Sun J, Lu Z. Matching Patients to Clinical Trials with Large Language Models. ARXIV 2024:arXiv:2307.15051v4. [PMID: 37576126 PMCID: PMC10418514] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Subscribe] [Scholar Register] [Indexed: 08/15/2023]
Abstract
Clinical trials are often hindered by the challenge of patient recruitment. In this work, we introduce TrialGPT, a first-of-its-kind large language model (LLM) framework to assist patient-to-trial matching. Given a patient note, TrialGPT predicts the patient's eligibility on a criterion-by-criterion basis and then consolidates these predictions to assess the patient's eligibility for the target trial. We evaluate the trial-level prediction performance of TrialGPT on three publicly available cohorts of 184 patients with over 18,000 trial annotations. We also engaged three physicians to label over 1,000 patient-criterion pairs to assess its criterion-level prediction accuracy. Experimental results show that TrialGPT achieves a criterion-level accuracy of 87.3% with faithful explanations, close to the expert performance (88.7%-90.0%). The aggregated TrialGPT scores are highly correlated with human eligibility judgments, and they outperform the best-competing models by 32.6% to 57.2% in ranking and excluding clinical trials. Furthermore, our user study reveals that TrialGPT can significantly reduce the screening time (by 42.6%) in a real-life clinical trial matching task. These results and analyses have demonstrated promising opportunities for clinical trial matching with LLMs such as TrialGPT.
Collapse
|
4
|
Robertshaw H, Karstensen L, Jackson B, Sadati H, Rhode K, Ourselin S, Granados A, Booth TC. Artificial intelligence in the autonomous navigation of endovascular interventions: a systematic review. Front Hum Neurosci 2023; 17:1239374. [PMID: 37600553 PMCID: PMC10438983 DOI: 10.3389/fnhum.2023.1239374] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 07/20/2023] [Indexed: 08/22/2023] Open
Abstract
Background Autonomous navigation of catheters and guidewires in endovascular interventional surgery can decrease operation times, improve decision-making during surgery, and reduce operator radiation exposure while increasing access to treatment. Objective To determine from recent literature, through a systematic review, the impact, challenges, and opportunities artificial intelligence (AI) has for the autonomous navigation of catheters and guidewires for endovascular interventions. Methods PubMed and IEEEXplore databases were searched to identify reports of AI applied to autonomous navigation methods in endovascular interventional surgery. Eligibility criteria included studies investigating the use of AI in enabling the autonomous navigation of catheters/guidewires in endovascular interventions. Following Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), articles were assessed using Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2). PROSPERO: CRD42023392259. Results Four hundred and sixty-two studies fulfilled the search criteria, of which 14 studies were included for analysis. Reinforcement learning (RL) (9/14, 64%) and learning from expert demonstration (7/14, 50%) were used as data-driven models for autonomous navigation. These studies evaluated models on physical phantoms (10/14, 71%) and in-silico (4/14, 29%) models. Experiments within or around the blood vessels of the heart were reported by the majority of studies (10/14, 71%), while non-anatomical vessel platforms "idealized" for simple navigation were used in three studies (3/14, 21%), and the porcine liver venous system in one study. We observed that risk of bias and poor generalizability were present across studies. No procedures were performed on patients in any of the studies reviewed. Moreover, all studies were limited due to the lack of patient selection criteria, reference standards, and reproducibility, which resulted in a low level of evidence for clinical translation. Conclusion Despite the potential benefits of AI applied to autonomous navigation of endovascular interventions, the field is in an experimental proof-of-concept stage, with a technology readiness level of 3. We highlight that reference standards with well-identified performance metrics are crucial to allow for comparisons of data-driven algorithms proposed in the years to come. Systematic review registration identifier: CRD42023392259.
Collapse
Affiliation(s)
- Harry Robertshaw
- School of Biomedical Engineering & Imaging Sciences, Kings College London, London, United Kingdom
| | - Lennart Karstensen
- Fraunhofer IPA, Mannheim, Germany
- AIBE, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
| | - Benjamin Jackson
- School of Biomedical Engineering & Imaging Sciences, Kings College London, London, United Kingdom
| | - Hadi Sadati
- School of Biomedical Engineering & Imaging Sciences, Kings College London, London, United Kingdom
| | - Kawal Rhode
- School of Biomedical Engineering & Imaging Sciences, Kings College London, London, United Kingdom
| | - Sebastien Ourselin
- School of Biomedical Engineering & Imaging Sciences, Kings College London, London, United Kingdom
| | - Alejandro Granados
- School of Biomedical Engineering & Imaging Sciences, Kings College London, London, United Kingdom
| | - Thomas C. Booth
- School of Biomedical Engineering & Imaging Sciences, Kings College London, London, United Kingdom
- Department of Neuroradiology, Kings College Hospital, London, United Kingdom
| |
Collapse
|
5
|
Tun PP, Luo J, Xie J, Wibowo S, Hao C. Automatic assessment of patient eligibility by utilizing NLP and rule-based analysis. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38082656 DOI: 10.1109/embc40787.2023.10340494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Assessment of patient eligibility is an essential process in the clinical trial but there are a lot of manual processes involved. Natural Language Processing (NLP) is a promising technique to automate analysing of the massive volume of Electronic Medical Records (EMRs) hence it can assist in the assessment of patient eligibility, especially in clinical trials that require complex inclusion/exclusion criteria. In this paper, we proposed a hybrid model which utilized both rule-based and NLP technologies to automate the assessment of patient eligibility. The result showed that the hybrid model had a better trade-off between sensitivity and precision compared to the rule-based model and NLP similarity model. Moreover, the accuracy of the hybrid model was validated on the larger dataset and it reached an accuracy of 87.3%. Therefore, this technique potentially can improve the efficiency of patient recruitment by eliminating the manual processes that involve in the assessment of patient eligibility.
Collapse
|
6
|
Idnay B, Fang Y, Dreisbach C, Marder K, Weng C, Schnall R. Clinical research staff perceptions on a natural language processing-driven tool for eligibility prescreening: An iterative usability assessment. Int J Med Inform 2023; 171:104985. [PMID: 36638583 PMCID: PMC9912278 DOI: 10.1016/j.ijmedinf.2023.104985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2022] [Revised: 12/30/2022] [Accepted: 01/02/2023] [Indexed: 01/07/2023]
Abstract
BACKGROUND Participant recruitment is a barrier to successful clinical research. One strategy to improve recruitment is to conduct eligibility prescreening, a resource-intensive process where clinical research staff manually reviews electronic health records data to identify potentially eligible patients. Criteria2Query (C2Q) was developed to address this problem by capitalizing on natural language processing to generate queries to identify eligible participants from clinical databases semi-autonomously. OBJECTIVE We examined the clinical research staff's perceived usability of C2Q for clinical research eligibility prescreening. METHODS Twenty clinical research staff evaluated the usability of C2Q using a cognitive walkthrough with a think-aloud protocol and a Post-Study System Usability Questionnaire. On-screen activity and audio were recorded and transcribed. After every-five evaluators completed an evaluation, usability problems were rated by informatics experts and prioritized for system refinement. There were four iterations of system refinement based on the evaluation feedback. Guided by the Organizational Framework for Intuitive Human-computer Interaction, we performed a directed deductive content analysis of the verbatim transcriptions. RESULTS Evaluators aged from 24 to 46 years old (33.8; SD: 7.32) demonstrated high computer literacy (6.36; SD:0.17); female (75 %), White (35 %), and clinical research coordinators (45 %). C2Q demonstrated high usability during the final cycle (2.26 out of 7 [lower scores are better], SD: 0.74). The number of unique usability issues decreased after each refinement. Fourteen subthemes emerged from three themes: seeking user goals, performing well-learned tasks, and determining what to do next. CONCLUSIONS The cognitive walkthrough with a think-aloud protocol informed iterative system refinement and demonstrated the usability of C2Q by clinical research staff. Key recommendations for system development and implementation include improving system intuitiveness and overall user experience through comprehensive consideration of user needs and requirements for task completion.
Collapse
Affiliation(s)
- Betina Idnay
- Columbia University, School of Nursing, New York, NY, USA; Columbia University, Department of Neurology, New York, NY, USA; Columbia University, Department of Biomedical Informatics, New York, NY, USA.
| | - Yilu Fang
- Columbia University, Department of Biomedical Informatics, New York, NY, USA
| | | | - Karen Marder
- Columbia University, Department of Neurology, New York, NY, USA
| | - Chunhua Weng
- Columbia University, Department of Biomedical Informatics, New York, NY, USA
| | - Rebecca Schnall
- Columbia University, School of Nursing, New York, NY, USA; Columbia University, Mailman School of Public Health, Department of Population and Family Health, New York, NY, USA
| |
Collapse
|
7
|
Yang S, Varghese P, Stephenson E, Tu K, Gronsbell J. Machine learning approaches for electronic health records phenotyping: a methodical review. J Am Med Inform Assoc 2023; 30:367-381. [PMID: 36413056 PMCID: PMC9846699 DOI: 10.1093/jamia/ocac216] [Citation(s) in RCA: 23] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 09/27/2022] [Accepted: 10/27/2022] [Indexed: 11/23/2022] Open
Abstract
OBJECTIVE Accurate and rapid phenotyping is a prerequisite to leveraging electronic health records for biomedical research. While early phenotyping relied on rule-based algorithms curated by experts, machine learning (ML) approaches have emerged as an alternative to improve scalability across phenotypes and healthcare settings. This study evaluates ML-based phenotyping with respect to (1) the data sources used, (2) the phenotypes considered, (3) the methods applied, and (4) the reporting and evaluation methods used. MATERIALS AND METHODS We searched PubMed and Web of Science for articles published between 2018 and 2022. After screening 850 articles, we recorded 37 variables on 100 studies. RESULTS Most studies utilized data from a single institution and included information in clinical notes. Although chronic conditions were most commonly considered, ML also enabled the characterization of nuanced phenotypes such as social determinants of health. Supervised deep learning was the most popular ML paradigm, while semi-supervised and weakly supervised learning were applied to expedite algorithm development and unsupervised learning to facilitate phenotype discovery. ML approaches did not uniformly outperform rule-based algorithms, but deep learning offered a marginal improvement over traditional ML for many conditions. DISCUSSION Despite the progress in ML-based phenotyping, most articles focused on binary phenotypes and few articles evaluated external validity or used multi-institution data. Study settings were infrequently reported and analytic code was rarely released. CONCLUSION Continued research in ML-based phenotyping is warranted, with emphasis on characterizing nuanced phenotypes, establishing reporting and evaluation standards, and developing methods to accommodate misclassified phenotypes due to algorithm errors in downstream applications.
Collapse
Affiliation(s)
- Siyue Yang
- Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada
| | | | - Ellen Stephenson
- Department of Family & Community Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Karen Tu
- Department of Family & Community Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Jessica Gronsbell
- Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada
- Department of Family & Community Medicine, University of Toronto, Toronto, Ontario, Canada
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
8
|
Lentzen M, Madan S, Lage-Rupprecht V, Kühnel L, Fluck J, Jacobs M, Mittermaier M, Witzenrath M, Brunecker P, Hofmann-Apitius M, Weber J, Fröhlich H. Critical assessment of transformer-based AI models for German clinical notes. JAMIA Open 2022; 5:ooac087. [PMID: 36380848 PMCID: PMC9663939 DOI: 10.1093/jamiaopen/ooac087] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2022] [Revised: 10/02/2022] [Accepted: 10/25/2022] [Indexed: 11/17/2022] Open
Abstract
Objective Healthcare data such as clinical notes are primarily recorded in an unstructured manner. If adequately translated into structured data, they can be utilized for health economics and set the groundwork for better individualized patient care. To structure clinical notes, deep-learning methods, particularly transformer-based models like Bidirectional Encoder Representations from Transformers (BERT), have recently received much attention. Currently, biomedical applications are primarily focused on the English language. While general-purpose German-language models such as GermanBERT and GottBERT have been published, adaptations for biomedical data are unavailable. This study evaluated the suitability of existing and novel transformer-based models for the German biomedical and clinical domain. Materials and Methods We used 8 transformer-based models and pre-trained 3 new models on a newly generated biomedical corpus, and systematically compared them with each other. We annotated a new dataset of clinical notes and used it with 4 other corpora (BRONCO150, CLEF eHealth 2019 Task 1, GGPONC, and JSynCC) to perform named entity recognition (NER) and document classification tasks. Results General-purpose language models can be used effectively for biomedical and clinical natural language processing (NLP) tasks, still, our newly trained BioGottBERT model outperformed GottBERT on both clinical NER tasks. However, training new biomedical models from scratch proved ineffective. Discussion The domain-adaptation strategy’s potential is currently limited due to a lack of pre-training data. Since general-purpose language models are only marginally inferior to domain-specific models, both options are suitable for developing German-language biomedical applications. Conclusion General-purpose language models perform remarkably well on biomedical and clinical NLP tasks. If larger corpora become available in the future, domain-adapting these models may improve performances.
Collapse
Affiliation(s)
- Manuel Lentzen
- Department of Bioinformatics, Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Schloss Birlinghoven, Sankt Augustin, Germany,Bonn-Aachen International Center for Information Technology (B-IT), University of Bonn, Bonn, Germany
| | - Sumit Madan
- Department of Bioinformatics, Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Schloss Birlinghoven, Sankt Augustin, Germany,Institute of Computer Science, University of Bonn, Bonn, Germany
| | - Vanessa Lage-Rupprecht
- Department of Bioinformatics, Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Schloss Birlinghoven, Sankt Augustin, Germany
| | - Lisa Kühnel
- Knowledge Management, ZB MED – Information Centre for Life Sciences, Cologne, Germany,Graduate School DILS, Bielefeld Institute for Bioinformatics Infrastructure (BIBI), Faculty of Technology, Bielefeld University, Bielefeld, Germany
| | - Juliane Fluck
- Knowledge Management, ZB MED – Information Centre for Life Sciences, Cologne, Germany,The Agricultural Faculty, University of Bonn, Bonn, Germany
| | - Marc Jacobs
- Department of Bioinformatics, Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Schloss Birlinghoven, Sankt Augustin, Germany
| | - Mirja Mittermaier
- Department of Infectious Diseases and Respiratory Medicine, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin, Germany,Berlin Institute of Health (BIH) at Charité – Universitätsmedizin Berlin, Berlin, Germany
| | - Martin Witzenrath
- Department of Infectious Diseases and Respiratory Medicine, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin, Germany,German Center for Lung Research (DZL), Partner Site Charité, Berlin, Germany
| | - Peter Brunecker
- Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Core Facility Research IT, Berlin, Germany
| | - Martin Hofmann-Apitius
- Department of Bioinformatics, Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Schloss Birlinghoven, Sankt Augustin, Germany,Bonn-Aachen International Center for Information Technology (B-IT), University of Bonn, Bonn, Germany
| | - Joachim Weber
- Berlin Institute of Health (BIH) at Charité – Universitätsmedizin Berlin, Berlin, Germany,Charité – Universitätsmedizin Berlin, Center for Stroke Research Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany,Department of Neurology, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Holger Fröhlich
- Corresponding Author: Prof. Dr. Holger Fröhlich, Department of Bioinformatics, Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Schloss Birlinghoven, 53757 Sankt Augustin, Germany;
| |
Collapse
|
9
|
Yu Z, Yang X, Sweeting GL, Ma Y, Stolte SE, Fang R, Wu Y. Identify diabetic retinopathy-related clinical concepts and their attributes using transformer-based natural language processing methods. BMC Med Inform Decis Mak 2022; 22:255. [PMID: 36167551 PMCID: PMC9513862 DOI: 10.1186/s12911-022-01996-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2022] [Accepted: 09/14/2022] [Indexed: 11/16/2022] Open
Abstract
BACKGROUND Diabetic retinopathy (DR) is a leading cause of blindness in American adults. If detected, DR can be treated to prevent further damage causing blindness. There is an increasing interest in developing artificial intelligence (AI) technologies to help detect DR using electronic health records. The lesion-related information documented in fundus image reports is a valuable resource that could help diagnoses of DR in clinical decision support systems. However, most studies for AI-based DR diagnoses are mainly based on medical images; there is limited studies to explore the lesion-related information captured in the free text image reports. METHODS In this study, we examined two state-of-the-art transformer-based natural language processing (NLP) models, including BERT and RoBERTa, compared them with a recurrent neural network implemented using Long short-term memory (LSTM) to extract DR-related concepts from clinical narratives. We identified four different categories of DR-related clinical concepts including lesions, eye parts, laterality, and severity, developed annotation guidelines, annotated a DR-corpus of 536 image reports, and developed transformer-based NLP models for clinical concept extraction and relation extraction. We also examined the relation extraction under two settings including 'gold-standard' setting-where gold-standard concepts were used-and end-to-end setting. RESULTS For concept extraction, the BERT model pretrained with the MIMIC III dataset achieve the best performance (0.9503 and 0.9645 for strict/lenient evaluation). For relation extraction, BERT model pretrained using general English text achieved the best strict/lenient F1-score of 0.9316. The end-to-end system, BERT_general_e2e, achieved the best strict/lenient F1-score of 0.8578 and 0.8881, respectively. Another end-to-end system based on the RoBERTa architecture, RoBERTa_general_e2e, also achieved the same performance as BERT_general_e2e in strict scores. CONCLUSIONS This study demonstrated the efficiency of transformer-based NLP models for clinical concept extraction and relation extraction. Our results show that it's necessary to pretrain transformer models using clinical text to optimize the performance for clinical concept extraction. Whereas, for relation extraction, transformers pretrained using general English text perform better.
Collapse
Affiliation(s)
- Zehao Yu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL USA
| | - Xi Yang
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL USA
| | - Gianna L. Sweeting
- Department of Biomedical Engineering, College of Engineering, University of Florida, Gainesville, FL USA
| | - Yinghan Ma
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL USA
| | - Skylar E. Stolte
- Department of Biomedical Engineering, College of Engineering, University of Florida, Gainesville, FL USA
| | - Ruogu Fang
- Department of Biomedical Engineering, College of Engineering, University of Florida, Gainesville, FL USA
| | - Yonghui Wu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL USA
| |
Collapse
|
10
|
Gao Y, Dligach D, Christensen L, Tesch S, Laffin R, Xu D, Miller T, Uzuner O, Churpek MM, Afshar M. A scoping review of publicly available language tasks in clinical natural language processing. J Am Med Inform Assoc 2022; 29:1797-1806. [PMID: 35923088 PMCID: PMC9471718 DOI: 10.1093/jamia/ocac127] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2021] [Revised: 06/16/2022] [Accepted: 08/01/2022] [Indexed: 11/14/2022] Open
Abstract
OBJECTIVE To provide a scoping review of papers on clinical natural language processing (NLP) shared tasks that use publicly available electronic health record data from a cohort of patients. MATERIALS AND METHODS We searched 6 databases, including biomedical research and computer science literature databases. A round of title/abstract screening and full-text screening were conducted by 2 reviewers. Our method followed the PRISMA-ScR guidelines. RESULTS A total of 35 papers with 48 clinical NLP tasks met inclusion criteria between 2007 and 2021. We categorized the tasks by the type of NLP problems, including named entity recognition, summarization, and other NLP tasks. Some tasks were introduced as potential clinical decision support applications, such as substance abuse detection, and phenotyping. We summarized the tasks by publication venue and dataset type. DISCUSSION The breadth of clinical NLP tasks continues to grow as the field of NLP evolves with advancements in language systems. However, gaps exist with divergent interests between the general domain NLP community and the clinical informatics community for task motivation and design, and in generalizability of the data sources. We also identified issues in data preparation. CONCLUSION The existing clinical NLP tasks cover a wide range of topics and the field is expected to grow and attract more attention from both general domain NLP and clinical informatics community. We encourage future work to incorporate multidisciplinary collaboration, reporting transparency, and standardization in data preparation. We provide a listing of all the shared task papers and datasets from this review in a GitLab repository.
Collapse
Affiliation(s)
- Yanjun Gao
- ICU Data Science Lab, Department of Medicine, School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA
| | - Dmitriy Dligach
- Department of Computer Science, Loyola University Chicago, Chicago, Illinois, USA
| | - Leslie Christensen
- School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA
| | - Samuel Tesch
- School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA
| | - Ryan Laffin
- School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA
| | - Dongfang Xu
- Computational Health Informatics Program, Boston Children's Hospital, Harvard University, Boston, Massachusetts, USA
| | - Timothy Miller
- Computational Health Informatics Program, Boston Children's Hospital, Harvard University, Boston, Massachusetts, USA
| | - Ozlem Uzuner
- Department of Information Sciences and Technology, George Mason University, Fairfax, Virginia, USA
| | - Matthew M Churpek
- ICU Data Science Lab, Department of Medicine, School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA
| | - Majid Afshar
- ICU Data Science Lab, Department of Medicine, School of Medicine and Public Health, University of Wisconsin, Madison, Wisconsin, USA
| |
Collapse
|
11
|
Liu J, Capurro D, Nguyen A, Verspoor K. "Note Bloat" impacts deep learning-based NLP models for clinical prediction tasks. J Biomed Inform 2022; 133:104149. [PMID: 35878821 DOI: 10.1016/j.jbi.2022.104149] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Revised: 05/28/2022] [Accepted: 07/19/2022] [Indexed: 10/17/2022]
Abstract
One unintended consequence of the Electronic Health Records (EHR) implementation is the overuse of content-importing technology, such as copy-and-paste, that creates "bloated" notes containing large amounts of textual redundancy. Despite the rising interest in applying machine learning models to learn from real-patient data, it is unclear how the phenomenon of note bloat might affect the Natural Language Processing (NLP) models derived from these notes. Therefore, in this work we examine the impact of redundancy on deep learning-based NLP models, considering four clinical prediction tasks using a publicly available EHR database. We applied two deduplication methods to the hospital notes, identifying large quantities of redundancy, and found that removing the redundancy usually has little negative impact on downstream performances, and can in certain circumstances assist models to achieve significantly better results. We also showed it is possible to attack model predictions by simply adding note duplicates, causing changes of correct predictions made by trained models into wrong predictions. In conclusion, we demonstrated that EHR text redundancy substantively affects NLP models for clinical prediction tasks, showing that the awareness of clinical contexts and robust modeling methods are important to create effective and reliable NLP systems in healthcare contexts.
Collapse
Affiliation(s)
- Jinghui Liu
- School of Computing and Information Systems, The University of Melbourne, Victoria, Australia; Australian e-Health Research Centre, CSIRO, Brisbane, Australia.
| | - Daniel Capurro
- School of Computing and Information Systems, The University of Melbourne, Victoria, Australia; Centre for Digital Transformation of Health, Melbourne Medical School, The University of Melbourne, Victoria, Australia.
| | - Anthony Nguyen
- Australian e-Health Research Centre, CSIRO, Brisbane, Australia.
| | - Karin Verspoor
- School of Computing and Information Systems, The University of Melbourne, Victoria, Australia; Centre for Digital Transformation of Health, Melbourne Medical School, The University of Melbourne, Victoria, Australia; School of Computing Technologies, RMIT University, Victoria, Australia.
| |
Collapse
|
12
|
Fang Y, Idnay B, Sun Y, Liu H, Chen Z, Marder K, Xu H, Schnall R, Weng C. Combining human and machine intelligence for clinical trial eligibility querying. J Am Med Inform Assoc 2022; 29:1161-1171. [PMID: 35426943 PMCID: PMC9196697 DOI: 10.1093/jamia/ocac051] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2022] [Accepted: 03/29/2022] [Indexed: 11/13/2022] Open
Abstract
OBJECTIVE To combine machine efficiency and human intelligence for converting complex clinical trial eligibility criteria text into cohort queries. MATERIALS AND METHODS Criteria2Query (C2Q) 2.0 was developed to enable real-time user intervention for criteria selection and simplification, parsing error correction, and concept mapping. The accuracy, precision, recall, and F1 score of enhanced modules for negation scope detection, temporal and value normalization were evaluated using a previously curated gold standard, the annotated eligibility criteria of 1010 COVID-19 clinical trials. The usability and usefulness were evaluated by 10 research coordinators in a task-oriented usability evaluation using 5 Alzheimer's disease trials. Data were collected by user interaction logging, a demographic questionnaire, the Health Information Technology Usability Evaluation Scale (Health-ITUES), and a feature-specific questionnaire. RESULTS The accuracies of negation scope detection, temporal and value normalization were 0.924, 0.916, and 0.966, respectively. C2Q 2.0 achieved a moderate usability score (3.84 out of 5) and a high learnability score (4.54 out of 5). On average, 9.9 modifications were made for a clinical study. Experienced researchers made more modifications than novice researchers. The most frequent modification was deletion (5.35 per study). Furthermore, the evaluators favored cohort queries resulting from modifications (score 4.1 out of 5) and the user engagement features (score 4.3 out of 5). DISCUSSION AND CONCLUSION Features to engage domain experts and to overcome the limitations in automated machine output are shown to be useful and user-friendly. We concluded that human-computer collaboration is key to improving the adoption and user-friendliness of natural language processing.
Collapse
Affiliation(s)
- Yilu Fang
- Department of Biomedical Informatics, Columbia University, New York, New York, USA
| | - Betina Idnay
- School of Nursing, Columbia University, New York, New York, USA.,Department of Neurology, Columbia University, New York, New York, USA
| | - Yingcheng Sun
- Department of Biomedical Informatics, Columbia University, New York, New York, USA
| | - Hao Liu
- Department of Biomedical Informatics, Columbia University, New York, New York, USA
| | - Zhehuan Chen
- Department of Biomedical Informatics, Columbia University, New York, New York, USA
| | - Karen Marder
- Department of Neurology, Columbia University, New York, New York, USA
| | - Hua Xu
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Rebecca Schnall
- School of Nursing, Columbia University, New York, New York, USA.,Heilbrunn Department of Population and Family Health, Mailman School of Public Health, Columbia University, New York, New York, USA
| | - Chunhua Weng
- Department of Biomedical Informatics, Columbia University, New York, New York, USA
| |
Collapse
|
13
|
Li M, Cai H, Nan S, Li J, Lu X, Duan H. A Patient-Screening Tool for Clinical Research Based on Electronic Health Records Using OpenEHR: Development Study. JMIR Med Inform 2021; 9:e33192. [PMID: 34673526 PMCID: PMC8569542 DOI: 10.2196/33192] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2021] [Revised: 09/27/2021] [Accepted: 09/27/2021] [Indexed: 11/28/2022] Open
Abstract
Background The widespread adoption of electronic health records (EHRs) has facilitated the secondary use of EHR data for clinical research. However, screening eligible patients from EHRs is a challenging task. The concepts in eligibility criteria are not completely matched with EHRs, especially derived concepts. The lack of high-level expression of Structured Query Language (SQL) makes it difficult and time consuming to express them. The openEHR Expression Language (EL) as a domain-specific language based on clinical information models shows promise to represent complex eligibility criteria. Objective The study aims to develop a patient-screening tool based on EHRs for clinical research using openEHR to solve concept mismatch and improve query performance. Methods A patient-screening tool based on EHRs using openEHR was proposed. It uses the advantages of information models and EL in openEHR to provide high-level expressions and improve query performance. First, openEHR archetypes and templates were chosen to define concepts called simple concepts directly from EHRs. Second, openEHR EL was used to generate derived concepts by combining simple concepts and constraints. Third, a hierarchical index corresponding to archetypes in Elasticsearch (ES) was generated to improve query performance for subqueries and join queries related to the derived concepts. Finally, we realized a patient-screening tool for clinical research. Results In total, 500 sentences randomly selected from 4691 eligibility criteria in 389 clinical trials on stroke from the Chinese Clinical Trial Registry (ChiCTR) were evaluated. An openEHR-based clinical data repository (CDR) in a grade A tertiary hospital in China was considered as an experimental environment. Based on these, 589 medical concepts were found in the 500 sentences. Of them, 513 (87.1%) concepts could be represented, while the others could not be, because of a lack of information models and coarse-grained requirements. In addition, our case study on 6 queries demonstrated that our tool shows better query performance among 4 cases (66.67%). Conclusions We developed a patient-screening tool using openEHR. It not only helps solve concept mismatch but also improves query performance to reduce the burden on researchers. In addition, we demonstrated a promising solution for secondary use of EHR data using openEHR, which can be referenced by other researchers.
Collapse
Affiliation(s)
- Mengyang Li
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China.,Key Laboratory for Biomedical Engineering, Ministry of Education, Hangzhou, Zhejiang, China
| | - Hailing Cai
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China.,Key Laboratory for Biomedical Engineering, Ministry of Education, Hangzhou, Zhejiang, China
| | - Shan Nan
- Hainan University School of Biomedical Engineering, Haikou City, China
| | - Jialin Li
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China.,Key Laboratory for Biomedical Engineering, Ministry of Education, Hangzhou, Zhejiang, China
| | - Xudong Lu
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China.,Key Laboratory for Biomedical Engineering, Ministry of Education, Hangzhou, Zhejiang, China
| | - Huilong Duan
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China.,Key Laboratory for Biomedical Engineering, Ministry of Education, Hangzhou, Zhejiang, China
| |
Collapse
|
14
|
Sarker A, Al-Garadi MA, Yang YC, Choi J, Quyyumi AA, Martin GS. Defining Patient-Oriented Natural Language Processing: A New Paradigm for Research and Development to Facilitate Adoption and Use by Medical Experts. JMIR Med Inform 2021; 9:e18471. [PMID: 34581670 PMCID: PMC8512184 DOI: 10.2196/18471] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2020] [Revised: 02/13/2021] [Accepted: 08/02/2021] [Indexed: 01/20/2023] Open
Abstract
The capabilities of natural language processing (NLP) methods have expanded significantly in recent years, and progress has been particularly driven by advances in data science and machine learning. However, NLP is still largely underused in patient-oriented clinical research and care (POCRC). A key reason behind this is that clinical NLP methods are typically developed, optimized, and evaluated with narrowly focused data sets and tasks (eg, those for the detection of specific symptoms in free texts). Such research and development (R&D) approaches may be described as problem oriented, and the developed systems perform specialized tasks well. As standalone systems, however, they generally do not comprehensively meet the needs of POCRC. Thus, there is often a gap between the capabilities of clinical NLP methods and the needs of patient-facing medical experts. We believe that to increase the practical use of biomedical NLP, future R&D efforts need to be broadened to a new research paradigm-one that explicitly incorporates characteristics that are crucial for POCRC. We present our viewpoint about 4 such interrelated characteristics that can increase NLP systems' suitability for POCRC (3 that represent NLP system properties and 1 associated with the R&D process)-(1) interpretability (the ability to explain system decisions), (2) patient centeredness (the capability to characterize diverse patients), (3) customizability (the flexibility for adapting to distinct settings, problems, and cohorts), and (4) multitask evaluation (the validation of system performance based on multiple tasks involving heterogeneous data sets). By using the NLP task of clinical concept detection as an example, we detail these characteristics and discuss how they may result in the increased uptake of NLP systems for POCRC.
Collapse
Affiliation(s)
- Abeed Sarker
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States
| | - Mohammed Ali Al-Garadi
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States
| | - Yuan-Chi Yang
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States
| | - Jinho Choi
- Department of Computer Science, College of Arts and Sciences, Emory University, Atlanta, GA, United States
| | - Arshed A Quyyumi
- Emory Clinical Cardiovascular Institute, Division of Cardiology, Department of Medicine, School of Medicine, Emory University, Atlanta, GA, United States
| | - Greg S Martin
- Predictive Health Institute and Center for Health Discovery and Well Being, Department of Medicine, School of Medicine, Emory University, Atlanta, GA, United States
| |
Collapse
|
15
|
Ni Y, Bachtel A, Nause K, Beal S. Automated detection of substance use information from electronic health records for a pediatric population. J Am Med Inform Assoc 2021; 28:2116-2127. [PMID: 34333636 PMCID: PMC8449626 DOI: 10.1093/jamia/ocab116] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 05/06/2021] [Accepted: 05/26/2021] [Indexed: 11/12/2022] Open
Abstract
OBJECTIVE Substance use screening in adolescence is unstandardized and often documented in clinical notes, rather than in structured electronic health records (EHRs). The objective of this study was to integrate logic rules with state-of-the-art natural language processing (NLP) and machine learning technologies to detect substance use information from both structured and unstructured EHR data. MATERIALS AND METHODS Pediatric patients (10-20 years of age) with any encounter between July 1, 2012, and October 31, 2017, were included (n = 3890 patients; 19 478 encounters). EHR data were extracted at each encounter, manually reviewed for substance use (alcohol, tobacco, marijuana, opiate, any use), and coded as lifetime use, current use, or family use. Logic rules mapped structured EHR indicators to screening results. A knowledge-based NLP system and a deep learning model detected substance use information from unstructured clinical narratives. System performance was evaluated using positive predictive value, sensitivity, negative predictive value, specificity, and area under the receiver-operating characteristic curve (AUC). RESULTS The dataset included 17 235 structured indicators and 27 141 clinical narratives. Manual review of clinical narratives captured 94.0% of positive screening results, while structured EHR data captured 22.0%. Logic rules detected screening results from structured data with 1.0 and 0.99 for sensitivity and specificity, respectively. The knowledge-based system detected substance use information from clinical narratives with 0.86, 0.79, and 0.88 for AUC, sensitivity, and specificity, respectively. The deep learning model further improved detection capacity, achieving 0.88, 0.81, and 0.85 for AUC, sensitivity, and specificity, respectively. Finally, integrating predictions from structured and unstructured data achieved high detection capacity across all cases (0.96, 0.85, and 0.87 for AUC, sensitivity, and specificity, respectively). CONCLUSIONS It is feasible to detect substance use screening and results among pediatric patients using logic rules, NLP, and machine learning technologies.
Collapse
Affiliation(s)
- Yizhao Ni
- Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA
- Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, Ohio, USA
- Corresponding Author: Yizhao Ni, PhD, Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, Department of Pediatrics, University of Cincinnati, 3333 Burnet Avenue, Cincinnati, OH 45229-3039, USA;
| | - Alycia Bachtel
- Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA
| | - Katie Nause
- Division of Psychology, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA
| | - Sarah Beal
- Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, Ohio, USA
- Division of Psychology, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA
| |
Collapse
|
16
|
Zeng K, Xu Y, Lin G, Liang L, Hao T. Automated classification of clinical trial eligibility criteria text based on ensemble learning and metric learning. BMC Med Inform Decis Mak 2021; 21:129. [PMID: 34330259 PMCID: PMC8323220 DOI: 10.1186/s12911-021-01492-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Accepted: 04/08/2021] [Indexed: 12/02/2022] Open
Abstract
BACKGROUND Eligibility criteria are the primary strategy for screening the target participants of a clinical trial. Automated classification of clinical trial eligibility criteria text by using machine learning methods improves recruitment efficiency to reduce the cost of clinical research. However, existing methods suffer from poor classification performance due to the complexity and imbalance of eligibility criteria text data. METHODS An ensemble learning-based model with metric learning is proposed for eligibility criteria classification. The model integrates a set of pre-trained models including Bidirectional Encoder Representations from Transformers (BERT), A Robustly Optimized BERT Pretraining Approach (RoBERTa), XLNet, Pre-training Text Encoders as Discriminators Rather Than Generators (ELECTRA), and Enhanced Representation through Knowledge Integration (ERNIE). Focal Loss is used as a loss function to address the data imbalance problem. Metric learning is employed to train the embedding of each base model for feature distinguish. Soft Voting is applied to achieve final classification of the ensemble model. The dataset is from the standard evaluation task 3 of 5th China Health Information Processing Conference containing 38,341 eligibility criteria text in 44 categories. RESULTS Our ensemble method had an accuracy of 0.8497, a precision of 0.8229, and a recall of 0.8216 on the dataset. The macro F1-score was 0.8169, outperforming state-of-the-art baseline methods by 0.84% improvement on average. In addition, the performance improvement had a p-value of 2.152e-07 with a standard t-test, indicating that our model achieved a significant improvement. CONCLUSIONS A model for classifying eligibility criteria text of clinical trials based on multi-model ensemble learning and metric learning was proposed. The experiments demonstrated that the classification performance was improved by our ensemble model significantly. In addition, metric learning was able to improve word embedding representation and the focal loss reduced the impact of data imbalance to model performance.
Collapse
Affiliation(s)
- Kun Zeng
- School of Data and Computer Science, Sun Yat-Sen University, Guangzhou, China
| | - Yibin Xu
- School of Data and Computer Science, Sun Yat-Sen University, Guangzhou, China
| | - Ge Lin
- National Engineering Research Center of Digital Life, Sun Yat-Sen University, Guangzhou, China
| | - Likeng Liang
- School of Computer Science, South China Normal University, Guangzhou, China
| | - Tianyong Hao
- School of Computer Science, South China Normal University, Guangzhou, China
| |
Collapse
|
17
|
Xiong Y, Peng W, Chen Q, Huang Z, Tang B. A Unified Machine Reading Comprehension Framework for Cohort Selection. IEEE J Biomed Health Inform 2021; 26:379-387. [PMID: 34236972 DOI: 10.1109/jbhi.2021.3095478] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Cohort selection is an essential prerequisite for clinical research, determining whether an individual satisfies given selection criteria. Previous works for cohort selection usually treated each selection criterion independently and ignored not only the meaning of each selection criterion but the relations among cohort selection criteria. To solve the problems above, we propose a novel unified machine reading comprehension (MRC) framework. In this MRC framework, we design simple rules to generate questions for each criterion from cohort selection guidelines and treat clues extracted by trigger words from patients' medical records as passages. A series of state-of-the-art MRC models based on BiDAF, BIMPM, BERT, BioBERT, NCBI-BERT, and RoBERTa are deployed to determine which question and passage pairs match. We also introduce a cross-criterion attention mechanism on representations of question and passage pairs to model relations among cohort selection criteria. Results on two datasets, that is, the dataset of the 2018 National NLP Clinical Challenge (N2C2) for cohort selection and a dataset from the MIMIC-III dataset, show that our NCBI-BERT MRC model with cross-criterion attention mechanism achieves the highest micro-averaged F1-score of 0.9070 on the N2C2 dataset and 0.8353 on the MIMIC-III dataset. It is competitive to the best system that relies on a large number of rules defined by medical experts on the N2C2 dataset. Comparing these two models, we find that the NCBI-BERT MRC model mainly performs worse on mathematical logic criteria. When using rules instead of the NCBI-BERT MRC model on some criteria regarding mathematical logic on the N2C2 dataset, we obtain a new benchmark with an F1-score of 0.9163, indicating that it is easy to integrate rules into MRC models for improvement.
Collapse
|
18
|
Zong H, Yang J, Zhang Z, Li Z, Zhang X. Semantic categorization of Chinese eligibility criteria in clinical trials using machine learning methods. BMC Med Inform Decis Mak 2021; 21:128. [PMID: 33858409 PMCID: PMC8050926 DOI: 10.1186/s12911-021-01487-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2020] [Accepted: 04/01/2021] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Semantic categorization analysis of clinical trials eligibility criteria based on natural language processing technology is crucial for the task of optimizing clinical trials design and building automated patient recruitment system. However, most of related researches focused on English eligibility criteria, and to the best of our knowledge, there are no researches studied the Chinese eligibility criteria. Thus in this study, we aimed to explore the semantic categories of Chinese eligibility criteria. METHODS We downloaded the clinical trials registration files from the website of Chinese Clinical Trial Registry (ChiCTR) and extracted both the Chinese eligibility criteria and corresponding English eligibility criteria. We represented the criteria sentences based on the Unified Medical Language System semantic types and conducted the hierarchical clustering algorithm for the induction of semantic categories. Furthermore, in order to explore the classification performance of Chinese eligibility criteria with our developed semantic categories, we implemented multiple classification algorithms, include four baseline machine learning algorithms (LR, NB, kNN, SVM), three deep learning algorithms (CNN, RNN, FastText) and two pre-trained language models (BERT, ERNIE). RESULTS We totally developed 44 types of semantic categories, summarized 8 topic groups, and investigated the average incidence and prevalence in 272 hepatocellular carcinoma related Chinese clinical trials. Compared with the previous proposed categories in English eligibility criteria, 13 novel categories are identified in Chinese eligibility criteria. The classification result shows that most of semantic categories performed quite well, the pre-trained language model ERNIE achieved best performance with macro-average F1 score of 0.7980 and micro-average F1 score of 0.8484. CONCLUSION As a pilot study of Chinese eligibility criteria analysis, we developed the 44 semantic categories by hierarchical clustering algorithms for the first times, and validated the classification capacity with multiple classification algorithms.
Collapse
Affiliation(s)
- Hui Zong
- Research Center for Translational Medicine, Shanghai East Hospital, School of Life Sciences and Technology, Tongji University, Shanghai, 200092, China
| | - Jinxuan Yang
- Research Center for Translational Medicine, Shanghai East Hospital, School of Life Sciences and Technology, Tongji University, Shanghai, 200092, China
| | - Zeyu Zhang
- Research Center for Translational Medicine, Shanghai East Hospital, School of Life Sciences and Technology, Tongji University, Shanghai, 200092, China
| | - Zuofeng Li
- Philips Research China, Shanghai, 200072, China
| | - Xiaoyan Zhang
- Research Center for Translational Medicine, Shanghai East Hospital, School of Life Sciences and Technology, Tongji University, Shanghai, 200092, China.
| |
Collapse
|
19
|
宗 辉, 张 泽, 杨 金, 雷 健, 李 作, 郝 天, 张 晓. [Artificial intelligence based Chinese clinical trials eligibility criteria classification]. SHENG WU YI XUE GONG CHENG XUE ZA ZHI = JOURNAL OF BIOMEDICAL ENGINEERING = SHENGWU YIXUE GONGCHENGXUE ZAZHI 2021; 38:105-110. [PMID: 33899434 PMCID: PMC10307579 DOI: 10.7507/1001-5515.202006035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 06/15/2020] [Revised: 11/18/2020] [Indexed: 11/03/2022]
Abstract
Subject recruitment is a key component that affects the progress and results of clinical trials, and generally conducted with eligibility criteria (includes inclusion criteria and exclusion criteria). The semantic category analysis of eligibility criteria can help optimizing clinical trials design and building automated patient recruitment system. This study explored the automatic semantic categories classification of Chinese eligibility criteria based on artificial intelligence by academic shared task. We totally collected 38 341 annotated eligibility criteria sentences and predefined 44 semantic categories. A total of 75 teams participated in competition, with 27 teams having submitted system outputs. Based on the results, we found out that most teams adopted mixed models. The mainstream resolution was applying pre-trained language models capable of providing rich semantic representation, which were combined with neural network models and used to fine-tune the models with reference to classifier tasks, and finally improved classification performance could be obtained by ensemble modeling. The best-performing system achieved a macro F1 score of 0.81 by using a pre-trained language model, i.e. bidirectional encoder representations from transformers (BERT) and ensemble modeling. With the error analysis we found out that from the point of data processing steps the data pre-processing and post-processing were very important for classification, while from the point of data volume these categories with less data volume showed lower classification performance. Finally, we hope that this study could provide a valuable dataset and state-of-the-art result for the research of Chinese medical short text classification.
Collapse
Affiliation(s)
- 辉 宗
- 同济大学 生命科学与技术学院(上海 200092)School of Life Sciences and Technology, Tongji University, Shanghai 200092, P.R.China
| | - 泽宇 张
- 同济大学 生命科学与技术学院(上海 200092)School of Life Sciences and Technology, Tongji University, Shanghai 200092, P.R.China
| | - 金璇 杨
- 同济大学 生命科学与技术学院(上海 200092)School of Life Sciences and Technology, Tongji University, Shanghai 200092, P.R.China
| | - 健波 雷
- 同济大学 生命科学与技术学院(上海 200092)School of Life Sciences and Technology, Tongji University, Shanghai 200092, P.R.China
| | - 作峰 李
- 同济大学 生命科学与技术学院(上海 200092)School of Life Sciences and Technology, Tongji University, Shanghai 200092, P.R.China
| | - 天永 郝
- 同济大学 生命科学与技术学院(上海 200092)School of Life Sciences and Technology, Tongji University, Shanghai 200092, P.R.China
| | - 晓艳 张
- 同济大学 生命科学与技术学院(上海 200092)School of Life Sciences and Technology, Tongji University, Shanghai 200092, P.R.China
| |
Collapse
|
20
|
Vydiswaran VGV, Strayhorn A, Zhao X, Robinson P, Agarwal M, Bagazinski E, Essiet M, Iott BE, Joo H, Ko P, Lee D, Lu JX, Liu J, Murali A, Sasagawa K, Wang T, Yuan N. Hybrid bag of approaches to characterize selection criteria for cohort identification. J Am Med Inform Assoc 2021; 26:1172-1180. [PMID: 31197354 DOI: 10.1093/jamia/ocz079] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2019] [Revised: 03/23/2019] [Accepted: 05/01/2019] [Indexed: 12/14/2022] Open
Abstract
OBJECTIVE The 2018 National NLP Clinical Challenge (2018 n2c2) focused on the task of cohort selection for clinical trials, where participating systems were tasked with analyzing longitudinal patient records to determine if the patients met or did not meet any of the 13 selection criteria. This article describes our participation in this shared task. MATERIALS AND METHODS We followed a hybrid approach combining pattern-based, knowledge-intensive, and feature weighting techniques. After preprocessing the notes using publicly available natural language processing tools, we developed individual criterion-specific components that relied on collecting knowledge resources relevant for these criteria and pattern-based and weighting approaches to identify "met" and "not met" cases. RESULTS As part of the 2018 n2c2 challenge, 3 runs were submitted. The overall micro-averaged F1 on the training set was 0.9444. On the test set, the micro-averaged F1 for the 3 submitted runs were 0.9075, 0.9065, and 0.9056. The best run was placed second in the overall challenge and all 3 runs were statistically similar to the top-ranked system. A reimplemented system achieved the best overall F1 of 0.9111 on the test set. DISCUSSION We highlight the need for a focused resource-intensive effort to address the class imbalance in the cohort selection identification task. CONCLUSION Our hybrid approach was able to identify all selection criteria with high F1 performance on both training and test sets. Based on our participation in the 2018 n2c2 task, we conclude that there is merit in continuing a focused criterion-specific analysis and developing appropriate knowledge resources to build a quality cohort selection system.
Collapse
Affiliation(s)
- V G Vinod Vydiswaran
- Department of Learning Health Sciences, University of Michigan, Ann Arbor, Michigan, USA.,School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Asher Strayhorn
- Department of Learning Health Sciences, University of Michigan, Ann Arbor, Michigan, USA
| | - Xinyan Zhao
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Phil Robinson
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Mahesh Agarwal
- Department of Mathematics and Statistics, College of Arts, Sciences, and Letters, University of Michigan-Dearborn, Dearborn, Michigan, USA
| | - Erin Bagazinski
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Madia Essiet
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Bradley E Iott
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Hyeon Joo
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - PingJui Ko
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Dahee Lee
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Jin Xiu Lu
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Jinghui Liu
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Adharsh Murali
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Koki Sasagawa
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Tianshi Wang
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Nalingna Yuan
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| |
Collapse
|
21
|
Hourali S, Zahedi M, Fateh M. A new model for coreference resolution based on knowledge representation and multi-criteria ranking. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2021. [DOI: 10.3233/jifs-201050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Coreference resolution is critical for improving the performance of all text-based systems including information extraction, document summarization, machine translation, and question-answering. Most of coreference resolution solutions rely on using knowledge resources like lexical knowledge, syntactic knowledge, world knowledge and semantic knowledge. This paper presents a new knowledge-based coreference resolution model using neural network architecture. It uses XLNet embeddings as input and does not rely on any syntactic or dependency parsers. For more efficient span representation and mention detection, we used entity-level information. Mentions were extracted from the text with an unhand engineered mention detector, and the features were extracted from a deep neural network. We also propose a nonlinear multi-criteria ranking model to rank the candidate antecedents. This model simultaneously determines the total score of alternatives and the weight of the features in order to speed up the process of ranking alternatives. Compared to the state-of-the-art models, the simulation results showed significant improvements on the English CoNLL-2012 shared task (+6.4 F1). Moreover, we achieved 96.1% F1 score on the n2c2 medical dataset.
Collapse
Affiliation(s)
- Samira Hourali
- Faculty of Computer and IT Engineering, Shahrood University of Technology, Shahrood, Iran
| | - Morteza Zahedi
- Faculty of Computer and IT Engineering, Shahrood University of Technology, Shahrood, Iran
| | - Mansour Fateh
- Faculty of Computer and IT Engineering, Shahrood University of Technology, Shahrood, Iran
| |
Collapse
|
22
|
Xiong Y, Chen S, Chen Q, Yan J, Tang B. Using Character-Level and Entity-Level Representations to Enhance Bidirectional Encoder Representation From Transformers-Based Clinical Semantic Textual Similarity Model: ClinicalSTS Modeling Study. JMIR Med Inform 2020; 8:e23357. [PMID: 33372664 PMCID: PMC7803475 DOI: 10.2196/23357] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2020] [Revised: 11/10/2020] [Accepted: 11/16/2020] [Indexed: 12/03/2022] Open
Abstract
Background With the popularity of electronic health records (EHRs), the quality of health care has been improved. However, there are also some problems caused by EHRs, such as the growing use of copy-and-paste and templates, resulting in EHRs of low quality in content. In order to minimize data redundancy in different documents, Harvard Medical School and Mayo Clinic organized a national natural language processing (NLP) clinical challenge (n2c2) on clinical semantic textual similarity (ClinicalSTS) in 2019. The task of this challenge is to compute the semantic similarity among clinical text snippets. Objective In this study, we aim to investigate novel methods to model ClinicalSTS and analyze the results. Methods We propose a semantically enhanced text matching model for the 2019 n2c2/Open Health NLP (OHNLP) challenge on ClinicalSTS. The model includes 3 representation modules to encode clinical text snippet pairs at different levels: (1) character-level representation module based on convolutional neural network (CNN) to tackle the out-of-vocabulary problem in NLP; (2) sentence-level representation module that adopts a pretrained language model bidirectional encoder representation from transformers (BERT) to encode clinical text snippet pairs; and (3) entity-level representation module to model clinical entity information in clinical text snippets. In the case of entity-level representation, we compare 2 methods. One encodes entities by the entity-type label sequence corresponding to text snippet (called entity I), whereas the other encodes entities by their representation in MeSH, a knowledge graph in the medical domain (called entity II). Results We conduct experiments on the ClinicalSTS corpus of the 2019 n2c2/OHNLP challenge for model performance evaluation. The model only using BERT for text snippet pair encoding achieved a Pearson correlation coefficient (PCC) of 0.848. When character-level representation and entity-level representation are individually added into our model, the PCC increased to 0.857 and 0.854 (entity I)/0.859 (entity II), respectively. When both character-level representation and entity-level representation are added into our model, the PCC further increased to 0.861 (entity I) and 0.868 (entity II). Conclusions Experimental results show that both character-level information and entity-level information can effectively enhance the BERT-based STS model.
Collapse
Affiliation(s)
- Ying Xiong
- Harbin Institute of Technology, Shenzhen, China
| | - Shuai Chen
- Harbin Institute of Technology, Shenzhen, China
| | - Qingcai Chen
- Harbin Institute of Technology, Shenzhen, China.,Peng Cheng Laboratory, Shenzhen, China
| | - Jun Yan
- Yidu Cloud Technology Company Limited, Beijing, China
| | - Buzhou Tang
- Harbin Institute of Technology, Shenzhen, China.,Peng Cheng Laboratory, Shenzhen, China
| |
Collapse
|
23
|
Liu S, Wang Y, Wen A, Wang L, Hong N, Shen F, Bedrick S, Hersh W, Liu H. Implementation of a Cohort Retrieval System for Clinical Data Repositories Using the Observational Medical Outcomes Partnership Common Data Model: Proof-of-Concept System Validation. JMIR Med Inform 2020; 8:e17376. [PMID: 33021486 PMCID: PMC7576539 DOI: 10.2196/17376] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2019] [Revised: 06/04/2020] [Accepted: 07/28/2020] [Indexed: 12/17/2022] Open
Abstract
BACKGROUND Widespread adoption of electronic health records has enabled the secondary use of electronic health record data for clinical research and health care delivery. Natural language processing techniques have shown promise in their capability to extract the information embedded in unstructured clinical data, and information retrieval techniques provide flexible and scalable solutions that can augment natural language processing systems for retrieving and ranking relevant records. OBJECTIVE In this paper, we present the implementation of a cohort retrieval system that can execute textual cohort selection queries on both structured data and unstructured text-Cohort Retrieval Enhanced by Analysis of Text from Electronic Health Records (CREATE). METHODS CREATE is a proof-of-concept system that leverages a combination of structured queries and information retrieval techniques on natural language processing results to improve cohort retrieval performance using the Observational Medical Outcomes Partnership Common Data Model to enhance model portability. The natural language processing component was used to extract common data model concepts from textual queries. We designed a hierarchical index to support the common data model concept search utilizing information retrieval techniques and frameworks. RESULTS Our case study on 5 cohort identification queries, evaluated using the precision at 5 information retrieval metric at both the patient-level and document-level, demonstrates that CREATE achieves a mean precision at 5 of 0.90, which outperforms systems using only structured data or only unstructured text with mean precision at 5 values of 0.54 and 0.74, respectively. CONCLUSIONS The implementation and evaluation of Mayo Clinic Biobank data demonstrated that CREATE outperforms cohort retrieval systems that only use one of either structured data or unstructured text in complex textual cohort queries.
Collapse
Affiliation(s)
- Sijia Liu
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Yanshan Wang
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Andrew Wen
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Liwei Wang
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Na Hong
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Feichen Shen
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| | - Steven Bedrick
- Department of Computer Science and Electrical Engineering, Oregon Health & Science University, Portland, OR, United States
| | - William Hersh
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR, United States
| | - Hongfang Liu
- Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States
| |
Collapse
|
24
|
Chamberlin SR, Bedrick SD, Cohen AM, Wang Y, Wen A, Liu S, Liu H, Hersh WR. Evaluation of patient-level retrieval from electronic health record data for a cohort discovery task. JAMIA Open 2020; 3:395-404. [PMID: 33215074 PMCID: PMC7660955 DOI: 10.1093/jamiaopen/ooaa026] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2019] [Revised: 04/17/2020] [Accepted: 06/03/2020] [Indexed: 11/24/2022] Open
Abstract
OBJECTIVE Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well understood. The objective of this research was to assess patient-level information retrieval methods using electronic health records for different types of cohort definition retrieval. MATERIALS AND METHODS We developed a test collection consisting of about 100 000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated information retrieval tasks using word-based approaches were performed, varying 4 different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics. RESULTS The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision but were still not able to recall all relevant patients found by the automated queries. CONCLUSION While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Future work will focus on using the test collection to develop and evaluate new approaches to query structure, weighting algorithms, and application of semantic methods.
Collapse
Affiliation(s)
- Steven R Chamberlin
- Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, USA
| | - Steven D Bedrick
- Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, USA
- Center for Spoken Language Understanding, Oregon Health & Science University, Portland, Oregon, USA
| | - Aaron M Cohen
- Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, USA
| | - Yanshan Wang
- Division of Digital Health Sciences, Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA
| | - Andrew Wen
- Division of Digital Health Sciences, Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA
| | - Sijia Liu
- Division of Digital Health Sciences, Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA
| | - Hongfang Liu
- Division of Digital Health Sciences, Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA
| | - William R Hersh
- Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, USA
| |
Collapse
|
25
|
Luo YF, Henry S, Wang Y, Shen F, Uzuner O, Rumshisky A. The 2019 National Natural language processing (NLP) Clinical Challenges (n2c2)/Open Health NLP (OHNLP) shared task on clinical concept normalization for clinical records. J Am Med Inform Assoc 2020; 27:1529-1537. [PMID: 32968800 PMCID: PMC7647359 DOI: 10.1093/jamia/ocaa106] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2020] [Revised: 05/01/2020] [Accepted: 05/14/2020] [Indexed: 01/19/2023] Open
Abstract
OBJECTIVE The 2019 National Natural language processing (NLP) Clinical Challenges (n2c2)/Open Health NLP (OHNLP) shared task track 3, focused on medical concept normalization (MCN) in clinical records. This track aimed to assess the state of the art in identifying and matching salient medical concepts to a controlled vocabulary. In this paper, we describe the task, describe the data set used, compare the participating systems, present results, identify the strengths and limitations of the current state of the art, and identify directions for future research. MATERIALS AND METHODS Participating teams were provided with narrative discharge summaries in which text spans corresponding to medical concepts were identified. This paper refers to these text spans as mentions. Teams were tasked with normalizing these mentions to concepts, represented by concept unique identifiers, within the Unified Medical Language System. Submitted systems represented 4 broad categories of approaches: cascading dictionary matching, cosine distance, deep learning, and retrieve-and-rank systems. Disambiguation modules were common across all approaches. RESULTS A total of 33 teams participated in the MCN task. The best-performing team achieved an accuracy of 0.8526. The median and mean performances among all teams were 0.7733 and 0.7426, respectively. CONCLUSIONS Overall performance among the top 10 teams was high. However, several mention types were challenging for all teams. These included mentions requiring disambiguation of misspelled words, acronyms, abbreviations, and mentions with more than 1 possible semantic type. Also challenging were complex mentions of long, multi-word terms that may require new ways of extracting and representing mention meaning, the use of domain knowledge, parse trees, or hand-crafted rules.
Collapse
Affiliation(s)
- Yen-Fu Luo
- Department of Computer Science, University of Massachusetts
Lowell, Lowell, Massachusetts, USA
| | - Sam Henry
- Department of Information Sciences and Technology, George Mason
University, Fairfax, Virginia, USA
| | - Yanshan Wang
- Department of Health Sciences Research, Mayo Clinic, Rochester,
New York, USA
| | - Feichen Shen
- Department of Health Sciences Research, Mayo Clinic, Rochester,
New York, USA
| | - Ozlem Uzuner
- Department of Information Sciences and Technology, George Mason
University, Fairfax, Virginia, USA
- Department of Biomedical Informatics, Harvard Medical School,
Boston, Massachusetts, USA
- Computer Science and Artificial Intelligence Laboratory, Massachusetts
Institute of Technology, Cambridge, Massachusetts, USA
| | - Anna Rumshisky
- Department of Computer Science, University of Massachusetts
Lowell, Lowell, Massachusetts, USA
- Computer Science and Artificial Intelligence Laboratory, Massachusetts
Institute of Technology, Cambridge, Massachusetts, USA
| |
Collapse
|
26
|
Fu S, Chen D, He H, Liu S, Moon S, Peterson KJ, Shen F, Wang L, Wang Y, Wen A, Zhao Y, Sohn S, Liu H. Clinical concept extraction: A methodology review. J Biomed Inform 2020; 109:103526. [PMID: 32768446 PMCID: PMC7746475 DOI: 10.1016/j.jbi.2020.103526] [Citation(s) in RCA: 60] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2020] [Revised: 07/30/2020] [Accepted: 08/02/2020] [Indexed: 01/11/2023]
Abstract
BACKGROUND Concept extraction, a subdomain of natural language processing (NLP) with a focus on extracting concepts of interest, has been adopted to computationally extract clinical information from text for a wide range of applications ranging from clinical decision support to care quality improvement. OBJECTIVES In this literature review, we provide a methodology review of clinical concept extraction, aiming to catalog development processes, available methods and tools, and specific considerations when developing clinical concept extraction applications. METHODS Based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, a literature search was conducted for retrieving EHR-based information extraction articles written in English and published from January 2009 through June 2019 from Ovid MEDLINE In-Process & Other Non-Indexed Citations, Ovid MEDLINE, Ovid EMBASE, Scopus, Web of Science, and the ACM Digital Library. RESULTS A total of 6,686 publications were retrieved. After title and abstract screening, 228 publications were selected. The methods used for developing clinical concept extraction applications were discussed in this review.
Collapse
Affiliation(s)
- Sunyang Fu
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States; University of Minnesota - Twin Cities, Minneapolis, MN 55455, United States.
| | - David Chen
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Huan He
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Sijia Liu
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Sungrim Moon
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Kevin J Peterson
- Department of Information Technology, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States; University of Minnesota - Twin Cities, Minneapolis, MN 55455, United States.
| | - Feichen Shen
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Liwei Wang
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Yanshan Wang
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Andrew Wen
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Yiqing Zhao
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Sunghwan Sohn
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States.
| | - Hongfang Liu
- Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905, United States; University of Minnesota - Twin Cities, Minneapolis, MN 55455, United States.
| |
Collapse
|
27
|
Benchmarking machine learning models on multi-centre eICU critical care dataset. PLoS One 2020; 15:e0235424. [PMID: 32614874 PMCID: PMC7332047 DOI: 10.1371/journal.pone.0235424] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2020] [Accepted: 06/15/2020] [Indexed: 02/06/2023] Open
Abstract
Progress of machine learning in critical care has been difficult to track, in part due to absence of public benchmarks. Other fields of research (such as computer vision and natural language processing) have established various competitions and public benchmarks. Recent availability of large clinical datasets has enabled the possibility of establishing public benchmarks. Taking advantage of this opportunity, we propose a public benchmark suite to address four areas of critical care, namely mortality prediction, estimation of length of stay, patient phenotyping and risk of decompensation. We define each task and compare the performance of both clinical models as well as baseline and deep learning models using eICU critical care dataset of around 73,000 patients. This is the first public benchmark on a multi-centre critical care dataset, comparing the performance of clinical gold standard with our predictive model. We also investigate the impact of numerical variables as well as handling of categorical variables on each of the defined tasks. The source code, detailing our methods and experiments is publicly available such that anyone can replicate our results and build upon our work.
Collapse
|
28
|
Bhardwaj P, Yadav RK, Kurian S. Digitizing the Pharma Neurons - A Technological Operation in Progress! Rev Recent Clin Trials 2020; 15:178-187. [PMID: 32564760 DOI: 10.2174/1574887115666200621183459] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2020] [Revised: 04/27/2020] [Accepted: 05/22/2020] [Indexed: 11/22/2022]
Abstract
BACKGROUND Digitization and automation are the buzzwords in clinical research and pharma companies are investigating heavily here. Right from drug discovery to personalized medicine, digital patients and patient engagement, there is great consideration of technology at each step. METHODS The published data and online information available is reviewed to give an overview of digitization in pharma, across the drug development cycle, industry collaborations and innovations. The regulatory guidelines, innovative collaborations across industry, academics and thought leadership are presented. Also included are some ideas, suggestions, way forwards while digitizing the pharma neurons, the regulatory stand, benefits and challenges. RESULTS The innovations range from discovering personalized medicine to conducting virtual clinical trials, and maximizing data collection from the real-world experience. To address the increasing demand for the real-world data and the needs of tech-savvy patients, the innovations are shaping up accordingly. Pharma companies are collaborating with academics and they are co-innovating the technology for example Massachusetts Institute of Technology's program. This focuses on the modernization of clinical trials, strategic use of artificial intelligence and machine learning using real-world evidence, assess the risk-benefit ratio of deploying digital analytics in medicine, and proactively identifying the solutions. CONCLUSION With unfolding data on the impact of science and technology amalgamation, we need shared mindset between data scientists and medical professionals to maximize the utility of enormous health and medical data. To tackle this efficiently, there is a need of cross-collaboration and education, and align with ethical and regulatory requirements. A perfect blend of industry, regulatory, and academia will ensure successful digitization of pharma neurons.
Collapse
Affiliation(s)
| | - Raj Kumar Yadav
- Integral Health Clinic, Department of Physiology, All India Institute of Medical Sciences, New Delhi-110029, India
| | - Sojan Kurian
- Tata Consultancy Services, New York, NY, United States
| |
Collapse
|
29
|
Dai HJ, Wang FD, Chen CW, Su CH, Wu CS, Jonnagaddala J. Cohort selection for clinical trials using multiple instance learning. J Biomed Inform 2020; 107:103438. [PMID: 32360937 DOI: 10.1016/j.jbi.2020.103438] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2019] [Revised: 02/29/2020] [Accepted: 04/27/2020] [Indexed: 10/24/2022]
Abstract
Identifying patients eligible for clinical trials using electronic health records (EHRs) is a challenging task usually requiring a comprehensive analysis of information stored in multiple EHRs of a patient. The goal of this study is to investigate different methods and their effectiveness in identifying patients that meet specific eligibility selection criteria based on patients' longitudinal records. An unstructured dataset released by the n2c2 cohort selection for clinical trials track was used, each of which included 2-5 records manually annotated to thirteen pre-defined selection criteria. Unlike the other studies, we formulated the problem as a multiple instance learning (MIL) task and compared the performance with that of the rule-based and the single instance-based classifiers. Our official best run achieved an average micro-F score of 0.8765 which was ranked as one of the top ten results in the track. Further experiments demonstrated that the performance of the MIL-based classifiers consistently yield better performance than their single-instance counterparts in the criteria that require the overall comprehension of the information distributed among all of the patient's EHRs. Rule-based and single instance learning approaches exhibited better performance in criteria that don't require a consideration of several factors across records. This study demonstrated that cohort selection using longitudinal patient records can be formulated as a MIL problem. Our results exhibit that the MIL-based classifiers supplement the rule-based methods and provide better results in comparison to the single instance learning approaches.
Collapse
Affiliation(s)
- Hong-Jie Dai
- Department of Electrical Engineering, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan, ROC; School of Post-Baccalaureate Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan, ROC; National Institute of Cancer Research, National Health Research Institutes, Taiwan, R.O.C..
| | - Feng-Duo Wang
- Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan, ROC
| | - Chih-Wei Chen
- Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei, Taiwan, ROC
| | - Chu-Hsien Su
- Department of Psychiatry, National Taiwan University Hospital, Taipei, Taiwan, ROC
| | - Chi-Shin Wu
- Department of Psychiatry, National Taiwan University Hospital, Taipei, Taiwan, ROC
| | | |
Collapse
|
30
|
Hassanzadeh H, Karimi S, Nguyen A. Matching patients to clinical trials using semantically enriched document representation. J Biomed Inform 2020; 105:103406. [DOI: 10.1016/j.jbi.2020.103406] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2019] [Revised: 01/28/2020] [Accepted: 03/02/2020] [Indexed: 12/16/2022]
|
31
|
Trends and Features of the Applications of Natural Language Processing Techniques for Clinical Trials Text Analysis. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10062157] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
Natural language processing (NLP) is an effective tool for generating structured information from unstructured data, the one that is commonly found in clinical trial texts. Such interdisciplinary research has gradually grown into a flourishing research field with accumulated scientific outputs available. In this study, bibliographical data collected from Web of Science, PubMed, and Scopus databases from 2001 to 2018 had been investigated with the use of three prominent methods, including performance analysis, science mapping, and, particularly, an automatic text analysis approach named structural topic modeling. Topical trend visualization and test analysis were further employed to quantify the effects of the year of publication on topic proportions. Topical diverse distributions across prolific countries/regions and institutions were also visualized and compared. In addition, scientific collaborations between countries/regions, institutions, and authors were also explored using social network analysis. The findings obtained were essential for facilitating the development of the NLP-enhanced clinical trial texts processing, boosting scientific and technological NLP-enhanced clinical trial research, and facilitating inter-country/region and inter-institution collaborations.
Collapse
|
32
|
Karystianis G, Florez-Vargas O, Butler T, Nenadic G. A rule-based approach to identify patient eligibility criteria for clinical trials from narrative longitudinal records. JAMIA Open 2020; 2:521-527. [PMID: 32025649 PMCID: PMC6993990 DOI: 10.1093/jamiaopen/ooz041] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2019] [Revised: 07/16/2019] [Accepted: 10/03/2019] [Indexed: 11/13/2022] Open
Abstract
Objective Achieving unbiased recognition of eligible patients for clinical trials from their narrative longitudinal clinical records can be time consuming. We describe and evaluate a knowledge-driven method that identifies whether a patient meets a selected set of 13 eligibility clinical trial criteria from their longitudinal clinical records, which was one of the tasks of the 2018 National NLP Clinical Challenges. Materials and Methods The approach developed uses rules combined with manually crafted dictionaries that characterize the domain. The rules are based on common syntactical patterns observed in text indicating or describing explicitly a criterion. Certain criteria were classified as "met" only when they occurred within a designated time period prior to the most recent narrative of a patient record and were dealt through their position in text. Results The system was applied to an evaluation set of 86 unseen clinical records and achieved a microaverage F1-score of 89.1% (with a micro F1-score of 87.0% and 91.2% for the patients that met and did not meet the criteria, respectively). Most criteria returned reliable results (drug abuse, 92.5%; Hba1c, 91.3%) while few (eg, advanced coronary artery disease, 72.0%; myocardial infarction within 6 months of the most recent narrative, 47.5%) proved challenging enough. Conclusion Overall, the results are encouraging and indicate that automated text mining methods can be used to process clinical records to recognize whether a patient meets a set of clinical trial criteria and could be leveraged to reduce the workload of humans screening patients for trials.
Collapse
Affiliation(s)
| | - Oscar Florez-Vargas
- Laboratory of Translational Genomics, Department of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, USA
| | - Tony Butler
- Kirby Institute, University of New South Wales, Sydney, Australia
| | - Goran Nenadic
- School of Computer Science, University of Manchester, Manchester, UK
| |
Collapse
|
33
|
Affiliation(s)
- Amber Stubbs
- Division of Mathematics and Computer Science, Simmons University, Boston, Massachusetts, USA
| | - Özlem Uzuner
- Information Sciences and Technology, George Mason University, Fairfax, Virginia, USA.,Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA.,Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|