1
|
Kovačević A, Bašaragin B, Milošević N, Nenadić G. De-identification of clinical free text using natural language processing: A systematic review of current approaches. Artif Intell Med 2024; 151:102845. [PMID: 38555848 DOI: 10.1016/j.artmed.2024.102845] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 03/13/2024] [Accepted: 03/18/2024] [Indexed: 04/02/2024]
Abstract
BACKGROUND Electronic health records (EHRs) are a valuable resource for data-driven medical research. However, the presence of protected health information (PHI) makes EHRs unsuitable to be shared for research purposes. De-identification, i.e. the process of removing PHI is a critical step in making EHR data accessible. Natural language processing has repeatedly demonstrated its feasibility in automating the de-identification process. OBJECTIVES Our study aims to provide systematic evidence on how the de-identification of clinical free text written in English has evolved in the last thirteen years, and to report on the performances and limitations of the current state-of-the-art systems for the English language. In addition, we aim to identify challenges and potential research opportunities in this field. METHODS A systematic search in PubMed, Web of Science, and the DBLP was conducted for studies published between January 2010 and February 2023. Titles and abstracts were examined to identify the relevant studies. Selected studies were then analysed in-depth, and information was collected on de-identification methodologies, data sources, and measured performance. RESULTS A total of 2125 publications were identified for the title and abstract screening. 69 studies were found to be relevant. Machine learning (37 studies) and hybrid (26 studies) approaches are predominant, while six studies relied only on rules. The majority of the approaches were trained and evaluated on public corpora. The 2014 i2b2/UTHealth corpus is the most frequently used (36 studies), followed by the 2006 i2b2 (18 studies) and 2016 CEGS N-GRID (10 studies) corpora. CONCLUSION Earlier de-identification approaches aimed at English were mainly rule and machine learning hybrids with extensive feature engineering and post-processing, while more recent performance improvements are due to feature-inferring recurrent neural networks. Current leading performance is achieved using attention-based neural models. Recent studies report state-of-the-art F1-scores (over 98 %) when evaluated in the manner usually adopted by the clinical natural language processing community. However, their performance needs to be more thoroughly assessed with different measures to judge their reliability to safely de-identify data in a real-world setting. Without additional manually labeled training data, state-of-the-art systems fail to generalise well across a wide range of clinical sub-domains.
Collapse
Affiliation(s)
- Aleksandar Kovačević
- The University of Novi Sad, Faculty of Technical Sciences, Trg Dositeja Obradovića 6, 21002 Novi Sad, Serbia
| | - Bojana Bašaragin
- The Institute for Artificial Intelligence Research and Development of Serbia, Fruškogorska 1, 21000 Novi Sad, Serbia.
| | - Nikola Milošević
- The Institute for Artificial Intelligence Research and Development of Serbia, Fruškogorska 1, 21000 Novi Sad, Serbia; Bayer A.G., Research and Development, Mullerstrasse 173, Berlin 13342, Germany
| | - Goran Nenadić
- The University of Manchester, Department of Computer Science, Manchester, United Kingdom
| |
Collapse
|
2
|
Yang R, Tan TF, Lu W, Thirunavukarasu AJ, Ting DSW, Liu N. Large language models in health care: Development, applications, and challenges. HEALTH CARE SCIENCE 2023; 2:255-263. [PMID: 38939520 PMCID: PMC11080827 DOI: 10.1002/hcs2.61] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 06/10/2023] [Accepted: 06/12/2023] [Indexed: 06/29/2024]
Abstract
Recently, the emergence of ChatGPT, an artificial intelligence chatbot developed by OpenAI, has attracted significant attention due to its exceptional language comprehension and content generation capabilities, highlighting the immense potential of large language models (LLMs). LLMs have become a burgeoning hotspot across many fields, including health care. Within health care, LLMs may be classified into LLMs for the biomedical domain and LLMs for the clinical domain based on the corpora used for pre-training. In the last 3 years, these domain-specific LLMs have demonstrated exceptional performance on multiple natural language processing tasks, surpassing the performance of general LLMs as well. This not only emphasizes the significance of developing dedicated LLMs for the specific domains, but also raises expectations for their applications in health care. We believe that LLMs may be used widely in preconsultation, diagnosis, and management, with appropriate development and supervision. Additionally, LLMs hold tremendous promise in assisting with medical education, medical writing and other related applications. Likewise, health care systems must recognize and address the challenges posed by LLMs.
Collapse
Affiliation(s)
- Rui Yang
- Department of Biomedical Informatics, Yong Loo Lin School of MedicineNational University of SingaporeSingaporeSingapore
| | - Ting Fang Tan
- Singapore National Eye Center, Singapore Eye Research InstituteSingapore Health ServiceSingaporeSingapore
| | - Wei Lu
- StatNLP Research GroupSingapore University of Technology and DesignSingapore
| | | | - Daniel Shu Wei Ting
- Singapore National Eye Center, Singapore Eye Research InstituteSingapore Health ServiceSingaporeSingapore
- Duke‐NUS Medical SchoolCentre for Quantitative MedicineSingaporeSingapore
| | - Nan Liu
- Duke‐NUS Medical SchoolCentre for Quantitative MedicineSingaporeSingapore
- Duke‐NUS Medical SchoolProgramme in Health Services and Systems ResearchSingaporeSingapore
| |
Collapse
|
3
|
Kotevski DP, Smee RI, Field M, Nemes YN, Broadley K, Vajdic CM. Evaluation of an automated Presidio anonymisation model for unstructured radiation oncology electronic medical records in an Australian setting. Int J Med Inform 2022; 168:104880. [DOI: 10.1016/j.ijmedinf.2022.104880] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2022] [Revised: 09/13/2022] [Accepted: 09/27/2022] [Indexed: 11/08/2022]
|
4
|
Wang P, Li Y, Yang L, Li S, Li L, Zhao Z, Long S, Wang F, Wang H, Li Y, Wang C. An Efficient Method for Deidentifying Protected Health Information in Chinese Electronic Health Records: Algorithm Development and Validation. JMIR Med Inform 2022; 10:e38154. [PMID: 36040774 PMCID: PMC9472063 DOI: 10.2196/38154] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Revised: 07/19/2022] [Accepted: 07/31/2022] [Indexed: 11/13/2022] Open
Abstract
Background With the popularization of electronic health records in China, the utilization of digitalized data has great potential for the development of real-world medical research. However, the data usually contains a great deal of protected health information and the direct usage of this data may cause privacy issues. The task of deidentifying protected health information in electronic health records can be regarded as a named entity recognition problem. Existing rule-based, machine learning–based, or deep learning–based methods have been proposed to solve this problem. However, these methods still face the difficulties of insufficient Chinese electronic health record data and the complex features of the Chinese language. Objective This paper proposes a method to overcome the difficulties of overfitting and a lack of training data for deep neural networks to enable Chinese protected health information deidentification. Methods We propose a new model that merges TinyBERT (bidirectional encoder representations from transformers) as a text feature extraction module and the conditional random field method as a prediction module for deidentifying protected health information in Chinese medical electronic health records. In addition, a hybrid data augmentation method that integrates a sentence generation strategy and a mention-replacement strategy is proposed for overcoming insufficient Chinese electronic health records. Results We compare our method with 5 baseline methods that utilize different BERT models as their feature extraction modules. Experimental results on the Chinese electronic health records that we collected demonstrate that our method had better performance (microprecision: 98.7%, microrecall: 99.13%, and micro-F1 score: 98.91%) and higher efficiency (40% faster) than all the BERT-based baseline methods. Conclusions Compared to baseline methods, the efficiency advantage of TinyBERT on our proposed augmented data set was kept while the performance improved for the task of Chinese protected health information deidentification.
Collapse
Affiliation(s)
- Peng Wang
- College of Computer Science, Chongqing University, Chongqing, China
| | - Yong Li
- School of Computer Science, South China Normal University, Guangzhou, China
| | - Liang Yang
- Yidu Cloud Technology Inc, Beijing, China
| | - Simin Li
- Yidu Cloud Technology Inc, Beijing, China
| | - Linfeng Li
- Yidu Cloud Technology Inc, Beijing, China
| | - Zehan Zhao
- School of Software & Microelectronics, Peking University, Beijing, China
| | - Shaopei Long
- School of Computer Science, South China Normal University, Guangzhou, China
| | - Fei Wang
- Medical Big Data Center of Southwest Hospital, Chongqing, China
| | - Hongqian Wang
- Medical Big Data Center of Southwest Hospital, Chongqing, China
| | - Ying Li
- Medical Big Data Center of Southwest Hospital, Chongqing, China
| | - Chengliang Wang
- College of Computer Science, Chongqing University, Chongqing, China
| |
Collapse
|
5
|
Jing Y. Research on fuzzy English automatic recognition and human-computer interaction based on machine learning. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2020. [DOI: 10.3233/jifs-189057] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/25/2022]
Abstract
Fuzzy English recognition is affected by many factors, which leads to certain accuracy problems in intelligent recognition results. In order to improve the automatic recognition efficiency of fuzzy English, based on machine learning technology, this study constructs a neural network model. At the same time, this paper analyzes the research status and existing problems of handwritten character recognition, analyzes the model, and adopts multiple modules for automatic English recognition. In addition, the system is built on the basis of algorithms and model support, which makes fuzzy English recognition intelligent. Finally, in order to study the algorithm and model performance, the fuzzy English recognition is carried out through experiments. The research shows that the model constructed in this paper has certain recognition effect, which can be applied to practice, and can provide theoretical reference for subsequent related research.
Collapse
Affiliation(s)
- Yuqin Jing
- School of Electronic Information Engineering, Chongqing Technology and Business Institute, Chongqing, China
| |
Collapse
|
6
|
Kajiyama K, Horiguchi H, Okumura T, Morita M, Kano Y. De-identifying free text of Japanese electronic health records. J Biomed Semantics 2020; 11:11. [PMID: 32958039 PMCID: PMC7504663 DOI: 10.1186/s13326-020-00227-9] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2019] [Accepted: 08/07/2020] [Indexed: 11/25/2022] Open
Abstract
Background Recently, more electronic data sources are becoming available in the healthcare domain. Electronic health records (EHRs), with their vast amounts of potentially available data, can greatly improve healthcare. Although EHR de-identification is necessary to protect personal information, automatic de-identification of Japanese language EHRs has not been studied sufficiently. This study was conducted to raise de-identification performance for Japanese EHRs through classic machine learning, deep learning, and rule-based methods, depending on the dataset. Results Using three datasets, we implemented de-identification systems for Japanese EHRs and compared the de-identification performances found for rule-based, Conditional Random Fields (CRF), and Long-Short Term Memory (LSTM)-based methods. Gold standard tags for de-identification are annotated manually for age, hospital, person, sex, and time. We used different combinations of our datasets to train and evaluate our three methods. Our best F1-scores were 84.23, 68.19, and 81.67 points, respectively, for evaluations of the MedNLP dataset, a dummy EHR dataset that was virtually written by a medical doctor, and a Pathology Report dataset. Our LSTM-based method was the best performing, except for the MedNLP dataset. The rule-based method was best for the MedNLP dataset. The LSTM-based method achieved a good score of 83.07 points for this MedNLP dataset, which differs by 1.16 points from the best score obtained using the rule-based method. Results suggest that LSTM adapted well to different characteristics of our datasets. Our LSTM-based method performed better than our CRF-based method, yielding a 7.41 point F1-score, when applied to our Pathology Report dataset. This report is the first of study applying this LSTM-based method to any de-identification task of a Japanese EHR. Conclusions Our LSTM-based machine learning method was able to extract named entities to be de-identified with better performance, in general, than that of our rule-based methods. However, machine learning methods are inadequate for processing expressions with low occurrence. Our future work will specifically examine the combination of LSTM and rule-based methods to achieve better performance. Our currently achieved level of performance is sufficiently higher than that of publicly available Japanese de-identification tools. Therefore, our system will be applied to actual de-identification tasks in hospitals.
Collapse
Affiliation(s)
- Kohei Kajiyama
- Faculty of Informatics, Shizuoka University, Johoku 3-5-1, Naka-ku, Hamamatsu, Shizuoka, 432-8011, Japan
| | - Hiromasa Horiguchi
- National Hospital Organization Headquaters, 2-5-21 Higashigaoka, Meguro-ku, Tokyo, 152-8621, Japan
| | - Takashi Okumura
- National University Corporation Kitami Institute of Technology, 165, Koencho, Kitami, Hokkaido, 090-8507, Japan
| | - Mizuki Morita
- Graduate School of Interdisciplinary Science and Engineering in Health Systems, Okayama University, 2-5-1, Kita-ku, Okayama, Okayama, 700-8558, Japan
| | - Yoshinobu Kano
- Faculty of Informatics, Shizuoka University, Johoku 3-5-1, Naka-ku, Hamamatsu, Shizuoka, 432-8011, Japan.
| |
Collapse
|
7
|
Zhao H, Bian J, Wei L, Li L, Ying Y, Zhang Z, Yao X, Zhuo L, Cao B, Zhang M, Zhan S. Validation of an algorithm to evaluate the appropriateness of outpatient antibiotic prescribing using big data of Chinese diagnosis text. BMJ Open 2020; 10:e031191. [PMID: 32198296 PMCID: PMC7103794 DOI: 10.1136/bmjopen-2019-031191] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/21/2022] Open
Abstract
OBJECTIVE We aimed to evaluate the validity of an algorithm to classify diagnoses according to the appropriateness of outpatient antibiotic use in the context of Chinese free text. SETTING AND PARTICIPANTS A random sample of 10 000 outpatient visits was selected between January and April 2018 from a national database for monitoring rational use of drugs, which included data from 194 secondary and tertiary hospitals in China. RESEARCH DESIGN Diagnoses for outpatient visits were classified as tier 1 if associated with at least one condition that 'always' justified antibiotic use; as tier 2 if associated with at least one condition that only 'sometimes' justified antibiotic use but no conditions that 'always' justified antibiotic use; or as tier 3 if associated with only conditions that never justified antibiotic use, using a tier-fashion method and regular expression (RE)-based algorithm. MEASURES Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the classification algorithm, using classification made by chart review as the standard reference, were calculated. RESULTS The sensitivities of the algorithm for classifying tier 1, tier 2 and tier 3 diagnoses were 98.2% (95% CI 96.4% to 99.3%), 98.4% (95% CI 97.6% to 99.1%) and 100.0% (95% CI 100.0% to 100.0%), respectively. The specificities were 100.0% (95% CI 100.0% to 100.0%), 100.0% (95% CI 99.9% to 100.0%) and 98.6% (95% CI 97.9% to 99.1%), respectively. The PPVs for classifying tier 1, tier 2 and tier 3 diagnoses were 100.0% (95% CI 99.1% to 100.0%), 99.7% (95% CI 99.2% to 99.9%) and 99.7% (95% CI 99.6% to 99.8%), respectively. The NPVs were 99.9% (95% CI 99.8% to 100.0%), 99.8% (95% CI 99.7% to 99.9%) and 100.0% (95% CI 99.8% to 100.0%), respectively. CONCLUSIONS The RE-based classification algorithm in the context of Chinese free text had sufficiently high validity for further evaluating the appropriateness of outpatient antibiotic prescribing.
Collapse
Affiliation(s)
- Houyu Zhao
- Department of Epidemiology and Biostatistics, School of Public Health, Peking University, Beijing, China
| | - Jiaming Bian
- Department of Pharmacology, Chinese PLA General Hospital, Beijing, China
| | - Li Wei
- School of Pharmacy, University College London, London, UK
| | - Liuyi Li
- Department of Healthcare-associated Infection Management and Disease Prevention and Control, Peking University First Hospital, Beijing, China
| | - Yingqiu Ying
- Department of Pharmacy, Peking University Third Hospital, Beijing, China
| | - Zeyu Zhang
- Department of Pulmonary and Critical Care Medicine, China-Japan Friendship Hospital, Beijing, China
| | - Xiaoying Yao
- Department of Epidemiology and Biostatistics, School of Public Health, Peking University, Beijing, China
| | - Lin Zhuo
- Research Center of Clinical Epidemiology, Peking University Third Hospital, Beijing, China
| | - Bin Cao
- Department of Pulmonary and Critical Care Medicine, China-Japan Friendship Hospital, Beijing, China
| | - Mei Zhang
- Department of Pharmacology, Chinese PLA General Hospital, Beijing, China
| | - Siyan Zhan
- Department of Epidemiology and Biostatistics, School of Public Health, Peking University, Beijing, China
| |
Collapse
|
8
|
Bloomrosen M, Berner ES. Findings from the 2019 International Medical Informatics Association Yearbook Section on Health Information Management. Yearb Med Inform 2019; 28:65-68. [PMID: 31419817 PMCID: PMC6697497 DOI: 10.1055/s-0039-1677941] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022] Open
Abstract
OBJECTIVES To summarize the recent literature and research and present a selection of the best papers published in 2018 in the field of Health Information Management (HIM) and Health Informatics. METHODS A systematic review of the literature was performed, with the help of a medical librarian, by the two editors of the HIM section of the International Medical Informatics Association (IMIA) Yearbook. In order to include papers that would address the special theme of the 2019 Yearbook on artificial intelligence (AI) as well as HIM, we searched bibliographic databases for HIM-related papers with an AI focus using both Medical Subject Headings (MeSH) descriptors and keywords in titles and abstracts. A shortlist of 15 candidate best papers was first selected by section editors before being peer-reviewed by independent external reviewers. RESULTS While there were a significant number of manuscripts that addressed issues relevant to HIM, there were virtually none with MeSH headings indicating an HIM focus. Manuscripts that were considered related to the HIM field in terms of the practice of health information management as well as the profession included those that examined using machine learning and other AI approaches to identify protected health information in clinical text to aid with de-identification, automated coding approaches to translate free-text into standardized codes, and natural language processing approaches to extract clinical data to assist with populating cancer and other registries. CONCLUSIONS The papers discussed in the HIM section reflect the special theme of the use of AI in healthcare on issues particularly relevant to the field of HIM. This synopsis discusses these papers and recommends that HIM practitioners be more involved in research and that researchers in AI and related areas recognize the applicability and relevance of their work to the field of HIM.
Collapse
Affiliation(s)
| | - Eta S Berner
- Graduate Programs in Health Informatics, Department of Health Services Administration, University of Alabama at Birmingham, Birmingham, AL, USA
| | | |
Collapse
|