1
|
Kim Y, Kim JH, Lee JM, Jang MJ, Yum YJ, Kim S, Shin U, Kim YM, Joo HJ, Song S. A pre-trained BERT for Korean medical natural language processing. Sci Rep 2022; 12:13847. [PMID: 35974113 PMCID: PMC9381714 DOI: 10.1038/s41598-022-17806-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2021] [Accepted: 08/01/2022] [Indexed: 11/10/2022] Open
Abstract
With advances in deep learning and natural language processing (NLP), the analysis of medical texts is becoming increasingly important. Nonetheless, despite the importance of processing medical texts, no research on Korean medical-specific language models has been conducted. The Korean medical text is highly difficult to analyze because of the agglutinative characteristics of the language, as well as the complex terminologies in the medical domain. To solve this problem, we collected a Korean medical corpus and used it to train the language models. In this paper, we present a Korean medical language model based on deep learning NLP. The model was trained using the pre-training framework of BERT for the medical context based on a state-of-the-art Korean language model. The pre-trained model showed increased accuracies of 0.147 and 0.148 for the masked language model with next sentence prediction. In the intrinsic evaluation, the next sentence prediction accuracy improved by 0.258, which is a remarkable enhancement. In addition, the extrinsic evaluation of Korean medical semantic textual similarity data showed a 0.046 increase in the Pearson correlation, and the evaluation for the Korean medical named entity recognition showed a 0.053 increase in the F1-score.
Collapse
Affiliation(s)
- Yoojoong Kim
- School of Computer Science and Information Engineering, The Catholic University of Korea, Bucheon, Republic of Korea
| | - Jong-Ho Kim
- Korea University Research Institute for Medical Bigdata Science, Korea University, Seoul, Republic of Korea.,Department of Cardiology, Cardiovascular Center, Korea University College of Medicine, Seoul, Republic of Korea
| | - Jeong Moon Lee
- Korea University Research Institute for Medical Bigdata Science, Korea University, Seoul, Republic of Korea
| | - Moon Joung Jang
- Korea University Research Institute for Medical Bigdata Science, Korea University, Seoul, Republic of Korea
| | - Yun Jin Yum
- Korea University Research Institute for Medical Bigdata Science, Korea University, Seoul, Republic of Korea.,Department of Biostatistics, Korea University College of Medicine, Seoul, Republic of Korea
| | - Seongtae Kim
- Department of Linguistics, Korea University, Seoul, Republic of Korea
| | - Unsub Shin
- Department of Linguistics, Korea University, Seoul, Republic of Korea
| | - Young-Min Kim
- School of Interdisciplinary Industrial Studies, Hanyang University, Seoul, Republic of Korea
| | - Hyung Joon Joo
- Korea University Research Institute for Medical Bigdata Science, Korea University, Seoul, Republic of Korea.,Department of Cardiology, Cardiovascular Center, Korea University College of Medicine, Seoul, Republic of Korea.,Department of Medical Informatics, Korea University College of Medicine, Seoul, Republic of Korea
| | - Sanghoun Song
- Department of Linguistics, Korea University, Seoul, Republic of Korea.
| |
Collapse
|