1
|
Rana S, Hosen MJ, Tonni TJ, Rony MAH, Fatema K, Hasan MZ, Rahman MT, Khan RT, Jan T, Whaiduzzaman M. DeepChestGNN: A Comprehensive Framework for Enhanced Lung Disease Identification through Advanced Graphical Deep Features. SENSORS (BASEL, SWITZERLAND) 2024; 24:2830. [PMID: 38732936 PMCID: PMC11086108 DOI: 10.3390/s24092830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Revised: 04/06/2024] [Accepted: 04/16/2024] [Indexed: 05/13/2024]
Abstract
Lung diseases are the third-leading cause of mortality in the world. Due to compromised lung function, respiratory difficulties, and physiological complications, lung disease brought on by toxic substances, pollution, infections, or smoking results in millions of deaths every year. Chest X-ray images pose a challenge for classification due to their visual similarity, leading to confusion among radiologists. To imitate those issues, we created an automated system with a large data hub that contains 17 datasets of chest X-ray images for a total of 71,096, and we aim to classify ten different disease classes. For combining various resources, our large datasets contain noise and annotations, class imbalances, data redundancy, etc. We conducted several image pre-processing techniques to eliminate noise and artifacts from images, such as resizing, de-annotation, CLAHE, and filtering. The elastic deformation augmentation technique also generates a balanced dataset. Then, we developed DeepChestGNN, a novel medical image classification model utilizing a deep convolutional neural network (DCNN) to extract 100 significant deep features indicative of various lung diseases. This model, incorporating Batch Normalization, MaxPooling, and Dropout layers, achieved a remarkable 99.74% accuracy in extensive trials. By combining graph neural networks (GNNs) with feedforward layers, the architecture is very flexible when it comes to working with graph data for accurate lung disease classification. This study highlights the significant impact of combining advanced research with clinical application potential in diagnosing lung diseases, providing an optimal framework for precise and efficient disease identification and classification.
Collapse
Affiliation(s)
- Shakil Rana
- Health Informatics Research Laboratory (HIRL), Department of Computer Science and Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (S.R.); (M.J.H.); (T.J.T.); (M.A.H.R.); (K.F.); (M.Z.H.)
| | - Md Jabed Hosen
- Health Informatics Research Laboratory (HIRL), Department of Computer Science and Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (S.R.); (M.J.H.); (T.J.T.); (M.A.H.R.); (K.F.); (M.Z.H.)
| | - Tasnim Jahan Tonni
- Health Informatics Research Laboratory (HIRL), Department of Computer Science and Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (S.R.); (M.J.H.); (T.J.T.); (M.A.H.R.); (K.F.); (M.Z.H.)
| | - Md. Awlad Hossen Rony
- Health Informatics Research Laboratory (HIRL), Department of Computer Science and Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (S.R.); (M.J.H.); (T.J.T.); (M.A.H.R.); (K.F.); (M.Z.H.)
| | - Kaniz Fatema
- Health Informatics Research Laboratory (HIRL), Department of Computer Science and Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (S.R.); (M.J.H.); (T.J.T.); (M.A.H.R.); (K.F.); (M.Z.H.)
| | - Md. Zahid Hasan
- Health Informatics Research Laboratory (HIRL), Department of Computer Science and Engineering, Daffodil International University, Dhaka 1207, Bangladesh; (S.R.); (M.J.H.); (T.J.T.); (M.A.H.R.); (K.F.); (M.Z.H.)
| | - Md. Tanvir Rahman
- School of Health and Rehabilitation Sciences, The University of Queensland, St. Lucia, QLD 4072, Australia
- Department of Information and Communication Technology, Mawlana Bhashani Science and Technology University, Tangail 1902, Bangladesh
| | - Risala Tasin Khan
- Institute of Information Technology, Jahangirnagar University, Dhaka 1342, Bangladesh;
| | - Tony Jan
- Centre for Artificial Intelligence Research and Optimisation (AIRO), Torrens University, Ultimo, NSW 2007, Australia;
| | - Md Whaiduzzaman
- Centre for Artificial Intelligence Research and Optimisation (AIRO), Torrens University, Ultimo, NSW 2007, Australia;
- School of Information Systems, Queensland University of Technology, Brisbane, QLD 4000, Australia
| |
Collapse
|
2
|
Xie J, Wang Y, Sheng Q, Liu X, Li J, Sun F, Wang Y, Li S, Li Y, Yu Y, Yu G. Identification of mycoplasma pneumonia in children based on fusion of multi-modal clinical free-text description and structured test data. Health Informatics J 2024; 30:14604582241255818. [PMID: 38779978 DOI: 10.1177/14604582241255818] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/25/2024]
Abstract
Mycoplasma pneumonia may lead to hospitalizations and pose life-threatening risks in children. The automated identification of mycoplasma pneumonia from electronic medical records holds significant potential for improving the efficiency of hospital resource allocation. In this study, we proposed a novel method for identifying mycoplasma pneumonia by integrating multi-modal features derived from both free-text descriptions and structured test data in electronic medical records. Our approach begins with the extraction of free-text and structured data from clinical records through a systematic preprocessing pipeline. Subsequently, we employ a pre-trained transformer language model to extract features from the free-text, while multiple additive regression trees are used to transform features from the structured data. An attention-based fusion mechanism is then applied to integrate these multi-modal features for effective classification. We validated our method using clinic records of 7157 patients, retrospectively collected for training and testing purposes. The experimental results demonstrate that our proposed multi-modal fusion approach achieves significant improvements over other methods across four key performance metrics.
Collapse
Affiliation(s)
- Jingna Xie
- The Children's Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Yingshuo Wang
- The Children's Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Qiuyang Sheng
- Deepwise Healthcare Artificial Intelligence Laboratory, Beijing, China
| | - Xiaoqing Liu
- Deepwise Healthcare Artificial Intelligence Laboratory, Beijing, China
| | - Jing Li
- The Children's Hospital, Zhejiang University School of Medicine, Hangzhou, China; Sino-Finland Joint AI Laboratory for Child Health of Zhejiang Province, Hangzhou, China
| | - Fenglei Sun
- Deepwise Healthcare Artificial Intelligence Laboratory, Beijing, China
| | - Yuqi Wang
- The Children's Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Shuxian Li
- The Children's Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Yiming Li
- Deepwise Healthcare Artificial Intelligence Laboratory, Beijing, China
| | - Yizhou Yu
- Deepwise Healthcare Artificial Intelligence Laboratory, Beijing, China; Department of Computer Science, The University of Hong Kong, Hong Kong, China
| | - Gang Yu
- The Children's Hospital, Zhejiang University School of Medicine, Hangzhou, China; Sino-Finland Joint AI Laboratory for Child Health of Zhejiang Province, Hangzhou, China; Polytechnic Institute, Zhejiang University, Hangzhou, China
| |
Collapse
|
3
|
Ying H, Liu X, Zhang M, Ren Y, Zhen S, Wang X, Liu B, Hu P, Duan L, Cai M, Jiang M, Cheng X, Gong X, Jiang H, Jiang J, Zheng J, Zhu K, Zhou W, Lu B, Zhou H, Shen Y, Du J, Ying M, Hong Q, Mo J, Li J, Ye G, Zhang S, Hu H, Sun J, Liu H, Li Y, Xu X, Bai H, Wang S, Cheng X, Xu X, Jiao L, Yu R, Lau WY, Yu Y, Cai X. A multicenter clinical AI system study for detection and diagnosis of focal liver lesions. Nat Commun 2024; 15:1131. [PMID: 38326351 PMCID: PMC10850133 DOI: 10.1038/s41467-024-45325-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2023] [Accepted: 01/22/2024] [Indexed: 02/09/2024] Open
Abstract
Early and accurate diagnosis of focal liver lesions is crucial for effective treatment and prognosis. We developed and validated a fully automated diagnostic system named Liver Artificial Intelligence Diagnosis System (LiAIDS) based on a diverse sample of 12,610 patients from 18 hospitals, both retrospectively and prospectively. In this study, LiAIDS achieved an F1-score of 0.940 for benign and 0.692 for malignant lesions, outperforming junior radiologists (benign: 0.830-0.890, malignant: 0.230-0.360) and being on par with senior radiologists (benign: 0.920-0.950, malignant: 0.550-0.650). Furthermore, with the assistance of LiAIDS, the diagnostic accuracy of all radiologists improved. For benign and malignant lesions, junior radiologists' F1-scores improved to 0.936-0.946 and 0.667-0.680 respectively, while seniors improved to 0.950-0.961 and 0.679-0.753. Additionally, in a triage study of 13,192 consecutive patients, LiAIDS automatically classified 76.46% of patients as low risk with a high NPV of 99.0%. The evidence suggests that LiAIDS can serve as a routine diagnostic tool and enhance the diagnostic capabilities of radiologists for liver lesions.
Collapse
Affiliation(s)
- Hanning Ying
- Department of General Surgery, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Xiaoqing Liu
- Deepwise Artificial Intelligence Laboratory, Beijing, China
| | - Min Zhang
- College of Computer Science and Technology, Zhejiang University, Hangzhou, China
| | - Yiyue Ren
- School of Medicine, Zhejiang University, Hangzhou, China
| | - Shihui Zhen
- School of Medicine, Zhejiang University, Hangzhou, China
| | - Xiaojie Wang
- School of Medicine, Zhejiang University, Hangzhou, China
| | - Bo Liu
- Deepwise Artificial Intelligence Laboratory, Beijing, China
| | - Peng Hu
- Department of Radiology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Lian Duan
- Department of General Surgery, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Mingzhi Cai
- Zhangzhou Municipal Hospital of Fujian Province, Zhangzhou, China
| | | | - Xiangdong Cheng
- Cancer Hospital of the University of Chinese Academy of Sciences (ZheJiang Cancer Hospital), Hangzhou, China
| | | | - Haitao Jiang
- Cancer Hospital of the University of Chinese Academy of Sciences (ZheJiang Cancer Hospital), Hangzhou, China
| | - Jianshuai Jiang
- Department of Hepatopancreatobiliary Surgery, Ningbo First Hospital, Ningbo, China
| | - Jianjun Zheng
- Hwa Mei Hospital, University of Chinese Academy of Sciences (Ningbo No.2 Hospital), Ningbo, China
| | - Kelei Zhu
- Department of Hepatopancreatobiliary Surgery, Yinzhou People's Hospital, Ningbo, China
| | - Wei Zhou
- Department of Radiology, Huzhou Central Hospital, Affiliated Central Hospital of Huzhou University, Huzhou, China
| | - Baochun Lu
- Shaoxing People's Hospital, Shaoxing, China
| | - Hongkun Zhou
- The First Hospital of Jiaxing Affiliated Hospital of Jiaxing University, Jiaxing, China
| | - Yiyu Shen
- The Second Hospital of Jiaxing Affiliated Hospital of Jiaxing University, Jiaxing, China
| | - Jinlin Du
- Jinhua Municipal Central Hospital, Jinhua, China
| | | | | | - Jingang Mo
- Taizhou Municipal Central Hospital, Taizhou, China
| | - Jianfeng Li
- The First People's Hospital of Wenling, Taizhou, China
| | | | - Shizheng Zhang
- Department of Radiology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Hongjie Hu
- Department of Radiology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Jihong Sun
- Department of Radiology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Hui Liu
- Central Laboratory of Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Yiming Li
- Deepwise Artificial Intelligence Laboratory, Beijing, China
| | - Xingxin Xu
- Deepwise Artificial Intelligence Laboratory, Beijing, China
| | - Huiping Bai
- Deepwise Artificial Intelligence Laboratory, Beijing, China
| | - Shuxin Wang
- Deepwise Artificial Intelligence Laboratory, Beijing, China
| | | | - Xiaoyin Xu
- Brigham and Women' Hospital, Harvard Medical School, Boston, MA, USA.
| | - Long Jiao
- Faculty of Medicine, Imperial College London, London, UK.
| | - Risheng Yu
- Department of Radiology, Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, China.
| | - Wan Yee Lau
- Faculty of Medicine, the Chinese University of Hong Kong, Hong Kong, China.
| | - Yizhou Yu
- Department of Computer Science, The University of Hong Kong, Hong Kong, China.
| | - Xiujun Cai
- Department of General Surgery, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China.
| |
Collapse
|
4
|
Yang W, Wu W, Wang L, Zhang S, Zhao J, Qiang Y. PMSG-Net: A priori-guided multilevel graph transformer fusion network for immunotherapy efficacy prediction. Comput Biol Med 2023; 164:107371. [PMID: 37586204 DOI: 10.1016/j.compbiomed.2023.107371] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Revised: 07/24/2023] [Accepted: 08/12/2023] [Indexed: 08/18/2023]
Abstract
In the case of specific immunotherapy regimens and access to pre-treatment CT scans, developing reliable, interpretable intelligent image biomarkers to predict efficacy is essential for physician decision-making and patient treatment selection. However, varying levels of prognosis show a similar appearance on CT scans. It becomes challenging to stratify patients by a single pre-treatment CT scan when presenting subtle differences in images for experienced experts and existing prognostic classification methods. In addition, the pattern of peri-tumoural radiological structures also determines the patient's response to ICIs. Therefore, it is essential to develop a method that focuses on the clinical priori features of the tumour edges but also makes full use of the rich information within the 3D tumour. This paper proposes a priori-guided multilevel graph transformer fusion network (PMSG-Net). Specifically, a graph convolutional network is first used to obtain a feature representation of the tumour edge, and complementary information from that detailed representation is used to enhance the global representation. In the tumour global representation branch (MSGNet), we designed the cascaded scale-enhanced swin transformer to obtain attributes of graph nodes, and efficiently learn and model spatial dependencies and semantic connections at different scales through multi-hop context-aware attention (MCA), yielding a richer global semantic representation. To our knowledge, this is the first attempt to use graph neural networks to predict the efficacy of immunotherapy, and the experimental results show that this method outperforms the current mainstream methods.
Collapse
Affiliation(s)
- Wanting Yang
- College of Information and Computer, Taiyuan University of Technology, 030000, Taiyuan, Shanxi, China
| | - Wei Wu
- Department of Clinical Laboratory, Affiliated People's Hospital of Shanxi Medical University, Shanxi Provincial People's Hospital, Taiyuan, Shanxi, China
| | - Long Wang
- Jinzhong College of Information, 030600, Taiyuan, Shanxi, China
| | - Shuming Zhang
- College of Information and Computer, Taiyuan University of Technology, 030000, Taiyuan, Shanxi, China
| | - Juanjuan Zhao
- College of Software, Taiyuan University of Technology, 030000, Taiyuan, Shanxi, China
| | - Yan Qiang
- College of Information and Computer, Taiyuan University of Technology, 030000, Taiyuan, Shanxi, China.
| |
Collapse
|
5
|
Zhou HY, Yu Y, Wang C, Zhang S, Gao Y, Pan J, Shao J, Lu G, Zhang K, Li W. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nat Biomed Eng 2023:10.1038/s41551-023-01045-x. [PMID: 37308585 DOI: 10.1038/s41551-023-01045-x] [Citation(s) in RCA: 42] [Impact Index Per Article: 42.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Accepted: 04/26/2023] [Indexed: 06/14/2023]
Abstract
During the diagnostic process, clinicians leverage multimodal information, such as the chief complaint, medical images and laboratory test results. Deep-learning models for aiding diagnosis have yet to meet this requirement of leveraging multimodal information. Here we report a transformer-based representation-learning model as a clinical diagnostic aid that processes multimodal input in a unified manner. Rather than learning modality-specific features, the model leverages embedding layers to convert images and unstructured and structured text into visual tokens and text tokens, and uses bidirectional blocks with intramodal and intermodal attention to learn holistic representations of radiographs, the unstructured chief complaint and clinical history, and structured clinical information such as laboratory test results and patient demographic information. The unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of pulmonary disease (by 12% and 9%, respectively) and in the prediction of adverse clinical outcomes in patients with COVID-19 (by 29% and 7%, respectively). Unified multimodal transformer-based models may help streamline the triaging of patients and facilitate the clinical decision-making process.
Collapse
Affiliation(s)
- Hong-Yu Zhou
- Department of Computer Science, The University of Hong Kong, Pokfulam, China
| | - Yizhou Yu
- Department of Computer Science, The University of Hong Kong, Pokfulam, China.
| | - Chengdi Wang
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, China.
| | - Shu Zhang
- AI Lab, Deepwise Healthcare, Beijing, China
| | | | - Jia Pan
- Department of Computer Science, The University of Hong Kong, Pokfulam, China
| | - Jun Shao
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, China
| | - Guangming Lu
- Department of Medical Imaging, Jinling Hospital, Nanjing University School of Medicine, Nanjing, China
| | - Kang Zhang
- Zhuhai International Eye Center and Provincial Key Laboratory of Tumor Interventional Diagnosis and Treatment, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology and University Hospital, Guangdong, China.
- Department of Big Data and Biomedical Artificial Intelligence, National Biomedical Imaging Center, College of Future Technology, Peking University, Beijing, China.
- Clinical Translational Research Center, West China Hospital, Sichuan University, Chengdu, China.
| | - Weimin Li
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, China.
| |
Collapse
|
6
|
Improving brain tumor classification performance with an effective approach based on new deep learning model named 3ACL from 3D MRI data. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104424] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
|
7
|
Hu Z, Wang B, Pan X, Cao D, Gao A, Yang X, Chen Y, Lin Z. Using deep learning to distinguish malignant from benign parotid tumors on plain computed tomography images. Front Oncol 2022; 12:919088. [PMID: 35978811 PMCID: PMC9376440 DOI: 10.3389/fonc.2022.919088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Accepted: 06/28/2022] [Indexed: 11/13/2022] Open
Abstract
Objectives Evaluating the diagnostic efficiency of deep-learning models to distinguish malignant from benign parotid tumors on plain computed tomography (CT) images. Materials and methods The CT images of 283 patients with parotid tumors were enrolled and analyzed retrospectively. Of them, 150 were benign and 133 were malignant according to pathology results. A total of 917 regions of interest of parotid tumors were cropped (456 benign and 461 malignant). Three deep-learning networks (ResNet50, VGG16_bn, and DenseNet169) were used for diagnosis (approximately 3:1 for training and testing). The diagnostic efficiencies (accuracy, sensitivity, specificity, and area under the curve [AUC]) of three networks were calculated and compared based on the 917 images. To simulate the process of human diagnosis, a voting model was developed at the end of the networks and the 283 tumors were classified as benign or malignant. Meanwhile, 917 tumor images were classified by two radiologists (A and B) and original CT images were classified by radiologist B. The diagnostic efficiencies of the three deep-learning network models (after voting) and the two radiologists were calculated. Results For the 917 CT images, ResNet50 presented high accuracy and sensitivity for diagnosing malignant parotid tumors; the accuracy, sensitivity, specificity, and AUC were 90.8%, 91.3%, 90.4%, and 0.96, respectively. For the 283 tumors, the accuracy, sensitivity, and specificity of ResNet50 (after voting) were 92.3%, 93.5% and 91.2%, respectively. Conclusion ResNet50 presented high sensitivity in distinguishing malignant from benign parotid tumors on plain CT images; this made it a promising auxiliary diagnostic method to screen malignant parotid tumors.
Collapse
Affiliation(s)
- Ziyang Hu
- Department of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Medical School of Nanjing University, Nanjing, China
| | - Baixin Wang
- School of Electronic Science and Engineering, Nanjing University, Nanjing, China
| | - Xiao Pan
- Department of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Medical School of Nanjing University, Nanjing, China
| | - Dantong Cao
- Department of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Medical School of Nanjing University, Nanjing, China
| | - Antian Gao
- Department of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Medical School of Nanjing University, Nanjing, China
| | - Xudong Yang
- Department of Oral and Maxillofacial Surgery, Nanjing Stomatological Hospital, Medical School of Nanjing University, Nanjing, China
- *Correspondence: Zitong Lin, ; Ying Chen, ; Xudong Yang,
| | - Ying Chen
- School of Electronic Science and Engineering, Nanjing University, Nanjing, China
- *Correspondence: Zitong Lin, ; Ying Chen, ; Xudong Yang,
| | - Zitong Lin
- Department of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Medical School of Nanjing University, Nanjing, China
- *Correspondence: Zitong Lin, ; Ying Chen, ; Xudong Yang,
| |
Collapse
|