51
|
Lu T, Fang Y, Liu H, Chen C, Li T, Lu M, Song D. Comparison of Machine Learning and Logic Regression Algorithms for Predicting Lymph Node Metastasis in Patients with Gastric Cancer: A two-Center Study. Technol Cancer Res Treat 2024; 23:15330338231222331. [PMID: 38190617 PMCID: PMC10775719 DOI: 10.1177/15330338231222331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2023] [Revised: 11/01/2023] [Accepted: 11/20/2023] [Indexed: 01/10/2024] Open
Abstract
OBJECTIVES This two-center study aimed to establish a model for predicting the risk of lymph node metastasis in gastric cancer patients using machine learning (ML) and logistic regression (LR) algorithms, and to evaluate its predictive performance in clinical practice. METHODS Data of a total of 369 patients who underwent radical gastrectomy in the Department of General Surgery of Affiliated Hospital of Xuzhou Medical University (Xuzhou, China) from March 2016 to November 2019 were collected and retrospectively analyzed as the training group. In addition, data of 123 patients who underwent radical gastrectomy in the Department of General Surgery of Jining First People's Hospital (Jining, China) were collected and analyzed as the verification group. Besides, 7 ML and logistic models were developed, including decision tree, random forest, support vector machine (SVM), gradient boosting machine (GBM), naive Bayes, neural network, and LR, in order to evaluate the occurrence of lymph node metastasis in patients with gastric cancer. The ML model was established following 10 cross-validation iterations within the training dataset, and subsequently, each model was assessed using the test dataset. The model's performance was evaluated by comparing the area under the receiver operating characteristic curve of each model. RESULTS Compared with the traditional logistic model, among the 7 ML algorithms, except for SVM, the other models exhibited higher accuracy and reliability, and the influences of various risk factors on the model were more intuitive. CONCLUSION For the prediction of lymph node metastasis in gastric cancer patients, the ML algorithm outperformed traditional LR, and the GBM algorithm exhibited the most robust predictive capability.
Collapse
Affiliation(s)
- Tong Lu
- Department of emergency medicine, Jining No.1 People's Hospital, Jining, China
| | - Yu Fang
- Jiangsu Normal University, Xuzhou, China
| | - Haonan Liu
- Department of Oncology, The Affiliated Hospital of Xuzhou Medical University, Xuzhou, China
| | - Chong Chen
- Department of Gastroenterology, Xuzhou No.1 People's Hospital, Xuzhou, China
| | - Taotao Li
- Department of emergency medicine, Jining No.1 People's Hospital, Jining, China
| | - Miao Lu
- Wuxi Mental Health Center, Wuxi, China
| | - Daqing Song
- Department of emergency medicine, Jining No.1 People's Hospital, Jining, China
| |
Collapse
|
52
|
Azad R, Kazerouni A, Heidari M, Aghdam EK, Molaei A, Jia Y, Jose A, Roy R, Merhof D. Advances in medical image analysis with vision Transformers: A comprehensive review. Med Image Anal 2024; 91:103000. [PMID: 37883822 DOI: 10.1016/j.media.2023.103000] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Revised: 09/30/2023] [Accepted: 10/11/2023] [Indexed: 10/28/2023]
Abstract
The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.
Collapse
Affiliation(s)
- Reza Azad
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Amirhossein Kazerouni
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Moein Heidari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | | | - Amirali Molaei
- School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Yiwei Jia
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Abin Jose
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Rijo Roy
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Dorit Merhof
- Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany; Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| |
Collapse
|
53
|
Huang Z, Ding Y, Yang Y, Zhao S, Zhang S, Xiao J, Ding C, Guo N, Li Z, Zhou S, Cao G, Wang X. Performance of machine learning-based coronary computed tomography angiography for selecting revascularization candidates. Acta Radiol 2024; 65:123-132. [PMID: 36847335 DOI: 10.1177/02841851231158730] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/01/2023]
Abstract
BACKGROUND Limited studies have investigated the accuracy of therapeutic decision-making using machine learning-based coronary computed tomography angiography (ML-CCTA) compared with CCTA. PURPOSE To investigate the performance of ML-CCTA for therapeutic decision compared with CCTA. MATERIAL AND METHODS The study population consisted of 322 consecutive patients with stable coronary artery disease. The SYNTAX score was calculated with an online calculator based on ML-CCTA results. Therapeutic decision-making was determined by ML-CCTA results and the ML-CCTA-based SYNTAX score. The therapeutic strategy and the appropriate revascularization procedure were selected using ML-CCTA, CCTA, and invasive coronary angiography (ICA) independently. RESULTS The sensitivity, specificity, positive predictive value, negative predictive value, accuracy of ML-CCTA and CCTA for selecting revascularization candidates were 87.01%, 96.43%, 95.71%, 89.01%, 91.93%, and 85.71%, 87.50%, 86.27%, 86.98%, 86.65%, respectively, using ICA as the standard reference. The area under the receiver operating characteristic curve (AUC) of ML-CCTA for selecting revascularization candidates was significantly higher than CCTA (0.917 vs. 0.866, P = 0.016). Subgroup analysis showed the AUC of ML-CCTA for selecting percutaneous coronary intervention (PCI) or coronary artery bypass graft (CABG) candidates was significantly higher than CCTA (0.883 vs. 0.777, P < 0.001, 0.912 vs. 0.826, P = 0.003, respectively). CONCLUSION ML-CCTA could distinguish between patients who need revascularization and those who do not. In addition, ML-CCTA showed a slightly superior to CCTA in making an appropriate decision for patients and selecting a suitable revascularization strategy.
Collapse
Affiliation(s)
- Zengfa Huang
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Yi Ding
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Yang Yang
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Shengchao Zhao
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Shutong Zhang
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Jianwei Xiao
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Chengyu Ding
- Shukun (Beijing) Technology Co., Ltd., Beijing, China
| | - Ning Guo
- Shukun (Beijing) Technology Co., Ltd., Beijing, China
| | - Zuoqin Li
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Shiguang Zhou
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Guijuan Cao
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Xiang Wang
- Department of Radiology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| |
Collapse
|
54
|
Wang C, Piao S, Huang Z, Gao Q, Zhang J, Li Y, Shan H. Joint learning framework of cross-modal synthesis and diagnosis for Alzheimer's disease by mining underlying shared modality information. Med Image Anal 2024; 91:103032. [PMID: 37995628 DOI: 10.1016/j.media.2023.103032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 08/31/2023] [Accepted: 11/13/2023] [Indexed: 11/25/2023]
Abstract
Alzheimer's disease (AD) is one of the most common neurodegenerative disorders presenting irreversible progression of cognitive impairment. How to identify AD as early as possible is critical for intervention with potential preventive measures. Among various neuroimaging modalities used to diagnose AD, functional positron emission tomography (PET) has higher sensitivity than structural magnetic resonance imaging (MRI), but it is also costlier and often not available in many hospitals. How to leverage massive unpaired unlabeled PET to improve the diagnosis performance of AD from MRI becomes rather important. To address this challenge, this paper proposes a novel joint learning framework of unsupervised cross-modal synthesis and AD diagnosis by mining underlying shared modality information, improving the AD diagnosis from MRI while synthesizing more discriminative PET images. We mine underlying shared modality information in two aspects: diversifying modality information through the cross-modal synthesis network and locating critical diagnosis-related patterns through the AD diagnosis network. First, to diversify the modality information, we propose a novel unsupervised cross-modal synthesis network, which implements the inter-conversion between 3D PET and MRI in a single model modulated by the AdaIN module. Second, to locate shared critical diagnosis-related patterns, we propose an interpretable diagnosis network based on fully 2D convolutions, which takes either 3D synthesized PET or original MRI as input. Extensive experimental results on the ADNI dataset show that our framework can synthesize more realistic images, outperform the state-of-the-art AD diagnosis methods, and have better generalization on external AIBL and NACC datasets.
Collapse
Affiliation(s)
- Chenhui Wang
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai 200433, China
| | - Sirong Piao
- Department of Radiology, Huashan Hospital, Fudan University, Shanghai 200040, China
| | - Zhizhong Huang
- Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai 200433, China; School of Computer Science, Fudan University, Shanghai 200433, China
| | - Qi Gao
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai 200433, China
| | - Junping Zhang
- Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai 200433, China; School of Computer Science, Fudan University, Shanghai 200433, China
| | - Yuxin Li
- Department of Radiology, Huashan Hospital, Fudan University, Shanghai 200040, China.
| | - Hongming Shan
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai 200433, China; MOE Frontiers Center for Brain Science, Fudan University, Shanghai, 200433, China; MOE Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Fudan University, Shanghai, 200433, China; Shanghai Center for Brain Science and Brain-inspired Technology, Shanghai 201210, China.
| |
Collapse
|
55
|
Hammer S, Nunes DW, Hammer M, Zeman F, Akers M, Götz A, Balla A, Doppler MC, Fellner C, Platz Batista da Silva N, Thurn S, Verloh N, Stroszczynski C, Wohlgemuth WA, Palm C, Uller W. Deep learning-based differentiation of peripheral high-flow and low-flow vascular malformations in T2-weighted short tau inversion recovery MRI. Clin Hemorheol Microcirc 2024; 87:221-235. [PMID: 38306026 DOI: 10.3233/ch-232071] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/03/2024]
Abstract
BACKGROUND Differentiation of high-flow from low-flow vascular malformations (VMs) is crucial for therapeutic management of this orphan disease. OBJECTIVE A convolutional neural network (CNN) was evaluated for differentiation of peripheral vascular malformations (VMs) on T2-weighted short tau inversion recovery (STIR) MRI. METHODS 527 MRIs (386 low-flow and 141 high-flow VMs) were randomly divided into training, validation and test set for this single-center study. 1) Results of the CNN's diagnostic performance were compared with that of two expert and four junior radiologists. 2) The influence of CNN's prediction on the radiologists' performance and diagnostic certainty was evaluated. 3) Junior radiologists' performance after self-training was compared with that of the CNN. RESULTS Compared with the expert radiologists the CNN achieved similar accuracy (92% vs. 97%, p = 0.11), sensitivity (80% vs. 93%, p = 0.16) and specificity (97% vs. 100%, p = 0.50). In comparison to the junior radiologists, the CNN had a higher specificity and accuracy (97% vs. 80%, p < 0.001; 92% vs. 77%, p < 0.001). CNN assistance had no significant influence on their diagnostic performance and certainty. After self-training, the junior radiologists' specificity and accuracy improved and were comparable to that of the CNN. CONCLUSIONS Diagnostic performance of the CNN for differentiating high-flow from low-flow VM was comparable to that of expert radiologists. CNN did not significantly improve the simulated daily practice of junior radiologists, self-training was more effective.
Collapse
Affiliation(s)
- Simone Hammer
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Danilo Weber Nunes
- Regensburg Medical Image Computing (ReMIC), Ostbayerische Technische Hochschule Regensburg (OTH Regensburg), Regensburg, Germany
| | - Michael Hammer
- Regensburg Medical Image Computing (ReMIC), Ostbayerische Technische Hochschule Regensburg (OTH Regensburg), Regensburg, Germany
| | - Florian Zeman
- Faculty of Medicine, Center for Clinical Trials, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Michael Akers
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Andrea Götz
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Annika Balla
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Michael Christian Doppler
- Department of Diagnostic and Interventional Radiology, Faculty of Medicine, Medical Center University of Freiburg, University of Freiburg, Freiburg, Germany
| | - Claudia Fellner
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Natascha Platz Batista da Silva
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Sylvia Thurn
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Niklas Verloh
- Department of Diagnostic and Interventional Radiology, Faculty of Medicine, Medical Center University of Freiburg, University of Freiburg, Freiburg, Germany
| | - Christian Stroszczynski
- Department of Radiology, Faculty of Medicine, Medical Center University of Regensburg, University of Regensburg, Regensburg, Germany
| | - Walter Alexander Wohlgemuth
- Department of Radiology, Faculty of Medicine, Medical Center University of Halle (Saale), University of Halle (Saale), Halle, Germany
| | - Christoph Palm
- Regensburg Medical Image Computing (ReMIC), Ostbayerische Technische Hochschule Regensburg (OTH Regensburg), Regensburg, Germany
- Regensburg Center of Biomedical Engineering (RCBE), OTH Regensburg and University of Regensburg, Regensburg, Germany
| | - Wibke Uller
- Department of Diagnostic and Interventional Radiology, Faculty of Medicine, Medical Center University of Freiburg, University of Freiburg, Freiburg, Germany
| |
Collapse
|
56
|
Kasuya S, Inaoka T, Wada A, Nakatsuka T, Nakagawa K, Terada H. Feasibility of the fat-suppression image-subtraction method using deep learning for abnormality detection on knee MRI. Pol J Radiol 2023; 88:e562-e573. [PMID: 38362017 PMCID: PMC10867951 DOI: 10.5114/pjr.2023.133660] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Accepted: 09/04/2023] [Indexed: 02/17/2024] Open
Abstract
Purpose To evaluate the feasibility of using a deep learning (DL) model to generate fat-suppression images and detect abnormalities on knee magnetic resonance imaging (MRI) through the fat-suppression image-subtraction method. Material and methods A total of 45 knee MRI studies in patients with knee disorders and 12 knee MRI studies in healthy volunteers were enrolled. The DL model was developed using 2-dimensional convolutional neural networks for generating fat-suppression images and subtracting generated fat-suppression images without any abnormal findings from those with normal/abnormal findings and detecting/classifying abnormalities on knee MRI. The image qualities of the generated fat-suppression images and subtraction-images were assessed. The accuracy, average precision, average recall, F-measure, sensitivity, and area under the receiver operator characteristic curve (AUROC) of DL for each abnormality were calculated. Results A total of 2472 image datasets, each consisting of one slice of original T1WI, original intermediate-weighted images, generated fat-suppression (FS)-intermediate-weighted images without any abnormal findings, generated FS-intermediate-weighted images with normal/abnormal findings, and subtraction images between the generated FS-intermediate-weighted images at the same cross-section, were created. The generated fat-suppression images were of adequate image quality. Of the 2472 subtraction-images, 2203 (89.1%) were judged to be of adequate image quality. The accuracies for overall abnormalities, anterior cruciate ligament, bone marrow, cartilage, meniscus, and others were 89.5-95.1%. The average precision, average recall, and F-measure were 73.4-90.6%, 77.5-89.4%, and 78.4-89.4%, respectively. The sensitivity was 57.4-90.5%. The AUROCs were 0.910-0.979. Conclusions The DL model was able to generate fat-suppression images of sufficient quality to detect abnormalities on knee MRI through the fat-suppression image-subtraction method.
Collapse
Affiliation(s)
- Shusuke Kasuya
- Department of Radiology, Toho University Sakura Medical Center, Sakura, Japan
| | - Tsutomu Inaoka
- Department of Radiology, Toho University Sakura Medical Center, Sakura, Japan
| | - Akihiko Wada
- Department of Radiology, Juntendo University, Tokyo, Japan
| | - Tomoya Nakatsuka
- Department of Radiology, Toho University Sakura Medical Center, Sakura, Japan
| | - Koichi Nakagawa
- Department of Orthopaedic Surgery, Toho University Sakura Medical Center, Sakura, Japan
| | - Hitoshi Terada
- Department of Radiology, Toho University Sakura Medical Center, Sakura, Japan
| |
Collapse
|
57
|
Zhang J, Santos C, Park C, Mazurowski MA, Colglazier R. Improving Image Classification of Knee Radiographs: An Automated Image Labeling Approach. J Digit Imaging 2023; 36:2402-2410. [PMID: 37620710 PMCID: PMC10584746 DOI: 10.1007/s10278-023-00894-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Revised: 07/28/2023] [Accepted: 07/31/2023] [Indexed: 08/26/2023] Open
Abstract
Large numbers of radiographic images are available in musculoskeletal radiology practices which could be used for training of deep learning models for diagnosis of knee abnormalities. However, those images do not typically contain readily available labels due to limitations of human annotations. The purpose of our study was to develop an automated labeling approach that improves the image classification model to distinguish normal knee images from those with abnormalities or prior arthroplasty. The automated labeler was trained on a small set of labeled data to automatically label a much larger set of unlabeled data, further improving the image classification performance for knee radiographic diagnosis. We used BioBERT and EfficientNet as the feature extraction backbone of the labeler and imaging model, respectively. We developed our approach using 7382 patients and validated it on a separate set of 637 patients. The final image classification model, trained using both manually labeled and pseudo-labeled data, had the higher weighted average AUC (WA-AUC 0.903) value and higher AUC values among all classes (normal AUC 0.894; abnormal AUC 0.896, arthroplasty AUC 0.990) compared to the baseline model (WA-AUC = 0.857; normal AUC 0.842; abnormal AUC 0.848, arthroplasty AUC 0.987), trained using only manually labeled data. Statistical tests show that the improvement is significant on normal (p value < 0.002), abnormal (p value < 0.001), and WA-AUC (p value = 0.001). Our findings demonstrated that the proposed automated labeling approach significantly improves the performance of image classification for radiographic knee diagnosis, allowing for facilitating patient care and curation of large knee datasets.
Collapse
Affiliation(s)
- Jikai Zhang
- Department of Electrical and Computer Engineering, Duke University, Room 10070, 2424 Erwin Road, Durham, NC, 27705, USA.
| | - Carlos Santos
- Wake Forest University, Winston-Salem, NC, 27109, USA
| | - Christine Park
- Department of Radiology, Duke University Medical Center, Durham, NC, USA
| | - Maciej A Mazurowski
- Department of Radiology, Duke University Medical Center, Durham, NC, USA
- Department of Electrical and Computer Engineering, Department of Biostatistics and Bioinformatics, Department of Computer Science, Duke University, Durham, NC, USA
| | - Roy Colglazier
- Department of Radiology, Duke University Medical Center, Durham, NC, USA
| |
Collapse
|
58
|
Ma Y, Li Q. An integrated model combined intra- and peritumoral regions for predicting chemoradiation response of non small cell lung cancers based on radiomics and deep learning. Cancer Radiother 2023; 27:705-711. [PMID: 37932182 DOI: 10.1016/j.canrad.2023.05.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2023] [Revised: 05/22/2023] [Accepted: 05/24/2023] [Indexed: 11/08/2023]
Abstract
PURPOSE The purpose of this study was to develop a model for predicting chemoradiation response in non-small cell lung cancer (NSCLC) patients by integrating radiomics and deep-learning features and combined intra- and peritumoral regions with pre-treated CT images. MATERIALS AND METHODS This study enrolled 462 patients with NSCLC who received chemoradiation. On the basis of pretreated CT images, we developed three models to compare the prediction of chemoradiation: intratumoral, peritumoral and combined regions. To further illustrate each model, we established different feature integration methods: a) radiomics model with 1500 features; b) deep learning model with a multiple instance learning algorithm; c) integrated model by integrating radiomic and deep learning features. For radiomics and integrated models, support vector machine and the least absolute shrinkage and selection operator were used to extract and select features. Transfer learning and max pooling algorithms were used to identify high informative features in deep learning models. We applied ten-fold cross validation in model training and testing. RESULTS The best area under the curve (AUC) of intratumoral, peritumoral and combined models were 0.89 (95% CI, 0.74-0.93), 0.86 (95% CI, 0.75-0.92) and 0.92 (95% CI, 0.81-0.95), respectively. It indicated the importance of the peritumoral region for treatment response prediction and should be used in combination with the intratumoral region. Integrated models gave better results than models with radiomics and deep learning features alone in all regions of interest and radiomics models outperformed deep learning models in any comparative models. CONCLUSIONS The model that integrate radiomic and deep learning features and combined intra- and peritumoral regions provide valuable information in predicting treatment response of chemoradiation. It can help oncologists customize personalized clinical treatment plans for NSCLC patients.
Collapse
Affiliation(s)
- Y Ma
- The First Affiliated Hospital of China Medical University, Department of Pathology, 110001 Shenyang, China.
| | - Q Li
- The First Affiliated Hospital of China Medical University, Department of Pathology, 110001 Shenyang, China.
| |
Collapse
|
59
|
Wolf D, Payer T, Lisson CS, Lisson CG, Beer M, Götz M, Ropinski T. Self-supervised pre-training with contrastive and masked autoencoder methods for dealing with small datasets in deep learning for medical imaging. Sci Rep 2023; 13:20260. [PMID: 37985685 PMCID: PMC10662445 DOI: 10.1038/s41598-023-46433-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Accepted: 10/31/2023] [Indexed: 11/22/2023] Open
Abstract
Deep learning in medical imaging has the potential to minimize the risk of diagnostic errors, reduce radiologist workload, and accelerate diagnosis. Training such deep learning models requires large and accurate datasets, with annotations for all training samples. However, in the medical imaging domain, annotated datasets for specific tasks are often small due to the high complexity of annotations, limited access, or the rarity of diseases. To address this challenge, deep learning models can be pre-trained on large image datasets without annotations using methods from the field of self-supervised learning. After pre-training, small annotated datasets are sufficient to fine-tune the models for a specific task. The most popular self-supervised pre-training approaches in medical imaging are based on contrastive learning. However, recent studies in natural image processing indicate a strong potential for masked autoencoder approaches. Our work compares state-of-the-art contrastive learning methods with the recently introduced masked autoencoder approach "SparK" for convolutional neural networks (CNNs) on medical images. Therefore, we pre-train on a large unannotated CT image dataset and fine-tune on several CT classification tasks. Due to the challenge of obtaining sufficient annotated training data in medical imaging, it is of particular interest to evaluate how the self-supervised pre-training methods perform when fine-tuning on small datasets. By experimenting with gradually reducing the training dataset size for fine-tuning, we find that the reduction has different effects depending on the type of pre-training chosen. The SparK pre-training method is more robust to the training dataset size than the contrastive methods. Based on our results, we propose the SparK pre-training for medical imaging tasks with only small annotated datasets.
Collapse
Affiliation(s)
- Daniel Wolf
- Visual Computing Research Group, Institute of Media Informatics, Ulm University, Ulm, Germany.
- Experimental Radiology Research Group, Department for Diagnostic and Interventional Radiology, Ulm University Medical Center, Ulm, Germany.
| | - Tristan Payer
- Visual Computing Research Group, Institute of Media Informatics, Ulm University, Ulm, Germany
| | - Catharina Silvia Lisson
- Experimental Radiology Research Group, Department for Diagnostic and Interventional Radiology, Ulm University Medical Center, Ulm, Germany
| | - Christoph Gerhard Lisson
- Experimental Radiology Research Group, Department for Diagnostic and Interventional Radiology, Ulm University Medical Center, Ulm, Germany
| | - Meinrad Beer
- Experimental Radiology Research Group, Department for Diagnostic and Interventional Radiology, Ulm University Medical Center, Ulm, Germany
| | - Michael Götz
- Experimental Radiology Research Group, Department for Diagnostic and Interventional Radiology, Ulm University Medical Center, Ulm, Germany
| | - Timo Ropinski
- Visual Computing Research Group, Institute of Media Informatics, Ulm University, Ulm, Germany
| |
Collapse
|
60
|
Zsidai B, Hilkert AS, Kaarre J, Narup E, Senorski EH, Grassi A, Ley C, Longo UG, Herbst E, Hirschmann MT, Kopf S, Seil R, Tischer T, Samuelsson K, Feldt R. A practical guide to the implementation of AI in orthopaedic research - part 1: opportunities in clinical application and overcoming existing challenges. J Exp Orthop 2023; 10:117. [PMID: 37968370 PMCID: PMC10651597 DOI: 10.1186/s40634-023-00683-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 10/21/2023] [Indexed: 11/17/2023] Open
Abstract
Artificial intelligence (AI) has the potential to transform medical research by improving disease diagnosis, clinical decision-making, and outcome prediction. Despite the rapid adoption of AI and machine learning (ML) in other domains and industry, deployment in medical research and clinical practice poses several challenges due to the inherent characteristics and barriers of the healthcare sector. Therefore, researchers aiming to perform AI-intensive studies require a fundamental understanding of the key concepts, biases, and clinical safety concerns associated with the use of AI. Through the analysis of large, multimodal datasets, AI has the potential to revolutionize orthopaedic research, with new insights regarding the optimal diagnosis and management of patients affected musculoskeletal injury and disease. The article is the first in a series introducing fundamental concepts and best practices to guide healthcare professionals and researcher interested in performing AI-intensive orthopaedic research studies. The vast potential of AI in orthopaedics is illustrated through examples involving disease- or injury-specific outcome prediction, medical image analysis, clinical decision support systems and digital twin technology. Furthermore, it is essential to address the role of human involvement in training unbiased, generalizable AI models, their explainability in high-risk clinical settings and the implementation of expert oversight and clinical safety measures for failure. In conclusion, the opportunities and challenges of AI in medicine are presented to ensure the safe and ethical deployment of AI models for orthopaedic research and clinical application. Level of evidence IV.
Collapse
Affiliation(s)
- Bálint Zsidai
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden.
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.
| | - Ann-Sophie Hilkert
- Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden
- Medfield Diagnostics AB, Gothenburg, Sweden
| | - Janina Kaarre
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- Department of Orthopaedic Surgery, UPMC Freddie Fu Sports Medicine Center, University of Pittsburgh, Pittsburgh, USA
| | - Eric Narup
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Eric Hamrin Senorski
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Health and Rehabilitation, Institute of Neuroscience and Physiology, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- Sportrehab Sports Medicine Clinic, Gothenburg, Sweden
| | - Alberto Grassi
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- IIa Clinica Ortopedica E Traumatologica, IRCCS Istituto Ortopedico Rizzoli, Bologna, Italy
| | - Christophe Ley
- Department of Mathematics, University of Luxembourg, Esch-Sur-Alzette, Luxembourg
| | - Umile Giuseppe Longo
- Department of Orthopaedic and Trauma Surgery, Campus Bio-Medico University, Rome, Italy
| | - Elmar Herbst
- Department of Trauma, Hand and Reconstructive Surgery, University Hospital Münster, Münster, Germany
| | - Michael T Hirschmann
- Department of Orthopedic Surgery and Traumatology, Head Knee Surgery and DKF Head of Research, Kantonsspital Baselland, 4101, Bruderholz, Switzerland
| | - Sebastian Kopf
- Center of Orthopaedics and Traumatology, University Hospital Brandenburg a.d.H., Brandenburg Medical School Theodor Fontane, 14770, Brandenburg a.d.H., Germany
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, 14770, Brandenburg a.d.H., Germany
| | - Romain Seil
- Department of Orthopaedic Surgery, Centre Hospitalier Luxembourg and Luxembourg Institute of Health, Luxembourg, Luxembourg
| | - Thomas Tischer
- Clinic for Orthopaedics and Trauma Surgery, Malteser Waldkrankenhaus St. Marien, Erlangen, Germany
| | - Kristian Samuelsson
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- Department of Orthopaedics, Sahlgrenska University Hospital, Mölndal, Sweden
| | - Robert Feldt
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| |
Collapse
|
61
|
Lee S, Jeon U, Lee JH, Kang S, Kim H, Lee J, Chung MJ, Cha HS. Artificial intelligence for the detection of sacroiliitis on magnetic resonance imaging in patients with axial spondyloarthritis. Front Immunol 2023; 14:1278247. [PMID: 38022576 PMCID: PMC10676202 DOI: 10.3389/fimmu.2023.1278247] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Accepted: 10/25/2023] [Indexed: 12/01/2023] Open
Abstract
Background Magnetic resonance imaging (MRI) is important for the early detection of axial spondyloarthritis (axSpA). We developed an artificial intelligence (AI) model for detecting sacroiliitis in patients with axSpA using MRI. Methods This study included MRI examinations of patients who underwent semi-coronal MRI scans of the sacroiliac joints owing to chronic back pain with short tau inversion recovery (STIR) sequences between January 2010 and December 2021. Sacroiliitis was defined as a positive MRI finding according to the ASAS classification criteria for axSpA. We developed a two-stage framework. First, the Faster R-CNN network extracted regions of interest (ROIs) to localize the sacroiliac joints. Maximum intensity projection (MIP) of three consecutive slices was used to mimic the reading of two adjacent slices. Second, the VGG-19 network determined the presence of sacroiliitis in localized ROIs. We augmented the positive dataset six-fold. The sacroiliitis classification performance was measured using the sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). The prediction models were evaluated using three-round three-fold cross-validation. Results A total of 296 participants with 4,746 MRI slices were included in the study. Sacroiliitis was identified in 864 MRI slices of 119 participants. The mean sensitivity, specificity, and AUROC for the detection of sacroiliitis were 0.725 (95% CI, 0.705-0.745), 0.936 (95% CI, 0.924-0.947), and 0.830 (95%CI, 0.792-0.868), respectively, at the image level and 0.947 (95% CI, 0.912-0.982), 0.691 (95% CI, 0.603-0.779), and 0.816 (95% CI, 0.776-0.856), respectively, at the patient level. In the original model, without using MIP and dataset augmentation, the mean sensitivity, specificity, and AUROC were 0.517 (95% CI, 0.493-0.780), 0.944 (95% CI, 0.933-0.955), and 0.731 (95% CI, 0.681-0.780), respectively, at the image level and 0.806 (95% CI, 0.729-0.883), 0.617 (95% CI, 0.523-0.711), and 0.711 (95% CI, 0.660-0.763), respectively, at the patient level. The performance was improved by MIP techniques and data augmentation. Conclusion An AI model was developed for the detection of sacroiliitis using MRI, compatible with the ASAS criteria for axSpA, with the potential to aid MRI application in a wider clinical setting.
Collapse
Affiliation(s)
- Seulkee Lee
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Uju Jeon
- Medical AI Research Center, Samsung Medical Center, Seoul, Republic of Korea
| | - Ji Hyun Lee
- Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Seonyoung Kang
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Hyungjin Kim
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Jaejoon Lee
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Myung Jin Chung
- Medical AI Research Center, Samsung Medical Center, Seoul, Republic of Korea
- Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Suwon, Republic of Korea
| | - Hoon-Suk Cha
- Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| |
Collapse
|
62
|
Hu Y, Chen Y, Qin Y, Huang R. Learning entity-oriented representation for biomedical relation extraction. J Biomed Inform 2023; 147:104527. [PMID: 37852347 DOI: 10.1016/j.jbi.2023.104527] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 10/11/2023] [Accepted: 10/15/2023] [Indexed: 10/20/2023]
Abstract
Biomedical Relation Extraction (BioRE) aims to automatically extract semantic relations for given entity pairs and is of great significance in biomedical research. Current popular methods often utilize pretrained language models to extract semantic features from individual input instances, which frequently suffer from overlapping semantics. Overlapping semantics refers to the situation in which a sentence contains multiple entity pairs that share the same context, leading to highly similar information between these entity pairs. In this study, we propose a model for learning Entity-oriented Representation (EoR) that aims to improve the performance of the model by enhancing the discriminability between entity pairs. It contains three modules: sentence representation, entity-oriented representation, and output. The first module learns the global semantic information of the input instance; the second module focuses on extracting the semantic information of the sentence from the target entities; and the third module enhances distinguishability among entity pairs and classifies the relation type. We evaluated our approach on four BioRE tasks with eight datasets, and the experiments showed that our EoR achieved state-of-the-art performance for PPI, DDI, CPI, and DPI tasks. Further analysis demonstrated the benefits of entity-oriented semantic information in handling multiple entity pairs in the BioRE task.
Collapse
Affiliation(s)
- Ying Hu
- Text Computing and Cognitive Intelligence Engineering Research Center of National Education Ministry, State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, China.
| | - Yanping Chen
- Text Computing and Cognitive Intelligence Engineering Research Center of National Education Ministry, State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, China.
| | - Yongbin Qin
- Text Computing and Cognitive Intelligence Engineering Research Center of National Education Ministry, State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, China.
| | - Ruizhang Huang
- Text Computing and Cognitive Intelligence Engineering Research Center of National Education Ministry, State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, China.
| |
Collapse
|
63
|
Li F, Zhai P, Yang C, Feng G, Yang J, Yuan Y. Automated diagnosis of anterior cruciate ligament via a weighted multi-view network. Front Bioeng Biotechnol 2023; 11:1268543. [PMID: 37885456 PMCID: PMC10598377 DOI: 10.3389/fbioe.2023.1268543] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2023] [Accepted: 09/22/2023] [Indexed: 10/28/2023] Open
Abstract
Objective: To build a three-dimensional (3D) deep learning-based computer-aided diagnosis (CAD) system and investigate its applicability for automatic detection of anterior cruciate ligament (ACL) of the knee joint in magnetic resonance imaging (MRI). Methods: In this study, we develop a 3D weighted multi-view convolutional neural network by fusing different views of MRI to detect ACL. The network is evaluated on two MRI datasets, the in-house MRI-ACL dataset and the publicly available MRNet-v1.0 dataset. In the MRI-ACL dataset, the retrospective study collects 100 cases, and four views per patient are included. There are 50 ACL patients and 50 normal patients, respectively. The MRNet-v1.0 dataset contains 1,250 cases with three views, of which 208 are ACL patients, and the rest are normal or other abnormal patients. Results: The area under the receiver operating characteristic curve (AUC) of the ACL diagnosis system is 97.00% and 92.86% at the optimal threshold for the MRI-ACL dataset and the MRNet-v1.0 dataset, respectively, indicating a high overall diagnostic accuracy. In comparison, the best AUC of the single-view diagnosis methods are 96.00% (MRI-ACL dataset) and 91.78% (MRNet-v1.0 dataset), and our method improves by about 1.00% and 1.08%. Furthermore, our method also improves by about 1.00% (MRI-ACL dataset) and 0.28% (MRNet-v1.0 dataset) compared with the multi-view network (i.e., MRNet). Conclusion: The presented 3D weighted multi-view network achieves superior AUC in diagnosing ACL, not only in the in-house MRI-ACL dataset but also in the publicly available MRNet-v1.0 dataset, which demonstrates its clinical applicability for the automatic detection of ACL.
Collapse
Affiliation(s)
- Feng Li
- Orthopedic Department, Ningbo No. 2 Hospital, Ningbo, China
| | - Penghua Zhai
- Center for Pattern Recognition and Intelligent Medicine, Guoke Ningbo Life science and Health industry Research Institute, Ningbo, China
| | - Chao Yang
- Orthopedic Department, Ningbo No. 2 Hospital, Ningbo, China
| | - Gong Feng
- Orthopedic Department, Ningbo No. 2 Hospital, Ningbo, China
| | - Ji Yang
- Orthopedic Department, Ningbo No. 2 Hospital, Ningbo, China
| | - Yi Yuan
- Orthopedic Department, Ningbo No. 2 Hospital, Ningbo, China
| |
Collapse
|
64
|
Zhen T, Fang J, Hu D, Ruan M, Wang L, Fan S, Shen Q. Risk stratification by nomogram of deep learning radiomics based on multiparametric magnetic resonance imaging in knee meniscus injury. INTERNATIONAL ORTHOPAEDICS 2023; 47:2497-2505. [PMID: 37386277 DOI: 10.1007/s00264-023-05875-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Accepted: 06/19/2023] [Indexed: 07/01/2023]
Abstract
PURPOSE To construct and validate a nomogram model that integrated deep learning radiomic features based on multiparametric MRI and clinical features for risk stratification of meniscus injury. METHODS A total of 167 knee MR images were collected from two institutions. All patients were classified into two groups based on the MR diagnostic criteria proposed by Stoller et al. The automatic meniscus segmentation model was constructed through V-net. LASSO regression was performed to extract the optimal features correlated to risk stratification. A nomogram model was constructed by combining the Radscore and clinical features. The performance of the models was evaluated by ROC analysis and calibration curve. Subsequently, the model was simulated by junior doctors in order to test its practical application effect. RESULTS The Dice similarity coefficients of automatic meniscus segmentation models were all over 0.8. Eight optimal features, identified by LASSO regression, were employed to calculate the Radscore. The combined model showed a better performance in both the training cohort (AUC = 0.90, 95%CI: 0.84-0.95) and the validation cohort (AUC = 0.84, 95%CI: 0.72-0.93). The calibration curve indicated a better accuracy of the combined model than either the Radscore or clinical model alone. The simulation results showed that the diagnostic accuracy of junior doctors increased from 74.9 to 86.2% after using the model. CONCLUSION Deep learning V-net demonstrated great performance in automatic meniscus segmentation of the knee joint. It was reliable for stratifying the risk of meniscus injury of the knee by nomogram which integrated the Radscores and clinical features.
Collapse
Affiliation(s)
- Tao Zhen
- Department of Radiology, Affiliated Hangzhou First People's Hospital, Zhejiang University School of Medicine, No. 261, Huansha Road, Zhejiang, 310006, Hangzhou, China
| | - Jing Fang
- Zhejiang Provincial Hospital of Chinese Medicine, Hangzhou, 310006, China
| | - Dacheng Hu
- Department of Radiology, Affiliated Hangzhou First People's Hospital, Zhejiang University School of Medicine, No. 261, Huansha Road, Zhejiang, 310006, Hangzhou, China
| | - Mei Ruan
- Department of Radiology, Affiliated Hangzhou First People's Hospital, Zhejiang University School of Medicine, No. 261, Huansha Road, Zhejiang, 310006, Hangzhou, China
| | - Luoyu Wang
- Department of Radiology, Affiliated Hangzhou First People's Hospital, Zhejiang University School of Medicine, No. 261, Huansha Road, Zhejiang, 310006, Hangzhou, China
| | - Sandra Fan
- Department of Radiology, Affiliated Hangzhou First People's Hospital, Zhejiang University School of Medicine, No. 261, Huansha Road, Zhejiang, 310006, Hangzhou, China
| | - Qijun Shen
- Department of Radiology, Affiliated Hangzhou First People's Hospital, Zhejiang University School of Medicine, No. 261, Huansha Road, Zhejiang, 310006, Hangzhou, China.
| |
Collapse
|
65
|
Park HJ, Kim SH, Choi JY, Cha D. Human-machine cooperation meta-model for clinical diagnosis by adaptation to human expert's diagnostic characteristics. Sci Rep 2023; 13:16204. [PMID: 37758800 PMCID: PMC10533492 DOI: 10.1038/s41598-023-43291-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 09/21/2023] [Indexed: 09/29/2023] Open
Abstract
Artificial intelligence (AI) using deep learning approaches the capabilities of human experts in medical image diagnosis. However, due to liability issues in medical decisions, AI is often relegated to an assistant role. Based on this responsibility constraint, the effective use of AI to assist human intelligence in real-world clinics remains a challenge. Given the significant inter-individual variations in clinical decisions among physicians based on their expertise, AI needs to adapt to individual experts, complementing weaknesses and enhancing strengths. For this adaptation, AI should not only acquire domain knowledge but also understand the specific human experts it assists. This study introduces a meta-model for human-machine cooperation that first evaluates each expert's class-specific diagnostic tendencies using conditional probability, based on which the meta-model adjusts the AI's predictions. This meta-model was applied to ear disease diagnosis using otoendoscopy, highlighting improved performance when incorporating individual diagnostic characteristics, even with limited evaluation data. The highest accuracy was achieved by combining each expert's conditional probabilities with machine classification probability, using optimal weights specific to each individual's overall classification accuracy. This tailored model aims to mitigate potential misjudgments due to psychological effects caused by machine suggestions and to capitalize on the unique expertise of individual clinicians.
Collapse
Affiliation(s)
- Hae-Jeong Park
- Department of Nuclear Medicine, Department of Psychiatry, Graduate School of Medical Science, Brain Korea 21 Project, Yonsei University College of Medicine, Seoul, South Korea.
- Department of Cognitive Science, Yonsei University, Seoul, Republic of Korea.
- Center for Systems and Translational Brain Sciences, Institute of Human Complexity and Systems Science, Yonsei University, 50-1, Yonsei-ro, Sinchon-dong, Seodaemun-gu, Seoul, 03722, Republic of Korea.
| | - Sung Huhn Kim
- Department of Otorhinolaryngology, Yonsei University College of Medicine, Seoul, South Korea
| | - Jae Young Choi
- Department of Otorhinolaryngology, Yonsei University College of Medicine, Seoul, South Korea
| | - Dongchul Cha
- Department of Otorhinolaryngology, Yonsei University College of Medicine, Seoul, South Korea.
- Center for Innovative Medicine, Healthcare Lab, NAVER Corporation, 95, Jeongjail-ro, Bundang-gu, Seongnam-si, Gyeonggi-do, 13561, Republic of Korea.
- Healthcare Lab, Naver Cloud Corporation, Seongnam-si, Republic of Korea.
| |
Collapse
|
66
|
Liang C, Li X, Qin Y, Li M, Ma Y, Wang R, Xu X, Yu J, Lv S, Luo H. Effective automatic detection of anterior cruciate ligament injury using convolutional neural network with two attention mechanism modules. BMC Med Imaging 2023; 23:120. [PMID: 37697236 PMCID: PMC10494428 DOI: 10.1186/s12880-023-01091-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Accepted: 08/30/2023] [Indexed: 09/13/2023] Open
Abstract
BACKGROUND To develop a fully automated CNN detection system based on magnetic resonance imaging (MRI) for ACL injury, and to explore the feasibility of CNN for ACL injury detection on MRI images. METHODS Including 313 patients aged 16 - 65 years old, the raw data are 368 pieces with injured ACL and 100 pieces with intact ACL. By adding flipping, rotation, scaling and other methods to expand the data, the final data set is 630 pieces including 355 pieces of injured ACL and 275 pieces of intact ACL. Using the proposed CNN model with two attention mechanism modules, data sets are trained and tested with fivefold cross-validation. RESULTS The performance is evaluated using accuracy, precision, sensitivity, specificity and F1 score of our proposed CNN model, with results of 0.8063, 0.7741, 0.9268, 0.6509 and 0.8436. The average accuracy in the fivefold cross-validation is 0.8064. For our model, the average area under curves (AUC) for detecting injured ACL has results of 0.8886. CONCLUSION We propose an effective and automatic CNN model to detect ACL injury from MRI of human knees. This model can effectively help clinicians diagnose ACL injury, improving diagnostic efficiency and reducing misdiagnosis and missed diagnosis.
Collapse
Affiliation(s)
- Chen Liang
- Department of Minimally Invasive Surgery and Sports Medicine, The 2Nd Affiliated Hospital of Harbin Medical University, Harbin, 150001, China
| | - Xiang Li
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Yong Qin
- Department of Minimally Invasive Surgery and Sports Medicine, The 2Nd Affiliated Hospital of Harbin Medical University, Harbin, 150001, China
| | - Minglei Li
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Yingkai Ma
- Department of Minimally Invasive Surgery and Sports Medicine, The 2Nd Affiliated Hospital of Harbin Medical University, Harbin, 150001, China
| | - Ren Wang
- Department of Minimally Invasive Surgery and Sports Medicine, The 2Nd Affiliated Hospital of Harbin Medical University, Harbin, 150001, China
| | - Xiangning Xu
- Department of Minimally Invasive Surgery and Sports Medicine, The 2Nd Affiliated Hospital of Harbin Medical University, Harbin, 150001, China
| | - Jinping Yu
- Department of Minimally Invasive Surgery and Sports Medicine, The 2Nd Affiliated Hospital of Harbin Medical University, Harbin, 150001, China
| | - Songcen Lv
- Department of Minimally Invasive Surgery and Sports Medicine, The 2Nd Affiliated Hospital of Harbin Medical University, Harbin, 150001, China.
| | - Hao Luo
- Department of Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China.
| |
Collapse
|
67
|
Aubonnet R, Ramos J, Recenti M, Jacob D, Ciliberti F, Guerrini L, Gislason MK, Sigurjonsson O, Tsirilaki M, Jónsson H, Gargiulo P. Toward New Assessment of Knee Cartilage Degeneration. Cartilage 2023; 14:351-374. [PMID: 36541701 PMCID: PMC10601563 DOI: 10.1177/19476035221144746] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Revised: 11/09/2022] [Accepted: 11/25/2022] [Indexed: 12/24/2022] Open
Abstract
OBJECTIVE Assessment of human joint cartilage is a crucial tool to detect and diagnose pathological conditions. This exploratory study developed a workflow for 3D modeling of cartilage and bone based on multimodal imaging. New evaluation metrics were created and, a unique set of data was gathered from healthy controls and patients with clinically evaluated degeneration or trauma. DESIGN We present a novel methodology to evaluate knee bone and cartilage based on features extracted from magnetic resonance imaging (MRI) and computed tomography (CT) data. We developed patient specific 3D models of the tibial, femoral, and patellar bones and cartilages. Forty-seven subjects with a history of degenerative disease, traumatic events, or no symptoms or trauma (control group) were recruited in this study. Ninety-six different measurements were extracted from each knee, 78 2D and 18 3D measurements. We compare the sensitivity of different metrics to classify the cartilage condition and evaluate degeneration. RESULTS Selected features extracted show significant difference between the 3 groups. We created a cumulative index of bone properties that demonstrated the importance of bone condition to assess cartilage quality, obtaining the greatest sensitivity on femur within medial and femoropatellar compartments. We were able to classify degeneration with a maximum recall value of 95.9 where feature importance analysis showed a significant contribution of the 3D parameters. CONCLUSION The present work demonstrates the potential for improving sensitivity in cartilage assessment. Indeed, current trends in cartilage research point toward improving treatments and therefore our contribution is a first step toward sensitive and personalized evaluation of cartilage condition.
Collapse
Affiliation(s)
- Romain Aubonnet
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | - Jorgelina Ramos
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | - Marco Recenti
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | - Deborah Jacob
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | - Federica Ciliberti
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | - Lorena Guerrini
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | - Magnus K. Gislason
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | - Olafur Sigurjonsson
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
| | | | - Halldór Jónsson
- Landspitali, University Hospital of Iceland, Reykjavik, Iceland
- Medical Faculty, University of Iceland, Reykjavik, Iceland
| | - Paolo Gargiulo
- Institute of Biomedical and Neural Engineering, Reykjavik University, Reykjavik, Iceland
- Landspitali, University Hospital of Iceland, Reykjavik, Iceland
| |
Collapse
|
68
|
Liu S, Roemer F, Ge Y, Bedrick EJ, Li ZM, Guermazi A, Sharma L, Eaton C, Hochberg MC, Hunter DJ, Nevitt MC, Wirth W, Kent Kwoh C, Sun X. Comparison of evaluation metrics of deep learning for imbalanced imaging data in osteoarthritis studies. Osteoarthritis Cartilage 2023; 31:1242-1248. [PMID: 37209993 PMCID: PMC10524686 DOI: 10.1016/j.joca.2023.05.006] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Revised: 04/14/2023] [Accepted: 05/12/2023] [Indexed: 05/22/2023]
Abstract
PURPOSE To compare the evaluation metrics for deep learning methods that were developed using imbalanced imaging data in osteoarthritis studies. MATERIALS AND METHODS This retrospective study utilized 2996 sagittal intermediate-weighted fat-suppressed knee MRIs with MRI Osteoarthritis Knee Score readings from 2467 participants in the Osteoarthritis Initiative study. We obtained probabilities of the presence of bone marrow lesions (BMLs) from MRIs in the testing dataset at the sub-region (15 sub-regions), compartment, and whole-knee levels based on the trained deep learning models. We compared different evaluation metrics (e.g., receiver operating characteristic (ROC) and precision-recall (PR) curves) in the testing dataset with various class ratios (presence of BMLs vs. absence of BMLs) at these three data levels to assess the model's performance. RESULTS In a subregion with an extremely high imbalance ratio, the model achieved a ROC-AUC of 0.84, a PR-AUC of 0.10, a sensitivity of 0, and a specificity of 1. CONCLUSION The commonly used ROC curve is not sufficiently informative, especially in the case of imbalanced data. We provide the following practical suggestions based on our data analysis: 1) ROC-AUC is recommended for balanced data, 2) PR-AUC should be used for moderately imbalanced data (i.e., when the proportion of the minor class is above 5% and less than 50%), and 3) for severely imbalanced data (i.e., when the proportion of the minor class is below 5%), it is not practical to apply a deep learning model, even with the application of techniques addressing imbalanced data issues.
Collapse
Affiliation(s)
- Shen Liu
- Department of Epidemiology and Biostatistics, University of Arizona, 1295 N. Martin Ave., Tucson, AZ 85724, USA.
| | - Frank Roemer
- Department of Radiology, University of Erlangen - Nuremberg, Erlangen, Germany; Department of Radiology, Boston University School of Medicine, MA, USA.
| | - Yong Ge
- Department of Management Information Systems, University of Arizona, AZ, USA.
| | - Edward J Bedrick
- Department of Epidemiology and Biostatistics, University of Arizona, 1295 N. Martin Ave., Tucson, AZ 85724, USA.
| | - Zong-Ming Li
- University of Arizona Arthritis Center, University of Arizona College of Medicine, Tucson, AZ, USA.
| | - Ali Guermazi
- Department of Radiology, Boston University School of Medicine, MA, USA.
| | - Leena Sharma
- Feinberh School of Medicine, Northwestern University, IL, USA.
| | - Charles Eaton
- Kent Memorial Hospital, and Department of Family Medicine, Warren Alpert Medical School, and Department of Epidemiology, School of Public Health, Brown University, RI, USA.
| | - Marc C Hochberg
- School of Medicine, University of Maryland, and Medical Care Clinical Center, VA Maryland Health Care System, Baltimore, MD, USA.
| | - David J Hunter
- Sydney Musculoskeletal Health, Kolling Institute, Faculty of Medicine and Health, The University of Sydney, Sydney, 2065 NSW, Australia, and Rheumatology Department, Royal North Shore Hospital, St Leonards, NSW 2065 Australia.
| | - Michael C Nevitt
- Department of Epidemiology and Biostatistics, University of California San Francisco, CA, USA.
| | - Wolfgang Wirth
- Department of Imaging & Functional Musculoskeletal Research, Institute of Anatomy & Cell Biology, Paracelsus Medical University Salzburg & Nuremberg, Salzburg, Austria, and Ludwig Boltzmann Inst. for Arthritis and Rehabilitation, Paracelsus Medical University Salzburg & Nuremberg, Salzburg, Austria, and Chondrometrics GmbH, Ainring, Germany.
| | - C Kent Kwoh
- University of Arizona Arthritis Center, University of Arizona College of Medicine, Tucson, AZ, USA.
| | - Xiaoxiao Sun
- Department of Epidemiology and Biostatistics, University of Arizona, 1295 N. Martin Ave., Tucson, AZ 85724, USA.
| |
Collapse
|
69
|
Liu Z, Lv Q, Yang Z, Li Y, Lee CH, Shen L. Recent progress in transformer-based medical image analysis. Comput Biol Med 2023; 164:107268. [PMID: 37494821 DOI: 10.1016/j.compbiomed.2023.107268] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 05/30/2023] [Accepted: 07/16/2023] [Indexed: 07/28/2023]
Abstract
The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.
Collapse
Affiliation(s)
- Zhaoshan Liu
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| | - Qiujie Lv
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore; School of Intelligent Systems Engineering, Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, 518107, China.
| | - Ziduo Yang
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore; School of Intelligent Systems Engineering, Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, 518107, China.
| | - Yifan Li
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| | - Chau Hung Lee
- Department of Radiology, Tan Tock Seng Hospital, 11 Jalan Tan Tock Seng, Singapore, 308433, Singapore.
| | - Lei Shen
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| |
Collapse
|
70
|
Shetty ND, Dhande R, Unadkat BS, Parihar P. A Comprehensive Review on the Diagnosis of Knee Injury by Deep Learning-Based Magnetic Resonance Imaging. Cureus 2023; 15:e45730. [PMID: 37868582 PMCID: PMC10590246 DOI: 10.7759/cureus.45730] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 09/21/2023] [Indexed: 10/24/2023] Open
Abstract
The continual improvement in the field of medical diagnosis has led to the monopoly of using deep learning (DL)-based magnetic resonance imaging (MRI) for the diagnosis of knee injury related to meniscal injury, ligament injury including the cruciate ligaments, collateral ligaments and medial patella-femoral ligament, and cartilage injury. The present systematic review was done by PubMed and Directory of Open Access Journals (DOAJ), wherein we finalised 24 studies conducted on the accuracy of DL MRI studies for knee injury identification. The studies showed an accuracy of 72.5% to 100% indicating that DL MRI holds an equivalent performance as humans in decision-making and management of knee injuries. This further opens up future exploration for improving MRI-based diagnosis keeping in mind the limitations of verification bias and data imbalance in ground truth subjectivity.
Collapse
Affiliation(s)
- Neha D Shetty
- Department of Radiodiagnosis, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Rajasbala Dhande
- Department of Radiodiagnosis, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Bhavik S Unadkat
- Department of Radiodiagnosis, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| | - Pratapsingh Parihar
- Department of Radiodiagnosis, Datta Meghe Institute of Higher Education and Research, Wardha, IND
| |
Collapse
|
71
|
Chang R, Qi S, Wu Y, Yue Y, Zhang X, Guan Y, Qian W. Deep radiomic model based on the sphere-shell partition for predicting treatment response to chemotherapy in lung cancer. Transl Oncol 2023; 35:101719. [PMID: 37320871 PMCID: PMC10277572 DOI: 10.1016/j.tranon.2023.101719] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2022] [Revised: 05/16/2023] [Accepted: 06/08/2023] [Indexed: 06/17/2023] Open
Abstract
BACKGROUND The prognosis of chemotherapy is important in clinical decision-making for non-small cell lung cancer (NSCLC) patients. OBJECTIVES To develop a model for predicting treatment response to chemotherapy in NSCLC patients from pre-chemotherapy CT images. MATERIALS AND METHODS This retrospective multicenter study enrolled 485 patients with NSCLC who received chemotherapy alone as a first-line treatment. Two integrated models were developed using radiomic and deep-learning-based features. First, we partitioned pre-chemotherapy CT images into spheres and shells with different radii around the tumor (0-3, 3-6, 6-9, 9-12, 12-15 mm) containing intratumoral and peritumoral regions. Second, we extracted radiomic and deep-learning-based features from each partition. Third, using radiomic features, five sphere-shell models, one feature fusion model, and one image fusion model were developed. Finally, the model with the best performance was validated in two cohorts. RESULTS Among the five partitions, the model of 9-12 mm achieved the highest area under the curve (AUC) of 0.87 (95% confidence interval: 0.77-0.94). The AUC was 0.94 (0.85-0.98) for the feature fusion model and 0.91 (0.82-0.97) for the image fusion model. For the model integrating radiomic and deep-learning-based features, the AUC was 0.96 (0.88-0.99) for the feature fusion method and 0.94 (0.85-0.98) for the image fusion method. The best-performing model had an AUC of 0.91 (0.81-0.97) and 0.89 (0.79-0.93) in two validation sets, respectively. CONCLUSIONS This integrated model can predict the response to chemotherapy in NSCLC patients and assist physicians in clinical decision-making.
Collapse
Affiliation(s)
- Runsheng Chang
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Shouliang Qi
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China; Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China.
| | - Yanan Wu
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Yong Yue
- Department of Radiology, Shengjing Hospital of China Medical University, Shenyang, China
| | - Xiaoye Zhang
- Department of Oncology, Shengjing Hospital of China Medical University, Shenyang, China
| | - Yubao Guan
- Department of Radiology, The Fifth Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
| | - Wei Qian
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| |
Collapse
|
72
|
Debs P, Fayad LM. The promise and limitations of artificial intelligence in musculoskeletal imaging. FRONTIERS IN RADIOLOGY 2023; 3:1242902. [PMID: 37609456 PMCID: PMC10440743 DOI: 10.3389/fradi.2023.1242902] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 07/26/2023] [Indexed: 08/24/2023]
Abstract
With the recent developments in deep learning and the rapid growth of convolutional neural networks, artificial intelligence has shown promise as a tool that can transform several aspects of the musculoskeletal imaging cycle. Its applications can involve both interpretive and non-interpretive tasks such as the ordering of imaging, scheduling, protocoling, image acquisition, report generation and communication of findings. However, artificial intelligence tools still face a number of challenges that can hinder effective implementation into clinical practice. The purpose of this review is to explore both the successes and limitations of artificial intelligence applications throughout the muscuskeletal imaging cycle and to highlight how these applications can help enhance the service radiologists deliver to their patients, resulting in increased efficiency as well as improved patient and provider satisfaction.
Collapse
Affiliation(s)
- Patrick Debs
- The Russell H. Morgan Department of Radiology & Radiological Science, The Johns Hopkins Medical Institutions, Baltimore, MD, United States
| | - Laura M. Fayad
- The Russell H. Morgan Department of Radiology & Radiological Science, The Johns Hopkins Medical Institutions, Baltimore, MD, United States
- Department of Orthopaedic Surgery, Johns Hopkins University School of Medicine, Baltimore, MD, United States
- Department of Oncology, Johns Hopkins University School of Medicine, Baltimore, MD, United States
| |
Collapse
|
73
|
Recht MP, White LM, Fritz J, Resnick DL. Advances in Musculoskeletal Imaging: Recent Developments and Predictions for the Future. Radiology 2023; 308:e230615. [PMID: 37642575 DOI: 10.1148/radiol.230615] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/31/2023]
Affiliation(s)
- Michael P Recht
- From the Department of Radiology, NYU Grossman School of Medicine, 660 First Ave, 3rd Floor, New York, NY 10016 (M.P.R., J.F.); Department of Medical Imaging, University Health Network, Sinai Health System and Women's College Hospital, Toronto, Canada (L.M.W.); and Department of Radiology, UCSD Teleradiology and Education Center, La Jolla, Calif (D.L.R.)
| | - Lawrence M White
- From the Department of Radiology, NYU Grossman School of Medicine, 660 First Ave, 3rd Floor, New York, NY 10016 (M.P.R., J.F.); Department of Medical Imaging, University Health Network, Sinai Health System and Women's College Hospital, Toronto, Canada (L.M.W.); and Department of Radiology, UCSD Teleradiology and Education Center, La Jolla, Calif (D.L.R.)
| | - Jan Fritz
- From the Department of Radiology, NYU Grossman School of Medicine, 660 First Ave, 3rd Floor, New York, NY 10016 (M.P.R., J.F.); Department of Medical Imaging, University Health Network, Sinai Health System and Women's College Hospital, Toronto, Canada (L.M.W.); and Department of Radiology, UCSD Teleradiology and Education Center, La Jolla, Calif (D.L.R.)
| | - Donald L Resnick
- From the Department of Radiology, NYU Grossman School of Medicine, 660 First Ave, 3rd Floor, New York, NY 10016 (M.P.R., J.F.); Department of Medical Imaging, University Health Network, Sinai Health System and Women's College Hospital, Toronto, Canada (L.M.W.); and Department of Radiology, UCSD Teleradiology and Education Center, La Jolla, Calif (D.L.R.)
| |
Collapse
|
74
|
Yin L, Kong Y, Guo M, Zhang X, Yan W, Zhang H. A preliminary attempt to use radiomic features in the diagnosis of extra-articular long head biceps tendinitis. MAGMA (NEW YORK, N.Y.) 2023; 36:651-658. [PMID: 36449124 DOI: 10.1007/s10334-022-01050-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Revised: 11/13/2022] [Accepted: 11/15/2022] [Indexed: 06/17/2023]
Abstract
BACKGROUND This study aims to present a radiomic application in diagnosing the long head of biceps (LHB) tendinitis. Moreover, we evaluated whether machine learning-derived radiomic features recognize LHB tendinitis. PATIENTS AND METHODS A total of 170 patients were reviewed. All LHB tendinitis patients were diagnosed under arthroscopy. Radiomic features were extracted from preoperative magnetic resonance imaging (MRI), and the input dataset was divided into a training set and a test set. For feature selection, the t test and least absolute shrinkage and selection operator (LASSO) methods were used, and random forest (RF) and support vector machine (SVM) were used as machine learning classifiers. The sensitivity, specificity, accuracy, and area under the curve (AUC) of each model's receiver operating characteristic (ROC) curves were calculated to evaluate model performance. RESULTS In total, 851 radiomic features were extracted, with 109 radiomic features extracted using a t test and 20 radiomic features extracted using the LASSO method. The random forest classifier shows the highest sensitivity, specificity, accuracy, and AUC (0.52, 0.92, 0.73, and 0.72). CONCLUSION The classifier contract by 20 radiomic features demonstrated a good ability to predict extra-articular LHB tendinitis.However because of poor segmentation reliability, the value of Radiomic in LHB tendinitis still needs to be further explored.
Collapse
Affiliation(s)
- Lifeng Yin
- The First Affiliated Hospital of Chongqing Medical University, Yuzhong District, Chongqing, China
| | - Yanggang Kong
- The First Affiliated Hospital of Chongqing Medical University, Yuzhong District, Chongqing, China
| | - Mingkang Guo
- The First Affiliated Hospital of Chongqing Medical University, Yuzhong District, Chongqing, China
| | - Xingyu Zhang
- The First Affiliated Hospital of Chongqing Medical University, Yuzhong District, Chongqing, China
| | - Wenlong Yan
- The First Affiliated Hospital of Chongqing Medical University, Yuzhong District, Chongqing, China
| | - Hua Zhang
- The First Affiliated Hospital of Chongqing Medical University, Yuzhong District, Chongqing, China.
| |
Collapse
|
75
|
Han M, Singh M, Karimi D, Kim JY, Flannery SW, Ecklund K, Murray MM, Fleming BC, Gholipour A, Kiapour AM. LigaNET: A multi-modal deep learning approach to predict the risk of subsequent anterior cruciate ligament injury after surgery. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.07.25.23293102. [PMID: 37546855 PMCID: PMC10402234 DOI: 10.1101/2023.07.25.23293102] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/08/2023]
Abstract
Anterior cruciate ligament (ACL) injuries are a common cause of soft tissue injuries in young active individuals, leading to a significant risk of premature joint degeneration. Postoperative management of such injuries, in particular returning patients to athletic activities, is a challenge with immediate and long-term implications including the risk of subsequent injury. In this study, we present LigaNET, a multi-modal deep learning pipeline that predicts the risk of subsequent ACL injury following surgical treatment. Postoperative MRIs (n=1,762) obtained longitudinally between 3 to 24 months after ACL surgery from a cohort of 159 patients along with 11 non-imaging outcomes were used to train and test: 1) a 3D CNN to predict subsequent ACL injury from segmented ACLs, 2) a 3D CNN to predict injury from the whole MRI, 3) a logistic regression classifier predict injury from non-imaging data, and 4) a multi-modal pipeline by fusing the predictions of each classifier. The CNN using the segmented ACL achieved an accuracy of 77.6% and AUROC of 0.84, which was significantly better than the CNN using the whole knee MRI (accuracy: 66.6%, AUROC: 0.70; P<.001) and the non-imaging classifier (accuracy: 70.1%, AUROC: 0.75; P=.039). The fusion of all three classifiers resulted in highest classification performance (accuracy: 80.6%, AUROC: 0.89), which was significantly better than each individual classifier (P<.001). The developed multi-modal approach had similar performance in predicting the risk of subsequent ACL injury from any of the imaging sequences (P>.10). Our results demonstrate that a deep learning approach can achieve high performance in identifying patients at high risk of subsequent ACL injury after surgery and may be used in clinical decision making to improve postoperative management (e.g., safe return to sports) of ACL injured patients.
Collapse
Affiliation(s)
- Mo Han
- Department of Orthopaedic Surgery, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Mallika Singh
- Department of Orthopaedic Surgery, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Davood Karimi
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Jin-Young Kim
- Department of Orthopaedic Surgery, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Sean W. Flannery
- Department of Orthopaedics, Warren Alpert Medical School of Brown University, Rhode Island Hospital, 1 Hoppin St, Providence RI 02903, USA
| | - BEAR Trial Team
- Department of Orthopaedic Surgery, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Kirsten Ecklund
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Martha M. Murray
- Department of Orthopaedic Surgery, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Braden C. Fleming
- Department of Orthopaedics, Warren Alpert Medical School of Brown University, Rhode Island Hospital, 1 Hoppin St, Providence RI 02903, USA
| | - Ali Gholipour
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| | - Ata M. Kiapour
- Department of Orthopaedic Surgery, Boston Children’s Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA 02115, USA
| |
Collapse
|
76
|
Dubin JA, Bains SS, Chen Z, Hameed D, Nace J, Mont MA, Delanois RE. Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint Arthroplasty. J Arthroplasty 2023; 38:1195-1202. [PMID: 37040823 DOI: 10.1016/j.arth.2023.04.007] [Citation(s) in RCA: 41] [Impact Index Per Article: 41.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 03/22/2023] [Accepted: 04/03/2023] [Indexed: 04/13/2023] Open
Abstract
BACKGROUND Rapid technological advancements have laid the foundations for the use of artificial intelligence in medicine. The promise of machine learning (ML) lies in its potential ability to improve treatment decision making, predict adverse outcomes, and streamline the management of perioperative healthcare. In an increasing consumer-focused health care model, unprecedented access to information may extend to patients using ChatGPT to gain insight into medical questions. The main objective of our study was to replicate a patient's internet search in order to assess the appropriateness of ChatGPT, a novel machine learning tool released in 2022 that provides dialogue responses to queries, in comparison to Google Web Search, the most widely used search engine in the United States today, as a resource for patients for online health information. For the 2 different search engines, we compared i) the most frequently asked questions (FAQs) associated with total knee arthroplasty (TKA) and total hip arthroplasty (THA) by question type and topic; ii) the answers to the most frequently asked questions; as well as iii) the FAQs yielding a numerical response. METHODS A Google web search was performed with the following search terms: "total knee replacement" and "total hip replacement." These terms were individually entered and the first 10 FAQs were extracted along with the source of the associated website for each question. The following statements were inputted into ChatGPT: 1) "Perform a google search with the search term 'total knee replacement' and record the 10 most FAQs related to the search term" as well as 2) "Perform a google search with the search term 'total hip replacement' and record the 10 most FAQs related to the search term." A Google web search was repeated with the same search terms to identify the first 10 FAQs that included a numerical response for both "total knee replacement" and "total hip replacement." These questions were then inputted into ChatGPT and the questions and answers were recorded. RESULTS There were 5 of 20 (25%) questions that were similar when performing a Google web search and a search of ChatGPT for all search terms. Of the 20 questions asked for the Google Web Search, 13 of 20 were provided by commercial websites. For ChatGPT, 15 of 20 (75%) questions were answered by government websites, with the most frequent one being PubMed. In terms of numerical questions, 11 of 20 (55%) of the most FAQs provided different responses between a Google web search and ChatGPT. CONCLUSION A comparison of the FAQs by a Google web search with attempted replication by ChatGPT revealed heterogenous questions and responses for open and discrete questions. ChatGPT should remain a trending use as a potential resource to patients that needs further corroboration until its ability to provide credible information is verified and concordant with the goals of the physician and the patient alike.
Collapse
Affiliation(s)
- Jeremy A Dubin
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Sandeep S Bains
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Zhongming Chen
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Daniel Hameed
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - James Nace
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Michael A Mont
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Ronald E Delanois
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| |
Collapse
|
77
|
Guo D, Liu X, Wang D, Tang X, Qin Y. Development and clinical validation of deep learning for auto-diagnosis of supraspinatus tears. J Orthop Surg Res 2023; 18:426. [PMID: 37308995 DOI: 10.1186/s13018-023-03909-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/11/2023] [Accepted: 06/04/2023] [Indexed: 06/14/2023] Open
Abstract
BACKGROUND Accurately diagnosing supraspinatus tears based on magnetic resonance imaging (MRI) is challenging and time-combusting due to the experience level variability of the musculoskeletal radiologists and orthopedic surgeons. We developed a deep learning-based model for automatically diagnosing supraspinatus tears (STs) using shoulder MRI and validated its feasibility in clinical practice. MATERIALS AND METHODS A total of 701 shoulder MRI data (2804 images) were retrospectively collected for model training and internal test. An additional 69 shoulder MRIs (276 images) were collected from patients who underwent shoulder arthroplasty and constituted the surgery test set for clinical validation. Two advanced convolutional neural networks (CNN) based on Xception were trained and optimized to detect STs. The diagnostic performance of the CNN was evaluated according to its sensitivity, specificity, precision, accuracy, and F1 score. Subgroup analyses were performed to verify its robustness, and we also compared the CNN's performance with that of 4 radiologists and 4 orthopedic surgeons on the surgery and internal test sets. RESULTS Optimal diagnostic performance was achieved on the 2D model, from which F1-scores of 0.824 and 0.75, and areas under the ROC curves of 0.921 (95% confidence interval, 0.841-1.000) and 0.882 (0.817-0.947) were observed on the surgery and internal test sets. For the subgroup analysis, the 2D CNN model demonstrated a sensitivity of 0.33-1.000 and 0.625-1.000 for different degrees of tears on the surgery and internal test sets, and there was no significant performance difference between 1.5 and 3.0 T data. Compared with eight clinicians, the 2D CNN model exhibited better diagnostic performance than the junior clinicians and was equivalent to senior clinicians. CONCLUSIONS The proposed 2D CNN model realized the adequate and efficient automatic diagnoses of STs, which achieved a comparable performance of junior musculoskeletal radiologists and orthopedic surgeons. It might be conducive to assisting poor-experienced radiologists, especially in community scenarios lacking consulting experts.
Collapse
Affiliation(s)
- Deming Guo
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China
- Jilin Provincial Key Laboratory of Orhtopeadics, Changchun, People's Republic of China
| | - Xiaoning Liu
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China
| | - Dawei Wang
- Beijing Infervision Technology Co Ltd, Beijing, People's Republic of China
| | - Xiongfeng Tang
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China.
- Jilin Provincial Key Laboratory of Orhtopeadics, Changchun, People's Republic of China.
| | - Yanguo Qin
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China.
- Jilin Provincial Key Laboratory of Orhtopeadics, Changchun, People's Republic of China.
| |
Collapse
|
78
|
Nuutinen M, Leskelä RL. Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system. HEALTH AND TECHNOLOGY 2023; 13:1-14. [PMID: 37363342 PMCID: PMC10262137 DOI: 10.1007/s12553-023-00763-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Accepted: 06/01/2023] [Indexed: 06/28/2023]
Abstract
Background For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required. Objective The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS. Methods The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered. Results The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment. Conclusion The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices. Supplementary Information The online version contains supplementary material available at 10.1007/s12553-023-00763-1.
Collapse
Affiliation(s)
- Mikko Nuutinen
- Nordic Healthcare Group, Helsinki, Finland
- Haartman Institute, University of Helsinki, Helsinki, Finland
| | | |
Collapse
|
79
|
Mangone M, Diko A, Giuliani L, Agostini F, Paoloni M, Bernetti A, Santilli G, Conti M, Savina A, Iudicelli G, Ottonello C, Santilli V. A Machine Learning Approach for Knee Injury Detection from Magnetic Resonance Imaging. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2023; 20:6059. [PMID: 37372646 DOI: 10.3390/ijerph20126059] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 05/27/2023] [Accepted: 06/05/2023] [Indexed: 06/29/2023]
Abstract
The knee is an essential part of our body, and identifying its injuries is crucial since it can significantly affect quality of life. To date, the preferred way of evaluating knee injuries is through magnetic resonance imaging (MRI), which is an effective imaging technique that accurately identifies injuries. The issue with this method is that the high amount of detail that comes with MRIs is challenging to interpret and time consuming for radiologists to analyze. The issue becomes even more concerning when radiologists are required to analyze a significant number of MRIs in a short period. For this purpose, automated tools may become helpful to radiologists assisting them in the evaluation of these images. Machine learning methods, in being able to extract meaningful information from data, such as images or any other type of data, are promising for modeling the complex patterns of knee MRI and relating it to its interpretation. In this study, using a real-life imaging protocol, a machine-learning model based on convolutional neural networks used for detecting medial meniscus tears, bone marrow edema, and general abnormalities on knee MRI exams is presented. Furthermore, the model's effectiveness in terms of accuracy, sensitivity, and specificity is evaluated. Based on this evaluation protocol, the explored models reach a maximum accuracy of 83.7%, a maximum sensitivity of 82.2%, and a maximum specificity of 87.99% for meniscus tears. For bone marrow edema, a maximum accuracy of 81.3%, a maximum sensitivity of 93.3%, and a maximum specificity of 78.6% is reached. Finally, for general abnormalities, the explored models reach 83.7%, 90.0% and 84.2% of maximum accuracy, sensitivity and specificity, respectively.
Collapse
Affiliation(s)
- Massimiliano Mangone
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Anxhelo Diko
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
- Department of Computer Science Sapienza, University of Rome, 00198 Rome, Italy
| | - Luca Giuliani
- San Salvatore Hospital, Department of Biotechnological and Applied Clinical Sciences, University of L'Aquila, Vetoio Stree, 67100 L'Aquila, Italy
| | - Francesco Agostini
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Marco Paoloni
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Andrea Bernetti
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Gabriele Santilli
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Marco Conti
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Alessio Savina
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Giovanni Iudicelli
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| | - Carlo Ottonello
- Fisiocard Medical Centre, Via Francesco Tovaglieri 17, 00155 Rome, Italy
| | - Valter Santilli
- Department of Anatomical and Histological Sciences, Legal Medicine and Orthopedics, Sapienza University, 00185 Rome, Italy
| |
Collapse
|
80
|
Lin DJ, Schwier M, Geiger B, Raithel E, von Busch H, Fritz J, Kline M, Brooks M, Dunham K, Shukla M, Alaia EF, Samim M, Joshi V, Walter WR, Ellermann JM, Ilaslan H, Rubin D, Winalski CS, Recht MP. Deep Learning Diagnosis and Classification of Rotator Cuff Tears on Shoulder MRI. Invest Radiol 2023; 58:405-412. [PMID: 36728041 DOI: 10.1097/rli.0000000000000951] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/03/2023]
Abstract
BACKGROUND Detection of rotator cuff tears, a common cause of shoulder disability, can be time-consuming and subject to reader variability. Deep learning (DL) has the potential to increase radiologist accuracy and consistency. PURPOSE The aim of this study was to develop a prototype DL model for detection and classification of rotator cuff tears on shoulder magnetic resonance imaging into no tear, partial-thickness tear, or full-thickness tear. MATERIALS AND METHODS This Health Insurance Portability and Accountability Act-compliant, institutional review board-approved study included a total of 11,925 noncontrast shoulder magnetic resonance imaging scans from 2 institutions, with 11,405 for development and 520 dedicated for final testing. A DL ensemble algorithm was developed that used 4 series as input from each examination: fluid-sensitive sequences in 3 planes and a sagittal oblique T1-weighted sequence. Radiology reports served as ground truth for training with categories of no tear, partial tear, or full-thickness tear. A multireader study was conducted for the test set ground truth, which was determined by the majority vote of 3 readers per case. The ensemble comprised 4 parallel 3D ResNet50 convolutional neural network architectures trained via transfer learning and then adapted to the targeted domain. The final tear-type prediction was determined as the class with the highest probability, after averaging the class probabilities of the 4 individual models. RESULTS The AUC overall for supraspinatus, infraspinatus, and subscapularis tendon tears was 0.93, 0.89, and 0.90, respectively. The model performed best for full-thickness supraspinatus, infraspinatus, and subscapularis tears with AUCs of 0.98, 0.99, and 0.95, respectively. Multisequence input demonstrated higher AUCs than single-sequence input for infraspinatus and subscapularis tendon tears, whereas coronal oblique fluid-sensitive and multisequence input showed similar AUCs for supraspinatus tendon tears. Model accuracy for tear types and overall accuracy were similar to that of the clinical readers. CONCLUSIONS Deep learning diagnosis of rotator cuff tears is feasible with excellent diagnostic performance, particularly for full-thickness tears, with model accuracy similar to subspecialty-trained musculoskeletal radiologists.
Collapse
Affiliation(s)
- Dana J Lin
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | | | | | | | | | - Jan Fritz
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Mitchell Kline
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Michael Brooks
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Kevin Dunham
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Mehool Shukla
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Erin F Alaia
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Mohammad Samim
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Vivek Joshi
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - William R Walter
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| | - Jutta M Ellermann
- Department of Radiology, Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN
| | | | | | | | - Michael P Recht
- From the Department of Radiology, NYU Grossman School of Medicine, New York, NY
| |
Collapse
|
81
|
Winkler JK, Blum A, Kommoss K, Enk A, Toberer F, Rosenberger A, Haenssle HA. Assessment of Diagnostic Performance of Dermatologists Cooperating With a Convolutional Neural Network in a Prospective Clinical Study: Human With Machine. JAMA Dermatol 2023; 159:621-627. [PMID: 37133847 PMCID: PMC10157508 DOI: 10.1001/jamadermatol.2023.0905] [Citation(s) in RCA: 15] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Accepted: 03/05/2023] [Indexed: 05/04/2023]
Abstract
Importance Studies suggest that convolutional neural networks (CNNs) perform equally to trained dermatologists in skin lesion classification tasks. Despite the approval of the first neural networks for clinical use, prospective studies demonstrating benefits of human with machine cooperation are lacking. Objective To assess whether dermatologists benefit from cooperation with a market-approved CNN in classifying melanocytic lesions. Design, Setting, and Participants In this prospective diagnostic 2-center study, dermatologists performed skin cancer screenings using naked-eye examination and dermoscopy. Dermatologists graded suspect melanocytic lesions by the probability of malignancy (range 0-1, threshold for malignancy ≥0.5) and indicated management decisions (no action, follow-up, excision). Next, dermoscopic images of suspect lesions were assessed by a market-approved CNN, Moleanalyzer Pro (FotoFinder Systems). The CNN malignancy scores (range 0-1, threshold for malignancy ≥0.5) were transferred to dermatologists with the request to re-evaluate lesions and revise initial decisions in consideration of CNN results. Reference diagnoses were based on histopathologic examination in 125 (54.8%) lesions or, in the case of nonexcised lesions, on clinical follow-up data and expert consensus. Data were collected from October 2020 to October 2021. Main Outcomes and Measures Primary outcome measures were diagnostic sensitivity and specificity of dermatologists alone and dermatologists cooperating with the CNN. Accuracy and receiver operator characteristic area under the curve (ROC AUC) were considered as additional measures. Results A total of 22 dermatologists detected 228 suspect melanocytic lesions (190 nevi, 38 melanomas) in 188 patients (mean [range] age, 53.4 [19-91] years; 97 [51.6%] male patients). Diagnostic sensitivity and specificity significantly improved when dermatologists additionally integrated CNN results into decision-making (mean sensitivity from 84.2% [95% CI, 69.6%-92.6%] to 100.0% [95% CI, 90.8%-100.0%]; P = .03; mean specificity from 72.1% [95% CI, 65.3%-78.0%] to 83.7% [95% CI, 77.8%-88.3%]; P < .001; mean accuracy from 74.1% [95% CI, 68.1%-79.4%] to 86.4% [95% CI, 81.3%-90.3%]; P < .001; and mean ROC AUC from 0.895 [95% CI, 0.836-0.954] to 0.968 [95% CI, 0.948-0.988]; P = .005). In addition, the CNN alone achieved a comparable sensitivity, higher specificity, and higher diagnostic accuracy compared with dermatologists alone in classifying melanocytic lesions. Moreover, unnecessary excisions of benign nevi were reduced by 19.2%, from 104 (54.7%) of 190 benign nevi to 84 nevi when dermatologists cooperated with the CNN (P < .001). Most lesions were examined by dermatologists with 2 to 5 years (96, 42.1%) or less than 2 years of experience (78, 34.2%); others (54, 23.7%) were evaluated by dermatologists with more than 5 years of experience. Dermatologists with less dermoscopy experience cooperating with the CNN had the most diagnostic improvement compared with more experienced dermatologists. Conclusions and Relevance In this prospective diagnostic study, these findings suggest that dermatologists may improve their performance when they cooperate with the market-approved CNN and that a broader application of this human with machine approach could be beneficial for dermatologists and patients.
Collapse
Affiliation(s)
- Julia K. Winkler
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Andreas Blum
- Public, Private and Teaching Practice of Dermatology, Konstanz, Germany
| | - Katharina Kommoss
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Alexander Enk
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Ferdinand Toberer
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| | - Albert Rosenberger
- Institute of Genetic Epidemiology, University Medical Center, Georg-August University of Goettingen, Goettingen, Germany
| | - Holger A. Haenssle
- Department of Dermatology, University of Heidelberg, Heidelberg, Germany
| |
Collapse
|
82
|
Rajpurkar P, Lungren MP. The Current and Future State of AI Interpretation of Medical Images. N Engl J Med 2023; 388:1981-1990. [PMID: 37224199 DOI: 10.1056/nejmra2301725] [Citation(s) in RCA: 96] [Impact Index Per Article: 96.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Affiliation(s)
- Pranav Rajpurkar
- From the Department of Biomedical Informatics, Harvard Medical School, Boston (P.R.); the Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, and the Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco - both in California (M.P.L.); and Microsoft, Redmond, Washington (M.P.L.)
| | - Matthew P Lungren
- From the Department of Biomedical Informatics, Harvard Medical School, Boston (P.R.); the Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, and the Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco - both in California (M.P.L.); and Microsoft, Redmond, Washington (M.P.L.)
| |
Collapse
|
83
|
Khader F, Müller-Franzes G, Tayebi Arasteh S, Han T, Haarburger C, Schulze-Hagen M, Schad P, Engelhardt S, Baeßler B, Foersch S, Stegmaier J, Kuhl C, Nebelung S, Kather JN, Truhn D. Denoising diffusion probabilistic models for 3D medical image generation. Sci Rep 2023; 13:7303. [PMID: 37147413 PMCID: PMC10163245 DOI: 10.1038/s41598-023-34341-2] [Citation(s) in RCA: 18] [Impact Index Per Article: 18.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2022] [Accepted: 04/27/2023] [Indexed: 05/07/2023] Open
Abstract
Recent advances in computer vision have shown promising results in image generation. Diffusion probabilistic models have generated realistic images from textual input, as demonstrated by DALL-E 2, Imagen, and Stable Diffusion. However, their use in medicine, where imaging data typically comprises three-dimensional volumes, has not been systematically evaluated. Synthetic images may play a crucial role in privacy-preserving artificial intelligence and can also be used to augment small datasets. We show that diffusion probabilistic models can synthesize high-quality medical data for magnetic resonance imaging (MRI) and computed tomography (CT). For quantitative evaluation, two radiologists rated the quality of the synthesized images regarding "realistic image appearance", "anatomical correctness", and "consistency between slices". Furthermore, we demonstrate that synthetic images can be used in self-supervised pre-training and improve the performance of breast segmentation models when data is scarce (Dice scores, 0.91 [without synthetic data], 0.95 [with synthetic data]).
Collapse
Affiliation(s)
- Firas Khader
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Soroosh Tayebi Arasteh
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Tianyu Han
- Physics of Molecular Imaging Systems, Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany
| | | | - Maximilian Schulze-Hagen
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Philipp Schad
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Sandy Engelhardt
- Artificial Intelligence in Cardiovascular Medicine, University Hospital, Heidelberg, Germany
| | - Bettina Baeßler
- Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
| | | | | | - Christiane Kuhl
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Sven Nebelung
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital Aachen, Aachen, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany.
| |
Collapse
|
84
|
Wu Z, Liao W, Yan C, Zhao M, Liu G, Ma N, Li X. Deep learning based MRI reconstruction with transformer. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 233:107452. [PMID: 36924533 DOI: 10.1016/j.cmpb.2023.107452] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/17/2022] [Revised: 02/19/2023] [Accepted: 02/27/2023] [Indexed: 06/18/2023]
Abstract
Magnetic resonance imaging (MRI) has become one of the most powerful imaging techniques in medical diagnosis, yet the prolonged scanning time becomes a bottleneck for application. Reconstruction methods based on compress sensing (CS) have made progress in reducing this cost by acquiring fewer points in the k-space. Traditional CS methods impose restrictions from different sparse domains to regularize the optimization that always requires balancing time with accuracy. Neural network techniques enable learning a better prior from sample pairs and generating the results in an analytic way. In this paper, we propose a deep learning based reconstruction method to restore high-quality MRI images from undersampled k-space data in an end-to-end style. Unlike prior literature adopting convolutional neural networks (CNN), advanced Swin Transformer is used as the backbone of our work, which proved to be powerful in extracting deep features of the image. In addition, we combined the k-space consistency in the output and further improved the quality. We compared our models with several reconstruction methods and variants, and the experiment results proved that our model achieves the best results in samples at low sampling rates. The source code of KTMR could be acquired at https://github.com/BITwzl/KTMR.
Collapse
Affiliation(s)
- Zhengliang Wu
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China.
| | - Weibin Liao
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China
| | - Chao Yan
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China
| | - Mangsuo Zhao
- Department of Neurology, Yuquan Hospital, School of Clinical Medicine, Tsinghua University, Beijing, 100039, China
| | - Guowen Liu
- Big Data and Engineering Research Center, Beijing Children's Hospital, Capital Medical University, Department of Echocardiography, Beijing, 100045, China; Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beihang University & Capital Medical University, Beijing, 100083, China
| | - Ning Ma
- Big Data and Engineering Research Center, Beijing Children's Hospital, Capital Medical University, Department of Echocardiography, Beijing, 100045, China; Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beihang University & Capital Medical University, Beijing, 100083, China.
| | - Xuesong Li
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China.
| |
Collapse
|
85
|
Martel-Pelletier J, Paiement P, Pelletier JP. Magnetic resonance imaging assessments for knee segmentation and their use in combination with machine/deep learning as predictors of early osteoarthritis diagnosis and prognosis. Ther Adv Musculoskelet Dis 2023; 15:1759720X231165560. [PMID: 37151912 PMCID: PMC10155034 DOI: 10.1177/1759720x231165560] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2022] [Accepted: 03/23/2023] [Indexed: 05/09/2023] Open
Abstract
Knee osteoarthritis (OA) is a prevalent and disabling disease that can develop over decades. This disease is heterogeneous and involves structural changes in the whole joint, encompassing multiple tissue types. Detecting OA before the onset of irreversible changes is crucial for early management, and this could be achieved by allowing knee tissue visualization and quantifying their changes over time. Although some imaging modalities are available for knee structure assessment, magnetic resonance imaging (MRI) is preferred. This narrative review looks at existing literature, first on MRI-developed approaches for evaluating knee articular tissues, and second on prediction using machine/deep-learning-based methodologies and MRI as input or outcome for early OA diagnosis and prognosis. A substantial number of MRI methodologies have been developed to assess several knee tissues in a semi-quantitative and quantitative fashion using manual, semi-automated and fully automated systems. This dynamic field has grown substantially since the advent of machine/deep learning. Another active area is predictive modelling using machine/deep-learning methodologies enabling robust early OA diagnosis/prognosis. Moreover, incorporating MRI markers as input/outcome in such predictive models is important for a more accurate OA structural diagnosis/prognosis. The main limitation of their usage is the ability to move them in rheumatology practice. In conclusion, MRI knee tissue determination and quantification provide early indicators for individuals at high risk of developing this disease or for patient prognosis. Such assessment of knee tissues, combined with the development of models/tools from machine/deep learning using, in addition to other parameters, MRI markers for early diagnosis/prognosis, will maximize opportunities for individualized risk assessment for use in clinical practice permitting precision medicine. Future efforts should be made to integrate such prediction models into open access, allowing early disease management to prevent or delay the OA outcome.
Collapse
Affiliation(s)
- Johanne Martel-Pelletier
- Osteoarthritis Research Unit, University of
Montreal Hospital Research Centre (CRCHUM), 900 Saint-Denis, R11.412B,
Montreal, QC H2X 0A9, Canada
| | - Patrice Paiement
- Osteoarthritis Research Unit, University of
Montreal Hospital Research Centre (CRCHUM), Montreal, QC, Canada
| | - Jean-Pierre Pelletier
- Osteoarthritis Research Unit, University of
Montreal Hospital Research Centre (CRCHUM), Montreal, QC, Canada
| |
Collapse
|
86
|
Qian J, Li H, Wang J, He L. Recent Advances in Explainable Artificial Intelligence for Magnetic Resonance Imaging. Diagnostics (Basel) 2023; 13:1571. [PMID: 37174962 PMCID: PMC10178221 DOI: 10.3390/diagnostics13091571] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Revised: 03/29/2023] [Accepted: 04/26/2023] [Indexed: 05/15/2023] Open
Abstract
Advances in artificial intelligence (AI), especially deep learning (DL), have facilitated magnetic resonance imaging (MRI) data analysis, enabling AI-assisted medical image diagnoses and prognoses. However, most of the DL models are considered as "black boxes". There is an unmet need to demystify DL models so domain experts can trust these high-performance DL models. This has resulted in a sub-domain of AI research called explainable artificial intelligence (XAI). In the last decade, many experts have dedicated their efforts to developing novel XAI methods that are competent at visualizing and explaining the logic behind data-driven DL models. However, XAI techniques are still in their infancy for medical MRI image analysis. This study aims to outline the XAI applications that are able to interpret DL models for MRI data analysis. We first introduce several common MRI data modalities. Then, a brief history of DL models is discussed. Next, we highlight XAI frameworks and elaborate on the principles of multiple popular XAI methods. Moreover, studies on XAI applications in MRI image analysis are reviewed across the tissues/organs of the human body. A quantitative analysis is conducted to reveal the insights of MRI researchers on these XAI techniques. Finally, evaluations of XAI methods are discussed. This survey presents recent advances in the XAI domain for explaining the DL models that have been utilized in MRI applications.
Collapse
Affiliation(s)
- Jinzhao Qian
- Imaging Research Center, Department of Radiology, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH 45229, USA
- Department of Computer Science, University of Cincinnati, Cincinnati, OH 45221, USA
| | - Hailong Li
- Imaging Research Center, Department of Radiology, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH 45229, USA
- Department of Radiology, College of Medicine, University of Cincinnati, Cincinnati, OH 45221, USA
| | - Junqi Wang
- Imaging Research Center, Department of Radiology, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH 45229, USA
| | - Lili He
- Imaging Research Center, Department of Radiology, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH 45229, USA
- Department of Computer Science, University of Cincinnati, Cincinnati, OH 45221, USA
- Department of Radiology, College of Medicine, University of Cincinnati, Cincinnati, OH 45221, USA
| |
Collapse
|
87
|
Huang SC, Pareek A, Jensen M, Lungren MP, Yeung S, Chaudhari AS. Self-supervised learning for medical image classification: a systematic review and implementation guidelines. NPJ Digit Med 2023; 6:74. [PMID: 37100953 PMCID: PMC10131505 DOI: 10.1038/s41746-023-00811-0] [Citation(s) in RCA: 39] [Impact Index Per Article: 39.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2022] [Accepted: 03/30/2023] [Indexed: 04/28/2023] Open
Abstract
Advancements in deep learning and computer vision provide promising solutions for medical image analysis, potentially improving healthcare and patient outcomes. However, the prevailing paradigm of training deep learning models requires large quantities of labeled training data, which is both time-consuming and cost-prohibitive to curate for medical images. Self-supervised learning has the potential to make significant contributions to the development of robust medical imaging models through its ability to learn useful insights from copious medical datasets without labels. In this review, we provide consistent descriptions of different self-supervised learning strategies and compose a systematic review of papers published between 2012 and 2022 on PubMed, Scopus, and ArXiv that applied self-supervised learning to medical imaging classification. We screened a total of 412 relevant studies and included 79 papers for data extraction and analysis. With this comprehensive effort, we synthesize the collective knowledge of prior work and provide implementation guidelines for future researchers interested in applying self-supervised learning to their development of medical imaging classification models.
Collapse
Affiliation(s)
- Shih-Cheng Huang
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA.
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA, USA.
| | - Anuj Pareek
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA, USA
| | - Malte Jensen
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
| | - Matthew P Lungren
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - Serena Yeung
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA, USA
- Department of Computer Science, Stanford University, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Clinical Excellence Research Center, Stanford University School of Medicine, Stanford, CA, USA
| | - Akshay S Chaudhari
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
- Stanford Cardiovascular Institute, Stanford University, Stanford, CA, USA
| |
Collapse
|
88
|
Farzaneh N, Ansari S, Lee E, Ward KR, Sjoding MW. Collaborative strategies for deploying artificial intelligence to complement physician diagnoses of acute respiratory distress syndrome. NPJ Digit Med 2023; 6:62. [PMID: 37031252 PMCID: PMC10082784 DOI: 10.1038/s41746-023-00797-9] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2022] [Accepted: 03/10/2023] [Indexed: 04/10/2023] Open
Abstract
There is a growing gap between studies describing the capabilities of artificial intelligence (AI) diagnostic systems using deep learning versus efforts to investigate how or when to integrate AI systems into a real-world clinical practice to support physicians and improve diagnosis. To address this gap, we investigate four potential strategies for AI model deployment and physician collaboration to determine their potential impact on diagnostic accuracy. As a case study, we examine an AI model trained to identify findings of the acute respiratory distress syndrome (ARDS) on chest X-ray images. While this model outperforms physicians at identifying findings of ARDS, there are several reasons why fully automated ARDS detection may not be optimal nor feasible in practice. Among several collaboration strategies tested, we find that if the AI model first reviews the chest X-ray and defers to a physician if it is uncertain, this strategy achieves a higher diagnostic accuracy (0.869, 95% CI 0.835-0.903) compared to a strategy where a physician reviews a chest X-ray first and defers to an AI model if uncertain (0.824, 95% CI 0.781-0.862), or strategies where the physician reviews the chest X-ray alone (0.808, 95% CI 0.767-0.85) or the AI model reviews the chest X-ray alone (0.847, 95% CI 0.806-0.887). If the AI model reviews a chest X-ray first, this allows the AI system to make decisions for up to 79% of cases, letting physicians focus on the most challenging subsets of chest X-rays.
Collapse
Affiliation(s)
- Negar Farzaneh
- The Max Harry Weil Institute for Critical Care Research & Innovation, University of Michigan, Ann Arbor, MI, USA.
- Department of Emergency Medicine, University of Michigan Medical School, Ann Arbor, MI, USA.
| | - Sardar Ansari
- The Max Harry Weil Institute for Critical Care Research & Innovation, University of Michigan, Ann Arbor, MI, USA
- Department of Emergency Medicine, University of Michigan Medical School, Ann Arbor, MI, USA
| | - Elizabeth Lee
- Department of Radiology, University of Michigan, Ann Arbor, MI, USA
| | - Kevin R Ward
- The Max Harry Weil Institute for Critical Care Research & Innovation, University of Michigan, Ann Arbor, MI, USA
- Department of Emergency Medicine, University of Michigan Medical School, Ann Arbor, MI, USA
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA
| | - Michael W Sjoding
- The Max Harry Weil Institute for Critical Care Research & Innovation, University of Michigan, Ann Arbor, MI, USA
- Department of Internal Medicine, Division of Pulmonary and Critical Care, University of Michigan Medical School, Ann Arbor, MI, USA
| |
Collapse
|
89
|
Chou YT, Lin CT, Chang TA, Wu YL, Yu CE, Ho TY, Chen HY, Hsu KC, Kuang-Sheng Lee O. Development of artificial intelligence-based clinical decision support system for diagnosis of meniscal injury using magnetic resonance images. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104523] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/06/2023]
|
90
|
Vega C, Schneider R, Satagopam V. Analysis: Flawed Datasets of Monkeypox Skin Images. J Med Syst 2023; 47:37. [PMID: 36933065 PMCID: PMC10024024 DOI: 10.1007/s10916-023-01928-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Accepted: 02/26/2023] [Indexed: 03/19/2023]
Abstract
The self-proclaimed first publicly available dataset of Monkeypox skin images consists of medically irrelevant images extracted from Google and photography repositories through a process denominated web-scrapping. Yet, this did not stop other researchers from employing it to build Machine Learning (ML) solutions aimed at computer-aided diagnosis of Monkeypox and other viral infections presenting skin lesions. Neither did it stop the reviewers or editors from publishing these subsequent works in peer-reviewed journals. Several of these works claimed extraordinary performance in the classification of Monkeypox, Chickenpox and Measles, employing ML and the aforementioned dataset. In this work, we analyse the initiator work that has catalysed the development of several ML solutions, and whose popularity is continuing to grow. Further, we provide a rebuttal experiment that showcases the risks of such methodologies, proving that the ML solutions do not necessarily obtain their performance from the features relevant to the diseases at issue.
Collapse
Affiliation(s)
- Carlos Vega
- Bioinformatics Core, University of Luxembourg, Luxembourg Centre for Systems Biomedicine, Av. du Swing 6, Belvaux, 4367, Luxembourg.
| | - Reinhard Schneider
- Bioinformatics Core, University of Luxembourg, Luxembourg Centre for Systems Biomedicine, Av. du Swing 6, Belvaux, 4367, Luxembourg
| | - Venkata Satagopam
- Bioinformatics Core, University of Luxembourg, Luxembourg Centre for Systems Biomedicine, Av. du Swing 6, Belvaux, 4367, Luxembourg
| |
Collapse
|
91
|
Hung TNK, Vy VPT, Tri NM, Hoang LN, Tuan LV, Ho QT, Le NQK, Kang JH. Automatic Detection of Meniscus Tears Using Backbone Convolutional Neural Networks on Knee MRI. J Magn Reson Imaging 2023; 57:740-749. [PMID: 35648374 DOI: 10.1002/jmri.28284] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 05/21/2022] [Accepted: 05/23/2022] [Indexed: 01/19/2023] Open
Abstract
BACKGROUND Timely diagnosis of meniscus injuries is key for preventing knee joint dysfunction and improving patient outcomes because it decreases morbidity and facilitates treatment planning. PURPOSE To train and evaluate a deep learning model for automated detection of meniscus tears on knee magnetic resonance imaging (MRI). STUDY TYPE Bicentric retrospective study. SUBJECTS In total, 584 knee MRI studies, divided among training (n = 234), testing (n = 200), and external validation (n = 150) data sets, were used in this study. The public data set MRNet was used as a second external validation data set to evaluate the performance of the model. SEQUENCE A 3 T, coronal, and sagittal images from T1-weighted proton density (PD) fast spin-echo (FSE) with fat saturation and T2-weighted FSE with fat saturation sequences. ASSESSMENT The detection system for meniscus tear was based on the improved YOLOv4 model with Darknet-53 as the backbone. The performance of the model was also compared with that of three radiologists of varying levels of experience. The determination of the presence of a meniscus tear from surgery reports was used as the ground truth for the images. STATISTICAL TESTS Sensitivity, specificity, prevalence, positive predictive value, negative predictive value, accuracy, and receiver operating characteristic curve were used to evaluate the performance of the detection model. Two-way analysis of variance, Wilcoxon signed-rank test, and Tukey's multiple tests were used to evaluate differences in performance between the model and radiologists. RESULTS The overall accuracies for detecting meniscus tears using our model on the internal testing, internal validation, and external validation data sets were 95.4%, 95.8%, and 78.8%, respectively. One radiologist had significantly lower performance than our model in detecting meniscal tears (accuracy: 0.9025 ± 0.093 vs. 0.9580 ± 0.025). DATA CONCLUSION The proposed model had high sensitivity, specificity, and accuracy for detecting meniscus tears on knee MRIs. EVIDENCE LEVEL 3 TECHNICAL EFFICACY: Stage 2.
Collapse
Affiliation(s)
- Truong Nguyen Khanh Hung
- International Master/Ph.D. Program in Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan.,Department of Orthopedic and Trauma, Cho Ray Hospital, Ho Chi Minh City, Vietnam
| | - Vu Pham Thao Vy
- International Master/Ph.D. Program in Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan.,Department of Radiology, Thai Nguyen National Hospital, Thai Nguyen City, Vietnam
| | - Nguyen Minh Tri
- Advance Program in Computer Science, University of Science, Ho Chi Minh City, Vietnam
| | - Le Ngoc Hoang
- Graduate Institute of Biomedical Materials & Tissue Engineering, College of Biomedical Engineering, Taipei Medical University, Taipei, Taiwan
| | - Le Van Tuan
- Department of Orthopedic and Trauma, Cho Ray Hospital, Ho Chi Minh City, Vietnam
| | - Quang Thai Ho
- Department of Computer Science and Engineering, Yuan Ze University, Chung-Li, Taiwan
| | - Nguyen Quoc Khanh Le
- Professional Master Program in Artificial Intelligence in Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan.,Research Center for Artificial Intelligence in Medicine, Taipei Medical University, Taipei, Taiwan
| | - Jiunn-Horng Kang
- Professional Master Program in Artificial Intelligence in Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan.,Department of Physical Medicine and Rehabilitation, Taipei Medical University Hospital, Taipei, Taiwan.,Graduate Institute of Nanomedicine and Medical Engineering, College of Biomedical Engineering, Taipei Medical University, Taipei, Taiwan
| |
Collapse
|
92
|
Dung NT, Thuan NH, Van Dung T, Van Nho L, Tri NM, Vy VPT, Hoang LN, Phat NT, Chuong DA, Dang LH. End-to-end deep learning model for segmentation and severity staging of anterior cruciate ligament injuries from MRI. Diagn Interv Imaging 2023; 104:133-141. [PMID: 36328943 DOI: 10.1016/j.diii.2022.10.010] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2022] [Revised: 10/14/2022] [Accepted: 10/19/2022] [Indexed: 11/24/2022]
Abstract
PURPOSE The purpose of this study was to develop a semi-supervised segmentation and classification deep learning model for the diagnosis of anterior cruciate ligament (ACL) tears on MRI based on a semi-supervised framework, double-linear layers U-Net (DCLU-Net). MATERIALS AND METHODS A total of 297 participants who underwent of total of 303 MRI examination of the knee with fat-saturated proton density (PD) fast spin-echo (FSE) sequence in the sagittal plane were included. There were 214 men and 83 women, with a mean age of 37.46 ± 1.40 (standard deviation) years (range: 29-44 years). Of these, 107 participants had intact ACL (36%), 98 had partially torn ACL (33%), and 92 had fully ruptured ACL (31%). The DCLU-Net was combined with radiomic features for enhancing performances in the diagnosis of ACL tear. The different evaluation metrics for both classification (accuracy, sensitivity, accuracy) and segmentation (mean Dice similarity coefficient and root mean square error) were compared individually for each image class across the three phases of the model, with each value being compared to its respective value from the previous phase. Findings at arthroscopic knee surgery were used as the standard of reference. RESULTS With the addition of radiomic features, the final model yielded accuracies of 90% (95% CI: 83-92), 82% (95% CI: 73-86), and 92% (95% CI: 87-94) for classifying ACL as intact, partially torn and fully ruptured, respectively. The DCLU-Net achieved mean Dice similarity coefficient and root mean square error of 0.78 (95% CI: 0.71-0.80) and 0.05 (95% CI: 0.06-0.07), respectively, when segmenting the three ACL conditions with pseudo data (P < 0.001). CONCLUSION A dual-modules deep learning model with segmentation and classification capabilities was successfully developed. In addition, the use of semi-supervised techniques significantly reduced the amount of manual segmentation data without compromising performance.
Collapse
Affiliation(s)
- Nguyen Tan Dung
- Department of Rehabilitation, Da Nang C Hospital, Da Nang City 50000, Viet Nam; Department of Rehabilitation, Da Nang University of Medical Technology and Pharmacy, Da Nang City 50000, Viet Nam
| | - Ngo Huu Thuan
- Department of Radiology, Da Nang C Hospital, Da Nang city 50000, Viet Nam; Department of Medical Imaging, Da Nang University of Medical Technology and Pharmacy, Da Nang city, 50000, Viet Nam
| | - Truong Van Dung
- Department of Rehabilitation, Da Nang C Hospital, Da Nang City 50000, Viet Nam
| | - Le Van Nho
- Faculty of Medicine, Da Nang University of Medical Technology and Pharmacy, Da Nang City, 50000, Viet Nam
| | - Nguyen Minh Tri
- Advance Program in Computer Science, University of Science, Ho Chi Minh City 70000, Viet Nam; YRDx-AI Lab, Ho Chi Minh City 70000, Viet Nam
| | - Vu Pham Thao Vy
- YRDx-AI Lab, Ho Chi Minh City 70000, Viet Nam; International Master/Ph.D. Program in Medicine, College of Medicine, Taipei Medical University, Taipei 110, Taiwan
| | - Le Ngoc Hoang
- YRDx-AI Lab, Ho Chi Minh City 70000, Viet Nam; Graduate Institute of Biomedical Materials and Tissue Engineering, College of Biomedical Engineering, Taipei Medical University, Taipei 110, Taiwan
| | - Nguyen Thuan Phat
- YRDx-AI Lab, Ho Chi Minh City 70000, Viet Nam; Department of Computer Science, Vietnamese German University, Ho Chi Minh City 70000, Viet Nam
| | | | - Luong Huu Dang
- Department of Otolaryngology, Faculty of Medicine, University of Medicine and Pharmacy at Ho Chi Minh City, Ho Chi Minh City 70000, Viet Nam
| |
Collapse
|
93
|
Lee JJ, Namiri NK, Astuto B, Link TM, Majumdar S, Pedoia V. Personalized Risk Model and Leveraging of Magnetic Resonance Imaging-Based Structural Phenotypes and Clinical Factors to Predict Incidence of Radiographic Osteoarthritis. Arthritis Care Res (Hoboken) 2023; 75:501-508. [PMID: 35245407 DOI: 10.1002/acr.24877] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2021] [Revised: 02/03/2022] [Accepted: 03/01/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Our study aimed to investigate the association between time to incidence of radiographic osteoarthritis (OA) and magnetic resonance imaging (MRI)-based structural phenotypes proposed by the Rapid Osteoarthritis MRI Eligibility Score (ROAMES). METHODS A retrospective cohort of 2,328 participants without radiographic OA at baseline were selected from the Osteoarthritis Initiative study. Utilizing a deep-learning model, we automatically assessed the presence of inflammatory, meniscus/cartilage, subchondral bone, and hypertrophic phenotypes from MRIs acquired at baseline and 12-, 24-, 36-, 48-, 72-, and 96-month follow-up visits. In addition to 4 structural phenotypes, we examined severe knee injury history and Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) pain scores as time dependent. We used Cox proportional hazards regression to analyze the association between 4 structural phenotypes and radiographic OA disease-free survival, both univariate and adjusted for known risk factors including age, sex, race, body mass index, presence of Heberden's nodes, and knee malalignment. RESULTS Inflammatory (hazard ratio [HR] 3.37 [95% confidence interval (95% CI) 2.45-4.63]), meniscus/cartilage (HR 1.55 [95% CI 1.21-1.98]), and subchondral bone (HR 1.84 [95% CI 1.63-2.09]) phenotypes were associated with time to radiographic OA at P < 0.05 when adjusted for the risk factors. Sex was a modifier of hypertrophic phenotype association with time to radiographic OA. Female participants with the hypertrophic phenotype were associated with 2.8 times higher risk of radiographic OA (95% CI 2.25-7.54) compared to male participants without the hypertrophic phenotype. CONCLUSION Four ROAMES phenotypes may contribute to time to radiographic OA incidence and if validated could be used as a promising tool for personalized OA management.
Collapse
Affiliation(s)
- Jinhee J Lee
- Center for Intelligent Imaging, University of California, San Francisco
| | - Nikan K Namiri
- Center for Intelligent Imaging, University of California, San Francisco
| | - Bruno Astuto
- Center for Intelligent Imaging, University of California, San Francisco
| | - Thomas M Link
- Center for Intelligent Imaging, University of California, San Francisco
| | - Sharmila Majumdar
- Center for Intelligent Imaging, University of California, San Francisco
| | - Valentina Pedoia
- Center for Intelligent Imaging, University of California, San Francisco
| |
Collapse
|
94
|
Bonnin M, Müller-Fouarge F, Estienne T, Bekadar S, Pouchy C, Ait Si Selmi T. Artificial Intelligence Radiographic Analysis Tool for Total Knee Arthroplasty. J Arthroplasty 2023:S0883-5403(23)00184-5. [PMID: 36858127 DOI: 10.1016/j.arth.2023.02.053] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Revised: 02/15/2023] [Accepted: 02/17/2023] [Indexed: 03/03/2023] Open
Abstract
BACKGROUND The postoperative follow-up of a patient after total knee arthroplasty (TKA) requires regular evaluation of the condition of the knee through interpretation of X-rays. This rigorous analysis requires expertize, time, and methodical standardization. Our work evaluated the use of an artificial intelligence tool, X-TKA, to assist surgeons in their interpretation. METHODS A series of 12 convolutional neural networks were trained on a large database containing 39,751 X-ray images. These algorithms are able to determine examination quality, identify image characteristics, assess prosthesis sizing and positioning, measure knee-prosthesis alignment angles, and detect anomalies in the bone-cement-implant complex. The individual interpretations of a pool of senior surgeons with and without the assistance of X-TKA were evaluated on a reference dataset built in consensus by senior surgeons. RESULTS The algorithms obtained a mean area under the curve value of 0.98 on the quality assurance and the image characteristics tasks. They reached a mean difference for the predicted angles of 1.71° (standard deviation, 1.53°), similar to the surgeon average difference of 1.69° (standard deviation, 1.52°). The comparative analysis showed that the assistance of X-TKA allowed surgeons to gain 5% in accuracy and 12% in sensitivity in the detection of interface anomalies. Moreover, this study demonstrated a gain in repeatability for each single surgeon (Light's kappa +0.17), as well as a gain in the reproducibility between surgeons (Light's kappa +0.1). CONCLUSION This study highlights the benefit of using an intelligent artificial tool for a standardized interpretation of postoperative knee X-rays and indicates the potential for its use in clinical practice.
Collapse
|
95
|
Sufriyana H, Wu YW, Su ECY. Human-guided deep learning with ante-hoc explainability by convolutional network from non-image data for pregnancy prognostication. Neural Netw 2023; 162:99-116. [PMID: 36898257 DOI: 10.1016/j.neunet.2023.02.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2022] [Revised: 01/30/2023] [Accepted: 02/14/2023] [Indexed: 02/26/2023]
Abstract
BACKGROUND AND OBJECTIVE Deep learning is applied in medicine mostly due to its state-of-the-art performance for diagnostic imaging. Supervisory authorities also require the model to be explainable, but most explain the model after development (post hoc) instead of incorporating explanation into the design (ante hoc). This study aimed to demonstrate a human-guided deep learning with ante-hoc explainability by convolutional network from non-image data to develop, validate, and deploy a prognostic prediction model for PROM and an estimator of time of delivery using a nationwide health insurance database. METHODS To guide modeling, we constructed and verified association diagrams respectively from literatures and electronic health records. Non-image data were transformed into meaningful images utilizing predictor-to-predictor similarities, harnessing the power of convolutional neural network mostly used for diagnostic imaging. The network architecture was also inferred from the similarities. RESULTS This resulted the best model for prelabor rupture of membranes (n=883, 376) with the area under curves 0.73 (95% CI 0.72 to 0.75) and 0.70 (95% CI 0.69 to 0.71) respectively by internal and external validations, and outperformed previous models found by systematic review. It was explainable by knowledge-based diagrams and model representation. CONCLUSIONS This allows prognostication with actionable insights for preventive medicine.
Collapse
Affiliation(s)
- Herdiantri Sufriyana
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, 250 Wu-Xing Street, Taipei 11031, Taiwan; Department of Medical Physiology, Faculty of Medicine, Universitas Nahdlatul Ulama Surabaya, 57 Raya Jemursari Road, Surabaya 60237, Indonesia
| | - Yu-Wei Wu
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, 250 Wu-Xing Street, Taipei 11031, Taiwan; Clinical Big Data Research Center, Taipei Medical University Hospital, 250 Wu-Xing Street, Taipei 11031, Taiwan
| | - Emily Chia-Yu Su
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, 250 Wu-Xing Street, Taipei 11031, Taiwan; Clinical Big Data Research Center, Taipei Medical University Hospital, 250 Wu-Xing Street, Taipei 11031, Taiwan; Research Center for Artificial Intelligence in Medicine, Taipei Medical University, 250 Wu-Xing Street, Taipei 11031, Taiwan.
| |
Collapse
|
96
|
Jurgensmeier K, Till SE, Lu Y, Arguello AM, Stuart MJ, Saris DBF, Camp CL, Krych AJ. Risk factors for secondary meniscus tears can be accurately predicted through machine learning, creating a resource for patient education and intervention. Knee Surg Sports Traumatol Arthrosc 2023; 31:518-529. [PMID: 35974194 PMCID: PMC10138786 DOI: 10.1007/s00167-022-07117-w] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Accepted: 08/05/2022] [Indexed: 02/07/2023]
Abstract
PURPOSE This study sought to develop and internally validate a machine learning model to identify risk factors and quantify overall risk of secondary meniscus injury in a longitudinal cohort after primary ACL reconstruction (ACLR). METHODS Patients with new ACL injury between 1990 and 2016 with minimum 2-year follow-up were identified. Records were extensively reviewed to extract demographic, treatment, and diagnosis of new meniscus injury following ACLR. Four candidate machine learning algorithms were evaluated to predict secondary meniscus tears. Performance was assessed through discrimination using area under the receiver operating characteristics curve (AUROC), calibration, and decision curve analysis; interpretability was enhanced utilizing global variable importance plots and partial dependence curves. RESULTS A total of 1187 patients underwent ACLR; 139 (11.7%) experienced a secondary meniscus tear at a mean time of 65 months post-op. The best performing model for predicting secondary meniscus tear was the random forest (AUROC = 0.790, 95% CI: 0.785-0.795; calibration intercept = 0.006, 95% CI: 0.005-0.007, calibration slope = 0.961 95% CI: 0.956-0.965, Brier's score = 0.10 95% CI: 0.09-0.12), and all four machine learning algorithms outperformed traditional logistic regression. The following risk factors were identified: shorter time to return to sport (RTS), lower VAS at injury, increased time from injury to surgery, older age at injury, and proximal ACL tear. CONCLUSION Machine learning models outperformed traditional prediction models and identified multiple risk factors for secondary meniscus tears after ACLR. Following careful external validation, these models can be deployed to provide real-time quantifiable risk for counseling and timely intervention to help guide patient expectations and possibly improve clinical outcomes. LEVEL OF EVIDENCE III.
Collapse
Affiliation(s)
- Kevin Jurgensmeier
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA
| | - Sara E Till
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA
| | - Yining Lu
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA
| | - Alexandra M Arguello
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA
| | - Michael J Stuart
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA
| | - Daniel B F Saris
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA
| | - Christopher L Camp
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA
| | - Aaron J Krych
- Department of Orthopedic Surgery, Mayo Clinic, 200 First St. SW, Rochester, MN, 55905, USA.
| |
Collapse
|
97
|
Zhuang Z, Si L, Wang S, Xuan K, Ouyang X, Zhan Y, Xue Z, Zhang L, Shen D, Yao W, Wang Q. Knee Cartilage Defect Assessment by Graph Representation and Surface Convolution. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:368-379. [PMID: 36094985 DOI: 10.1109/tmi.2022.3206042] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Knee osteoarthritis (OA) is the most common osteoarthritis and a leading cause of disability. Cartilage defects are regarded as major manifestations of knee OA, which are visible by magnetic resonance imaging (MRI). Thus early detection and assessment for knee cartilage defects are important for protecting patients from knee OA. In this way, many attempts have been made on knee cartilage defect assessment by applying convolutional neural networks (CNNs) to knee MRI. However, the physiologic characteristics of the cartilage may hinder such efforts: the cartilage is a thin curved layer, implying that only a small portion of voxels in knee MRI can contribute to the cartilage defect assessment; heterogeneous scanning protocols further challenge the feasibility of the CNNs in clinical practice; the CNN-based knee cartilage evaluation results lack interpretability. To address these challenges, we model the cartilages structure and appearance from knee MRI into a graph representation, which is capable of handling highly diverse clinical data. Then, guided by the cartilage graph representation, we design a non-Euclidean deep learning network with the self-attention mechanism, to extract cartilage features in the local and global, and to derive the final assessment with a visualized result. Our comprehensive experiments show that the proposed method yields superior performance in knee cartilage defect assessment, plus its convenient 3D visualization for interpretability.
Collapse
|
98
|
Jin W, Li X, Fatehi M, Hamarneh G. Guidelines and evaluation of clinical explainable AI in medical image analysis. Med Image Anal 2023; 84:102684. [PMID: 36516555 DOI: 10.1016/j.media.2022.102684] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2021] [Revised: 10/20/2022] [Accepted: 11/03/2022] [Indexed: 11/18/2022]
Abstract
Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1) Understandability and G2 Clinical relevance. For the chosen explanation form, its specific XAI technique should be optimized for G3 Truthfulness, G4 Informative plausibility, and G5 Computational efficiency. Following the guidelines, we conducted a systematic evaluation on a novel problem of multi-modal medical image explanation with two clinical tasks, and proposed new evaluation metrics accordingly. Sixteen commonly-used heatmap XAI techniques were evaluated and found to be insufficient for clinical use due to their failure in G3 and G4. Our evaluation demonstrated the use of Clinical XAI Guidelines to support the design and evaluation of clinically viable XAI.
Collapse
Affiliation(s)
- Weina Jin
- School of Computing Science, Simon Fraser University, Burnaby, BC, V5A 1S6, Canada.
| | - Xiaoxiao Li
- Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, V6T 1Z4, Canada.
| | - Mostafa Fatehi
- Division of Neurosurgery, The University of British Columbia, Vancouver, BC, V5Z 1M9, Canada.
| | - Ghassan Hamarneh
- School of Computing Science, Simon Fraser University, Burnaby, BC, V5A 1S6, Canada.
| |
Collapse
|
99
|
Zhao X, Yang T, Li B, Zhang X. SwinGAN: A dual-domain Swin Transformer-based generative adversarial network for MRI reconstruction. Comput Biol Med 2023; 153:106513. [PMID: 36603439 DOI: 10.1016/j.compbiomed.2022.106513] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 12/09/2022] [Accepted: 12/31/2022] [Indexed: 01/02/2023]
Abstract
Magnetic resonance imaging (MRI) is one of the most important modalities for clinical diagnosis. However, the main disadvantages of MRI are the long scanning time and the moving artifact caused by patient movement during prolonged imaging. It can also lead to patient anxiety and discomfort, so accelerated imaging is indispensable for MRI. Convolutional neural network (CNN) based methods have become the fact standard for medical image reconstruction, and generative adversarial network (GAN) have also been widely used. Nevertheless, due to the limited ability of CNN to capture long-distance information, it may lead to defects in the structure of the reconstructed images such as blurry contour. In this paper, we propose a novel Swin Transformer-based dual-domain generative adversarial network (SwinGAN) for accelerated MRI reconstruction. The SwinGAN consists of two generators: a frequency-domain generator and an image-domain generator. Both the generators utilize Swin Transformer as backbone for effectively capturing the long-distance dependencies. A contextual image relative position encoder (ciRPE) is designed to enhance the ability to capture local information. We extensively evaluate the method on the IXI brain dataset, MICCAI 2013 dataset and MRNet knee dataset. Compared with KIGAN, the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) are improved by 6.1% and 1.49% to 37.64 dB and 0.98 on IXI dataset respectively, which demonstrates that our model can sufficiently utilize the local and global information of image. The model shows promising performance and robustness under different undersampling masks, different acceleration rates and different datasets. But it needs high hardware requirements with the increasing of the network parameters. The code is available at: https://github.com/learnerzx/SwinGAN.
Collapse
Affiliation(s)
- Xiang Zhao
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Tiejun Yang
- School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou, 450001, China; Key Laboratory of Grain Information Processing and Control (HAUT), Ministry of Education, Zhengzhou, China; Henan Key Laboratory of Grain Photoelectric Detection and Control (HAUT), Zhengzhou, Henan, China.
| | - Bingjie Li
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Xin Zhang
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| |
Collapse
|
100
|
Korneev A, Lipina M, Lychagin A, Timashev P, Kon E, Telyshev D, Goncharuk Y, Vyazankin I, Elizarov M, Murdalov E, Pogosyan D, Zhidkov S, Bindeeva A, Liang XJ, Lasovskiy V, Grinin V, Anosov A, Kalinsky E. Systematic review of artificial intelligence tack in preventive orthopaedics: is the land coming soon? INTERNATIONAL ORTHOPAEDICS 2023; 47:393-403. [PMID: 36369394 DOI: 10.1007/s00264-022-05628-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/12/2022] [Accepted: 10/26/2022] [Indexed: 11/13/2022]
Abstract
PURPOSE This study aims to describe and assess the current stage of the artificial intelligence (AI) technology integration in preventive orthopaedics of the knee and hip joints. MATERIALS AND METHODS The study was conducted in strict compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. Literature databases were searched for articles describing the development and validation of AI models aimed at diagnosing knee or hip joint pathologies or predicting their development or course in patients. The quality of the included articles was assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) and QUADAS-AI tools. RESULTS 56 articles were found that meet all the inclusion criteria. We identified two problems that block the full integration of AI into the routine of an orthopaedic physician. The first of them is related to the insufficient amount, variety and quality of data for training, and validation and testing of AI models. The second problem is the rarity of rational evaluation of models, which is why their real quality cannot always be evaluated. CONCLUSION The vastness and relevance of the studied topic are beyond doubt. Qualitative and optimally validated models exist in all four scopes considered. Additional optimization and confirmation of the models' quality on various datasets are the last technical stumbling blocks for creating usable software and integrating them into the routine of an orthopaedic physician.
Collapse
Affiliation(s)
- Alexander Korneev
- Medical Polymer Synthesis Laboratory, Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia.,Laboratory of Clinical Smart Nanotechnologies, Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia.,N.V. Sklifosovsky Institute of Clinical Medicine, Sechenov University, Moscow, 119991, Russia
| | - Marina Lipina
- Laboratory of Clinical Smart Nanotechnologies, Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia. .,Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia.
| | - Alexey Lychagin
- Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia
| | - Peter Timashev
- Laboratory of Clinical Smart Nanotechnologies, Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia.,World-Class Research Center "Digital Biodesign and Personalized Healthcare", Sechenov University, Moscow, 119991, Russia.,Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia
| | - Elizaveta Kon
- Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia.,Humanitas Clinical and Research Center - IRCCS, Via Manzoni 56, Rozzano, 20089, Milan, Italy
| | - Dmitry Telyshev
- Russia Institute of Biomedical Systems, National Research University of Electronic Technology Moscow, Zelenograd, 124498, Russia.,Institute of Bionic Technologies and Engineering, Sechenov University, Moscow, 119991, Russia
| | - Yuliya Goncharuk
- Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia
| | - Ivan Vyazankin
- Laboratory of Clinical Smart Nanotechnologies, Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia.,Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia
| | - Mikhail Elizarov
- Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia
| | - Emirkhan Murdalov
- Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia
| | - David Pogosyan
- Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia.,Department of Life Safety and Disaster Medicine, Sechenov University, Moscow, 119991, Russia
| | - Sergei Zhidkov
- N.V. Sklifosovsky Institute of Clinical Medicine, Sechenov University, Moscow, 119991, Russia
| | - Anastasia Bindeeva
- N.V. Sklifosovsky Institute of Clinical Medicine, Sechenov University, Moscow, 119991, Russia
| | - Xing-Jie Liang
- Laboratory of Clinical Smart Nanotechnologies, Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia.,CAS Key Laboratory for Biomedical Effects of Nanomaterials and Nanosafety, CAS Center for Excellence in Nanoscience, National Center for Nanoscience and Technology of China, Beijing, 100190, China.,University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Vladimir Lasovskiy
- Department of Artificial Intelligence and Digital Products, VimpelCom, Moscow, 127083, Russia
| | - Victor Grinin
- Department of Artificial Intelligence and Digital Products, VimpelCom, Moscow, 127083, Russia
| | - Alexey Anosov
- Department of Artificial Intelligence and Digital Products, VimpelCom, Moscow, 127083, Russia
| | - Eugene Kalinsky
- Laboratory of Clinical Smart Nanotechnologies, Institute for Regenerative Medicine, Sechenov University, Moscow, 119991, Russia.,Department of Traumatology, Orthopaedics and Disaster Surgery, Sechenov University, Moscow, 119991, Russia
| |
Collapse
|