101
|
Azizi S, Culp L, Freyberg J, Mustafa B, Baur S, Kornblith S, Chen T, Tomasev N, Mitrović J, Strachan P, Mahdavi SS, Wulczyn E, Babenko B, Walker M, Loh A, Chen PHC, Liu Y, Bavishi P, McKinney SM, Winkens J, Roy AG, Beaver Z, Ryan F, Krogue J, Etemadi M, Telang U, Liu Y, Peng L, Corrado GS, Webster DR, Fleet D, Hinton G, Houlsby N, Karthikesalingam A, Norouzi M, Natarajan V. Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nat Biomed Eng 2023:10.1038/s41551-023-01049-7. [PMID: 37291435 DOI: 10.1038/s41551-023-01049-7] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Accepted: 05/02/2023] [Indexed: 06/10/2023]
Abstract
Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Ting Chen
- Google Research, Mountain View, CA, USA
| | | | | | | | | | | | | | | | - Aaron Loh
- Google Research, Mountain View, CA, USA
| | | | - Yuan Liu
- Google Research, Mountain View, CA, USA
| | | | | | | | | | | | - Fiona Ryan
- Georgia Institute of Technology, Computer Science, Atlanta, GA, USA
| | | | - Mozziyar Etemadi
- School of Medicine/School of Engineering, Northwestern University, Chicago, IL, USA
| | | | - Yun Liu
- Google Research, Mountain View, CA, USA
| | - Lily Peng
- Google Research, Mountain View, CA, USA
| | | | | | | | | | | | | | | | | |
Collapse
|
102
|
Agrawal A, Khatri GD, Khurana B, Sodickson AD, Liang Y, Dreizin D. A survey of ASER members on artificial intelligence in emergency radiology: trends, perceptions, and expectations. Emerg Radiol 2023; 30:267-277. [PMID: 36913061 PMCID: PMC10362990 DOI: 10.1007/s10140-023-02121-0] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Accepted: 02/28/2023] [Indexed: 03/14/2023]
Abstract
PURPOSE There is a growing body of diagnostic performance studies for emergency radiology-related artificial intelligence/machine learning (AI/ML) tools; however, little is known about user preferences, concerns, experiences, expectations, and the degree of penetration of AI tools in emergency radiology. Our aim is to conduct a survey of the current trends, perceptions, and expectations regarding AI among American Society of Emergency Radiology (ASER) members. METHODS An anonymous and voluntary online survey questionnaire was e-mailed to all ASER members, followed by two reminder e-mails. A descriptive analysis of the data was conducted, and results summarized. RESULTS A total of 113 members responded (response rate 12%). The majority were attending radiologists (90%) with greater than 10 years' experience (80%) and from an academic practice (65%). Most (55%) reported use of commercial AI CAD tools in their practice. Workflow prioritization based on pathology detection, injury or disease severity grading and classification, quantitative visualization, and auto-population of structured reports were identified as high-value tasks. Respondents overwhelmingly indicated a need for explainable and verifiable tools (87%) and the need for transparency in the development process (80%). Most respondents did not feel that AI would reduce the need for emergency radiologists in the next two decades (72%) or diminish interest in fellowship programs (58%). Negative perceptions pertained to potential for automation bias (23%), over-diagnosis (16%), poor generalizability (15%), negative impact on training (11%), and impediments to workflow (10%). CONCLUSION ASER member respondents are in general optimistic about the impact of AI in the practice of emergency radiology and its impact on the popularity of emergency radiology as a subspecialty. The majority expect to see transparent and explainable AI models with the radiologist as the decision-maker.
Collapse
Affiliation(s)
- Anjali Agrawal
- New Delhi operations, Teleradiology Solutions, Delhi, India
| | - Garvit D Khatri
- Nuclear Medicine, Department of Radiology, University of Washington School of Medicine, Seattle, WA, USA
| | - Bharti Khurana
- Emergency Radiology, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Aaron D Sodickson
- Emergency Radiology, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Yuanyuan Liang
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - David Dreizin
- Trauma and Emergency Radiology, Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| |
Collapse
|
103
|
Dreizin D, Staziaki PV, Khatri GD, Beckmann NM, Feng Z, Liang Y, Delproposto ZS, Klug M, Spann JS, Sarkar N, Fu Y. Artificial intelligence CAD tools in trauma imaging: a scoping review from the American Society of Emergency Radiology (ASER) AI/ML Expert Panel. Emerg Radiol 2023; 30:251-265. [PMID: 36917287 PMCID: PMC10640925 DOI: 10.1007/s10140-023-02120-1] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Accepted: 02/27/2023] [Indexed: 03/16/2023]
Abstract
BACKGROUND AI/ML CAD tools can potentially improve outcomes in the high-stakes, high-volume model of trauma radiology. No prior scoping review has been undertaken to comprehensively assess tools in this subspecialty. PURPOSE To map the evolution and current state of trauma radiology CAD tools along key dimensions of technology readiness. METHODS Following a search of databases, abstract screening, and full-text document review, CAD tool maturity was charted using elements of data curation, performance validation, outcomes research, explainability, user acceptance, and funding patterns. Descriptive statistics were used to illustrate key trends. RESULTS A total of 4052 records were screened, and 233 full-text articles were selected for content analysis. Twenty-one papers described FDA-approved commercial tools, and 212 reported algorithm prototypes. Works ranged from foundational research to multi-reader multi-case trials with heterogeneous external data. Scalable convolutional neural network-based implementations increased steeply after 2016 and were used in all commercial products; however, options for explainability were narrow. Of FDA-approved tools, 9/10 performed detection tasks. Dataset sizes ranged from < 100 to > 500,000 patients, and commercialization coincided with public dataset availability. Cross-sectional torso datasets were uniformly small. Data curation methods with ground truth labeling by independent readers were uncommon. No papers assessed user acceptance, and no method included human-computer interaction. The USA and China had the highest research output and frequency of research funding. CONCLUSIONS Trauma imaging CAD tools are likely to improve patient care but are currently in an early stage of maturity, with few FDA-approved products for a limited number of uses. The scarcity of high-quality annotated data remains a major barrier.
Collapse
Affiliation(s)
- David Dreizin
- Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| | - Pedro V Staziaki
- Cardiothoracic Imaging, Department of Radiology, Larner College of Medicine, University of Vermont, Burlington, VT, USA
| | - Garvit D Khatri
- Department of Radiology, University of Washington School of Medicine, Seattle, WA, USA
| | - Nicholas M Beckmann
- Memorial Hermann Orthopedic & Spine Hospital, McGovern Medical School at UTHealth, Houston, TX, USA
| | - Zhaoyong Feng
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Yuanyuan Liang
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Zachary S Delproposto
- Division of Emergency Radiology, Department of Radiology, University of Michigan, Ann Arbor, MI, USA
| | | | - J Stephen Spann
- Department of Radiology, University of Alabama at Birmingham Heersink School of Medicine, Birmingham, AL, USA
| | - Nathan Sarkar
- University of Maryland School of Medicine, Baltimore, MD, USA
| | - Yunting Fu
- Health Sciences and Human Services Library, University of Maryland, Baltimore, Baltimore, MD, USA
| |
Collapse
|
104
|
Lee GP, Park SH, Kim YJ, Chung JW, Kim KG. Enhancing Disease Classification in Abdominal CT Scans through RGB Superposition Methods and 2D Convolutional Neural Networks: A Study of Appendicitis and Diverticulitis. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2023; 2023:7714483. [PMID: 37284168 PMCID: PMC10241572 DOI: 10.1155/2023/7714483] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Figures] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 02/10/2023] [Accepted: 04/15/2023] [Indexed: 06/08/2023]
Abstract
The primary symptom of both appendicitis and diverticulitis is a pain in the right lower abdomen; it is almost impossible to diagnose these conditions through symptoms alone. However, there will be misdiagnoses happening when using abdominal computed tomography (CT) scans. Most previous studies have used a 3D convolutional neural network (CNN) suitable for processing sequences of images. However, 3D CNN models can be difficult to implement in typical computing systems because they require large amounts of data, GPU memory, and extensive training times. We propose a deep learning method, utilizing red, green, and blue (RGB) channel superposition images reconstructed from three slices of sequence images. Using the RGB superposition image as the input image of the model, the average accuracy was shown as 90.98% in EfficietNetB0, 91.27% in EfficietNetB2, and 91.98% in EfficietNetB4. The AUC score using the RGB superposition image was higher than the original image of the single channel for EfficientNetB4 (0.967 vs. 0.959, p = 0.0087). The comparison in performance between the model architectures using the RGB superposition method showed the highest learning performance in the EfficientNetB4 model among all indicators; accuracy was 91.98% and recall was 95.35%. EfficientNetB4 using the RGB superposition method had a 0.011 (p value = 0.0001) AUC score higher than EfficientNetB0 using the same method. The superposition of sequential slice images in CT scans was used to enhance the distinction in features like shape, size of the target, and spatial information used to classify disease. The proposed method has fewer constraints than the 3D CNN method and is suitable for an environment using 2D CNN; thus, we can achieve performance improvement with limited resources.
Collapse
Affiliation(s)
- Gi Pyo Lee
- Department of Health Sciences and Technology, Gachon Advanced Institute for Health Sciences and Technology (GAIHST), Gachon University, Incheon, Republic of Korea
| | - So Hyun Park
- Department of Radiology, Gil Medical Center, Gachon University College of Medicine, Incheon, Republic of Korea
| | - Young Jae Kim
- Department of Biomedical Engineering, College of IT Convergence, Gachon University, Gyeonggi-do, Republic of Korea
| | - Jun-Won Chung
- Division of Gastroenterology, Department of Internal Medicine, Gil Medical Center, Gachon University College of Medicine, Incheon, Republic of Korea
| | - Kwang Gi Kim
- Department of Health Sciences and Technology, Gachon Advanced Institute for Health Sciences and Technology (GAIHST), Gachon University, Incheon, Republic of Korea
- Department of Biomedical Engineering Medical Center, College of Medicine, Gachon University, Incheon, Republic of Korea
| |
Collapse
|
105
|
Ali MB, Gu IYH, Berger MS, Jakola AS. A novel federated deep learning scheme for glioma and its subtype classification. Front Neurosci 2023; 17:1181703. [PMID: 37287799 PMCID: PMC10242007 DOI: 10.3389/fnins.2023.1181703] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2023] [Accepted: 04/27/2023] [Indexed: 06/09/2023] Open
Abstract
Background Deep learning (DL) has shown promising results in molecular-based classification of glioma subtypes from MR images. DL requires a large number of training data for achieving good generalization performance. Since brain tumor datasets are usually small in size, combination of such datasets from different hospitals are needed. Data privacy issue from hospitals often poses a constraint on such a practice. Federated learning (FL) has gained much attention lately as it trains a central DL model without requiring data sharing from different hospitals. Method We propose a novel 3D FL scheme for glioma and its molecular subtype classification. In the scheme, a slice-based DL classifier, EtFedDyn, is exploited which is an extension of FedDyn, with the key differences on using focal loss cost function to tackle severe class imbalances in the datasets, and on multi-stream network to exploit MRIs in different modalities. By combining EtFedDyn with domain mapping as the pre-processing and 3D scan-based post-processing, the proposed scheme makes 3D brain scan-based classification on datasets from different dataset owners. To examine whether the FL scheme could replace the central learning (CL) one, we then compare the classification performance between the proposed FL and the corresponding CL schemes. Furthermore, detailed empirical-based analysis were also conducted to exam the effect of using domain mapping, 3D scan-based post-processing, different cost functions and different FL schemes. Results Experiments were done on two case studies: classification of glioma subtypes (IDH mutation and wild-type on TCGA and US datasets in case A) and glioma grades (high/low grade glioma HGG and LGG on MICCAI dataset in case B). The proposed FL scheme has obtained good performance on the test sets (85.46%, 75.56%) for IDH subtypes and (89.28%, 90.72%) for glioma LGG/HGG all averaged on five runs. Comparing with the corresponding CL scheme, the drop in test accuracy from the proposed FL scheme is small (-1.17%, -0.83%), indicating its good potential to replace the CL scheme. Furthermore, the empirically tests have shown that an increased classification test accuracy by applying: domain mapping (0.4%, 1.85%) in case A; focal loss function (1.66%, 3.25%) in case A and (1.19%, 1.85%) in case B; 3D post-processing (2.11%, 2.23%) in case A and (1.81%, 2.39%) in case B and EtFedDyn over FedAvg classifier (1.05%, 1.55%) in case A and (1.23%, 1.81%) in case B with fast convergence, which all contributed to the improvement of overall performance in the proposed FL scheme. Conclusion The proposed FL scheme is shown to be effective in predicting glioma and its subtypes by using MR images from test sets, with great potential of replacing the conventional CL approaches for training deep networks. This could help hospitals to maintain their data privacy, while using a federated trained classifier with nearly similar performance as that from a centrally trained one. Further detailed experiments have shown that different parts in the proposed 3D FL scheme, such as domain mapping (make datasets more uniform) and post-processing (scan-based classification), are essential.
Collapse
Affiliation(s)
- Muhaddisa Barat Ali
- Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden
| | - Irene Yu-Hua Gu
- Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden
| | - Mitchel S. Berger
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, United States
| | - Asgeir Store Jakola
- Department of Neuroscience and Physiology, University of Gothenburg, Gothenburg, Sweden
- Department of Neurosurgery, Sahlgrenska University Hospital, Gothenburg, Sweden
| |
Collapse
|
106
|
Obaid AM, Turki A, Bellaaj H, Ksantini M, AlTaee A, Alaerjan A. Detection of Gallbladder Disease Types Using Deep Learning: An Informative Medical Method. Diagnostics (Basel) 2023; 13:diagnostics13101744. [PMID: 37238227 DOI: 10.3390/diagnostics13101744] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 05/09/2023] [Accepted: 05/10/2023] [Indexed: 05/28/2023] Open
Abstract
Nowadays, despite all the conducted research and the provided efforts in advancing the healthcare sector, there is a strong need to rapidly and efficiently diagnose various diseases. The complexity of some disease mechanisms on one side and the dramatic life-saving potential on the other side raise big challenges for the development of tools for the early detection and diagnosis of diseases. Deep learning (DL), an area of artificial intelligence (AI), can be an informative medical tomography method that can aid in the early diagnosis of gallbladder (GB) disease based on ultrasound images (UI). Many researchers considered the classification of only one disease of the GB. In this work, we successfully managed to apply a deep neural network (DNN)-based classification model to a rich built database in order to detect nine diseases at once and to determine the type of disease using UI. In the first step, we built a balanced database composed of 10,692 UI of the GB organ from 1782 patients. These images were carefully collected from three hospitals over roughly three years and then classified by professionals. In the second step, we preprocessed and enhanced the dataset images in order to achieve the segmentation step. Finally, we applied and then compared four DNN models to analyze and classify these images in order to detect nine GB disease types. All the models produced good results in detecting GB diseases; the best was the MobileNet model, with an accuracy of 98.35%.
Collapse
Affiliation(s)
- Ahmed Mahdi Obaid
- CEMLab, National School of Electronics and Telecommunications of Sfax, University of Sfax, Sfax 3029, Tunisia
| | - Amina Turki
- CEMLab, National Engineering School of Sfax, University of Sfax, Sfax 3029, Tunisia
| | - Hatem Bellaaj
- ReDCAD, National Engineering School of Sfax, University of Sfax, Sfax 3029, Tunisia
| | - Mohamed Ksantini
- CEMLab, National Engineering School of Sfax, University of Sfax, Sfax 3029, Tunisia
| | | | - Alaa Alaerjan
- College of Computer and Information Sciences, Jouf University, Sakaka 72388, Saudi Arabia
| |
Collapse
|
107
|
Yepes-Calderon F, McComb JG. Eliminating the need for manual segmentation to determine size and volume from MRI. A proof of concept on segmenting the lateral ventricles. PLoS One 2023; 18:e0285414. [PMID: 37167315 PMCID: PMC10174587 DOI: 10.1371/journal.pone.0285414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Accepted: 04/23/2023] [Indexed: 05/13/2023] Open
Abstract
Manual segmentation, which is tedious, time-consuming, and operator-dependent, is currently used as the gold standard to validate automatic and semiautomatic methods that quantify geometries from 2D and 3D MR images. This study examines the accuracy of manual segmentation and generalizes a strategy to eliminate its use. Trained individuals manually measured MR lateral ventricles images of normal and hydrocephalus infants from 1 month to 9.5 years of age. We created 3D-printed models of the lateral ventricles from the MRI studies and accurately estimated their volume by water displacement. MRI phantoms were made from the 3D models and images obtained. Using a previously developed artificial intelligence (AI) algorithm that employs four features extracted from the images, we estimated the ventricular volume of the phantom images. The algorithm was certified when discrepancies between the volumes-gold standards-yielded by the water displacement device and those measured by the automation were smaller than 2%. Then, we compared volumes after manual segmentation with those obtained with the certified automation. As determined by manual segmentation, lateral ventricular volume yielded an inter and intra-operator variation up to 50% and 48%, respectively, while manually segmenting saggital images generated errors up to 71%. These errors were determined by direct comparisons with the volumes yielded by the certified automation. The errors induced by manual segmentation are large enough to adversely affect decisions that may lead to less-than-optimal treatment; therefore, we suggest avoiding manual segmentation whenever possible.
Collapse
Affiliation(s)
- Fernando Yepes-Calderon
- Science-Based Platforms, Fort Pierce, Florida, United States of America
- GYM Group SA, Cali, Colombia
| | - J. Gordon McComb
- Division of Neurosurgery, Children’s Hospital Los Angeles, Los Angeles, CA, United States of America
- Keck School of Medicine, University of Southern California, Los Angeles, CA, United States of America
| |
Collapse
|
108
|
Yang S, Wu X, Ge S, Zheng Z, Zhou SK, Xiao L. Radiology report generation with a learned knowledge base and multi-modal alignment. Med Image Anal 2023; 86:102798. [PMID: 36989850 DOI: 10.1016/j.media.2023.102798] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Revised: 02/10/2023] [Accepted: 03/10/2023] [Indexed: 03/28/2023]
Abstract
In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distill and restore medical knowledge from textual embedding without manual labor; (ii) Multi-modal alignment: to promote the semantic alignment among reports, disease labels, and images, we explicitly utilize textual embedding to guide the learning of the visual feature space. We evaluate the performance of the proposed model using metrics from both natural language generation and clinic efficacy on the public IU-Xray and MIMIC-CXR datasets. Our ablation study shows that each module contributes to improving the quality of generated reports. Furthermore, the assistance of both modules, our approach outperforms state-of-the-art methods over almost all the metrics. Code is available at https://github.com/LX-doctorAI1/M2KT.
Collapse
|
109
|
Yan R, Shen Y, Zhang X, Xu P, Wang J, Li J, Ren F, Ye D, Zhou SK. Histopathological bladder cancer gene mutation prediction with hierarchical deep multiple-instance learning. Med Image Anal 2023; 87:102824. [PMID: 37126973 DOI: 10.1016/j.media.2023.102824] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Revised: 03/13/2023] [Accepted: 04/17/2023] [Indexed: 05/03/2023]
Abstract
Gene mutation detection is usually carried out by molecular biological methods, which is expensive and has a long-time cycle. In contrast, pathological images are ubiquitous. If clinically significant gene mutations can be predicted only through pathological images, it will greatly promote the widespread use of gene mutation detection in clinical practice. However, current gene mutation prediction methods based on pathological images are ineffective because of the inability to identify mutated regions in gigapixel Whole Slide Image (WSI). To address this challenge, hereby we propose a carefully designed framework for WSI-based gene mutation prediction, which consists of three parts. (i) The first part of cancerous area segmentation, based on supervised learning, quickly filters out a large number of non-mutated regions; (ii) the second part of cancerous patch clustering, based on the representations derived from contrastive learning, ensures the comprehensiveness of patch selection; and (iii) the third part of mutation classification, based on the proposed hierarchical deep multi-instance learning method (HDMIL), ensures that sufficient patches are considered and inaccurate selections are ignored. In addition, benefiting from a two-stage attention mechanism in HDMIL, the patches that are highly correlated with gene mutations can be identified. This interpretability can help a pathologist to analyze the correlation between gene mutation and histopathological morphology. Experimental results demonstrate that the proposed gene mutation prediction framework significantly outperforms the state-of-the-art methods. In the TCGA bladder cancer dataset, five clinically relevant gene mutations are well predicted.
Collapse
Affiliation(s)
- Rui Yan
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China; University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Yijun Shen
- Department of Urology, Fudan University Shanghai Cancer Center, Shanghai, 200032, China; Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Xueyuan Zhang
- Zhijian Life Technology Co., Ltd., Beijing, 100036, China
| | - Peihang Xu
- Department of Urology, Fudan University Shanghai Cancer Center, Shanghai, 200032, China; Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Jun Wang
- Department of Urology, Sun Yat-sen University Cancer Center, Guangzhou, 510060, China
| | - Jintao Li
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China
| | - Fei Ren
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China; SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China.
| | - Dingwei Ye
- Department of Urology, Fudan University Shanghai Cancer Center, Shanghai, 200032, China; Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China.
| | - S Kevin Zhou
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China; School of Biomedical Engineering & Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, 215123, China.
| |
Collapse
|
110
|
Shimron E, Perlman O. AI in MRI: Computational Frameworks for a Faster, Optimized, and Automated Imaging Workflow. Bioengineering (Basel) 2023; 10:bioengineering10040492. [PMID: 37106679 PMCID: PMC10135995 DOI: 10.3390/bioengineering10040492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 04/12/2023] [Accepted: 04/18/2023] [Indexed: 04/29/2023] Open
Abstract
Over the last decade, artificial intelligence (AI) has made an enormous impact on a wide range of fields, including science, engineering, informatics, finance, and transportation [...].
Collapse
Affiliation(s)
- Efrat Shimron
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720, USA
| | - Or Perlman
- Department of Biomedical Engineering, Tel Aviv University, Tel Aviv 6997801, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
| |
Collapse
|
111
|
Zhao H, Liu Z, Tang J, Gao B, Qin Q, Li J, Zhou Y, Yao P, Xi Y, Lin Y, Qian H, Wu H. Energy-efficient high-fidelity image reconstruction with memristor arrays for medical diagnosis. Nat Commun 2023; 14:2276. [PMID: 37081008 PMCID: PMC10119144 DOI: 10.1038/s41467-023-38021-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Accepted: 04/06/2023] [Indexed: 04/22/2023] Open
Abstract
Medical imaging is an important tool for accurate medical diagnosis, while state-of-the-art image reconstruction algorithms raise critical challenges in massive data processing for high-speed and high-quality imaging. Here, we present a memristive image reconstructor (MIR) to greatly accelerate image reconstruction with discrete Fourier transformation (DFT) by computing-in-memory (CIM) with memristor arrays. A high-accuracy quasi-analogue mapping (QAM) method and generic complex matrix transfer (CMT) scheme was proposed to improve the mapping precision and transfer efficiency, respectively. High-fidelity magnetic resonance imaging (MRI) and computed tomography (CT) image reconstructions were demonstrated, achieving software-equivalent qualities and DICE scores after segmentation with nnU-Net algorithm. Remarkably, our MIR exhibited 153× and 79× improvements in energy efficiency and normalized image reconstruction speed, respectively, compared to graphics processing unit (GPU). This work demonstrates MIR as a promising high-fidelity image reconstruction platform for future medical diagnosis, and also largely extends the application of memristor-based CIM beyond artificial neural networks.
Collapse
Affiliation(s)
- Han Zhao
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - Zhengwu Liu
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - Jianshi Tang
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China.
- Beijing Innovation Center for Future Chips (ICFC), Tsinghua University, Beijing, 100084, China.
| | - Bin Gao
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
- Beijing Innovation Center for Future Chips (ICFC), Tsinghua University, Beijing, 100084, China
| | - Qi Qin
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - Jiaming Li
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - Ying Zhou
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - Peng Yao
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - Yue Xi
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - Yudeng Lin
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
| | - He Qian
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
- Beijing Innovation Center for Future Chips (ICFC), Tsinghua University, Beijing, 100084, China
| | - Huaqiang Wu
- School of Integrated Circuits, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
- Beijing Innovation Center for Future Chips (ICFC), Tsinghua University, Beijing, 100084, China
| |
Collapse
|
112
|
Dikici E, Nguyen XV, Takacs N, Prevedello LM. Prediction of model generalizability for unseen data: Methodology and case study in brain metastases detection in T1-Weighted contrast-enhanced 3D MRI. Comput Biol Med 2023; 159:106901. [PMID: 37068317 DOI: 10.1016/j.compbiomed.2023.106901] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Revised: 03/08/2023] [Accepted: 04/09/2023] [Indexed: 04/19/2023]
Abstract
BACKGROUND AND PURPOSE A medical AI system's generalizability describes the continuity of its performance acquired from varying geographic, historical, and methodologic settings. Previous literature on this topic has mostly focused on "how" to achieve high generalizability (e.g., via larger datasets, transfer learning, data augmentation, model regularization schemes), with limited success. Instead, we aim to understand "when" the generalizability is achieved: Our study presents a medical AI system that could estimate its generalizability status for unseen data on-the-fly. MATERIALS AND METHODS We introduce a latent space mapping (LSM) approach utilizing Fréchet distance loss to force the underlying training data distribution into a multivariate normal distribution. During the deployment, a given test data's LSM distribution is processed to detect its deviation from the forced distribution; hence, the AI system could predict its generalizability status for any previously unseen data set. If low model generalizability is detected, then the user is informed by a warning message integrated into a sample deployment workflow. While the approach is applicable for most classification deep neural networks (DNNs), we demonstrate its application to a brain metastases (BM) detector for T1-weighted contrast-enhanced (T1c) 3D MRI. The BM detection model was trained using 175 T1c studies acquired internally (from the authors' institution) and tested using (1) 42 internally acquired exams and (2) 72 externally acquired exams from the publicly distributed Brain Mets dataset provided by the Stanford University School of Medicine. Generalizability scores, false positive (FP) rates, and sensitivities of the BM detector were computed for the test datasets. RESULTS AND CONCLUSION The model predicted its generalizability to be low for 31% of the testing data (i.e., two of the internally and 33 of the externally acquired exams), where it produced (1) ∼13.5 false positives (FPs) at 76.1% BM detection sensitivity for the low and (2) ∼10.5 FPs at 89.2% BM detection sensitivity for the high generalizability groups respectively. These results suggest that the proposed formulation enables a model to predict its generalizability for unseen data.
Collapse
Affiliation(s)
- Engin Dikici
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA.
| | - Xuan V Nguyen
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA
| | - Noah Takacs
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA
| | - Luciano M Prevedello
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA
| |
Collapse
|
113
|
Pramanik R, Banerjee B, Efimenko G, Kaplun D, Sarkar R. Monkeypox detection from skin lesion images using an amalgamation of CNN models aided with Beta function-based normalization scheme. PLoS One 2023; 18:e0281815. [PMID: 37027356 PMCID: PMC10081766 DOI: 10.1371/journal.pone.0281815] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Accepted: 02/01/2023] [Indexed: 04/08/2023] Open
Abstract
We have recently been witnessing that our society is starting to heal from the impacts of COVID-19. The economic, social and cultural impacts of a pandemic cannot be ignored and we should be properly equipped to deal with similar situations in future. Recently, Monkeypox has been concerning the international health community with its lethal impacts for a probable pandemic. In such situations, having appropriate protocols and methodologies to deal with the outbreak efficiently is of paramount interest to the world. Early diagnosis and treatment stand as the only viable option to tackle such problems. To this end, in this paper, we propose an ensemble learning-based framework to detect the presence of the Monkeypox virus from skin lesion images. We first consider three pre-trained base learners, namely Inception V3, Xception and DenseNet169 to fine-tune on a target Monkeypox dataset. Further, we extract probabilities from these deep models to feed into the ensemble framework. To combine the outcomes, we propose a Beta function-based normalization scheme of probabilities to learn an efficient aggregation of complementary information obtained from the base learners followed by the sum rule-based ensemble. The framework is extensively evaluated on a publicly available Monkeypox skin lesion dataset using a five-fold cross-validation setup to evaluate its effectiveness. The model achieves an average of 93.39%, 88.91%, 96.78% and 92.35% accuracy, precision, recall and F1 scores, respectively. The supporting source codes are presented in https://github.com/BihanBanerjee/MonkeyPox.
Collapse
Affiliation(s)
- Rishav Pramanik
- Department of Computer Science and Engineering, Jadavpur University, Kolkata, West Bengal, India
| | - Bihan Banerjee
- Department of Computer Science and Engineering, University Institute of Technology, Burdwan, India
| | - George Efimenko
- Department of Automation and Control Processes, Saint Petersburg Electrotechnical University “LETI”, Saint Petersburg, Russian Federation
| | - Dmitrii Kaplun
- Department of Automation and Control Processes, Saint Petersburg Electrotechnical University “LETI”, Saint Petersburg, Russian Federation
| | - Ram Sarkar
- Department of Computer Science and Engineering, Jadavpur University, Kolkata, West Bengal, India
| |
Collapse
|
114
|
Yu H, Wang F, Teodoro G, Chen F, Guo X, Nickerson JM, Kong J. Self-supervised semantic segmentation of retinal pigment epithelium cells in flatmount fluorescent microscopy images. Bioinformatics 2023; 39:btad191. [PMID: 37067486 PMCID: PMC10139776 DOI: 10.1093/bioinformatics/btad191] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2022] [Revised: 03/23/2023] [Accepted: 04/10/2023] [Indexed: 04/18/2023] Open
Abstract
MOTIVATION Morphological analyses with flatmount fluorescent images are essential to retinal pigment epithelial (RPE) aging studies and thus require accurate RPE cell segmentation. Although rapid technology advances in deep learning semantic segmentation have achieved great success in many biomedical research, the performance of these supervised learning methods for RPE cell segmentation is still limited by inadequate training data with high-quality annotations. RESULTS To address this problem, we develop a Self-Supervised Semantic Segmentation (S4) method that utilizes a self-supervised learning strategy to train a semantic segmentation network with an encoder-decoder architecture. We employ a reconstruction and a pairwise representation loss to make the encoder extract structural information, while we create a morphology loss to produce the segmentation map. In addition, we develop a novel image augmentation algorithm (AugCut) to produce multiple views for self-supervised learning and enhance the network training performance. To validate the efficacy of our method, we applied our developed S4 method for RPE cell segmentation to a large set of flatmount fluorescent microscopy images, we compare our developed method for RPE cell segmentation with other state-of-the-art deep learning approaches. Compared with other state-of-the-art deep learning approaches, our method demonstrates better performance in both qualitative and quantitative evaluations, suggesting its promising potential to support large-scale cell morphological analyses in RPE aging investigations. AVAILABILITY AND IMPLEMENTATION The codes and the documentation are available at: https://github.com/jkonglab/S4_RPE.
Collapse
Affiliation(s)
- Hanyi Yu
- Department of Computer Science, Emory University, Atlanta, GA 30322, USA
| | - Fusheng Wang
- Department of Computer Science, Stony Brook University, Stony Brook, NY 11794, USA
| | - George Teodoro
- Department of Computer Science, Federal University of Minas Gerais, Belo Horizonte 31270, Brazil
| | - Fan Chen
- Huangpu Branch, Guangzhou Urban Planning & Design Survey Research Institute, Guangzhou 510060, China
| | - Xiaoyuan Guo
- Department of Computer Science, Emory University, Atlanta, GA 30322, USA
| | - John M Nickerson
- Department of Ophthalmology, Emory University, Atlanta, GA 30322, USA
| | - Jun Kong
- Department of Computer Science, Emory University, Atlanta, GA 30322, USA
- Department of Mathematics and Statistics, Georgia State University, Atlanta, GA 30303, USA
| |
Collapse
|
115
|
Shi J, Tu H, Park J, Marjanovic M, Higham AM, Luckey NN, Cradock KA, Liu ZG, Boppart SA. Weakly supervised identification of microscopic human breast cancer-related optical signatures from normal-appearing breast tissue. BIOMEDICAL OPTICS EXPRESS 2023; 14:1339-1354. [PMID: 37078030 PMCID: PMC10110327 DOI: 10.1364/boe.480687] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Revised: 02/08/2023] [Accepted: 02/08/2023] [Indexed: 05/03/2023]
Abstract
With the latest advancements in optical bioimaging, rich structural and functional information has been generated from biological samples, which calls for capable computational tools to identify patterns and uncover relationships between optical characteristics and various biomedical conditions. Constrained by the existing knowledge of the novel signals obtained by those bioimaging techniques, precise and accurate ground truth annotations can be difficult to obtain. Here we present a weakly supervised deep learning framework for optical signature discovery based on inexact and incomplete supervision. The framework consists of a multiple instance learning-based classifier for the identification of regions of interest in coarsely labeled images and model interpretation techniques for optical signature discovery. We applied this framework to investigate human breast cancer-related optical signatures based on virtual histopathology enabled by simultaneous label-free autofluorescence multiharmonic microscopy (SLAM), with the goal of exploring unconventional cancer-related optical signatures from normal-appearing breast tissues. The framework has achieved an average area under the curve (AUC) of 0.975 on the cancer diagnosis task. In addition to well-known cancer biomarkers, non-obvious cancer-related patterns were revealed by the framework, including NAD(P)H-rich extracellular vesicles observed in normal-appearing breast cancer tissue, which facilitate new insights into the tumor microenvironment and field cancerization. This framework can be further extended to diverse imaging modalities and optical signature discovery tasks.
Collapse
Affiliation(s)
- Jindou Shi
- Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, 306 N Wright Street, Urbana, IL 61801, USA
- Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, 405 N Mathews Avenue, Urbana, IL 61801, USA
| | - Haohua Tu
- Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, 306 N Wright Street, Urbana, IL 61801, USA
- Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, 405 N Mathews Avenue, Urbana, IL 61801, USA
| | - Jaena Park
- Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, 405 N Mathews Avenue, Urbana, IL 61801, USA
- Department of Bioengineering, University of Illinois at Urbana-Champaign, 1406 W Green Street, Urbana, IL 61801, USA
| | - Marina Marjanovic
- Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, 405 N Mathews Avenue, Urbana, IL 61801, USA
- Department of Bioengineering, University of Illinois at Urbana-Champaign, 1406 W Green Street, Urbana, IL 61801, USA
- Carle Illinois College of Medicine, University of Illinois at Urbana-Champaign, 506 S Mathews Avenue, Urbana, IL 61801, USA
| | - Anna M. Higham
- Carle Foundation Hospital, 611 W Park Street, Urbana, IL 61801, USA
| | | | | | - Z. George Liu
- Carle Foundation Hospital, 611 W Park Street, Urbana, IL 61801, USA
| | - Stephen A. Boppart
- Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, 306 N Wright Street, Urbana, IL 61801, USA
- Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, 405 N Mathews Avenue, Urbana, IL 61801, USA
- Department of Bioengineering, University of Illinois at Urbana-Champaign, 1406 W Green Street, Urbana, IL 61801, USA
- Carle Illinois College of Medicine, University of Illinois at Urbana-Champaign, 506 S Mathews Avenue, Urbana, IL 61801, USA
- Cancer Center at Illinois, University of Illinois at Urbana-Champaign, 405 N Mathews Avenue, Urbana, IL 61801, USA
| |
Collapse
|
116
|
Li J, Chen J, Tang Y, Wang C, Landman BA, Zhou SK. Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives. Med Image Anal 2023; 85:102762. [PMID: 36738650 PMCID: PMC10010286 DOI: 10.1016/j.media.2023.102762] [Citation(s) in RCA: 27] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 01/18/2023] [Accepted: 01/27/2023] [Indexed: 02/01/2023]
Abstract
Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.
Collapse
Affiliation(s)
- Jun Li
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Junyu Chen
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD, USA
| | - Yucheng Tang
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - Ce Wang
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Bennett A Landman
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - S Kevin Zhou
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China; School of Biomedical Engineering & Suzhou Institute for Advanced Research, Center for Medical Imaging, Robotics, and Analytic Computing & Learning (MIRACLE), University of Science and Technology of China, Suzhou 215123, China.
| |
Collapse
|
117
|
Khatatbeh IN, Alshurafat H, Al Shbail MO, Jamaani F. Factors Affecting Employees Use and Acceptance of Remote Working During the COVID-19 Pandemic: Evidence From the Jordanian Insurance Sector. SAGE OPEN 2023; 13:21582440231181390. [PMID: 37398984 PMCID: PMC10291219 DOI: 10.1177/21582440231181390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 07/04/2023]
Abstract
Remote working during the COVID-19 pandemic comes as an "enforced experiment," where companies and individuals have turned to work from home to preserve business continuity. Drawing on a theoretical framework that integrates the Theory of Reasoned Action (TRA), Social Capital Theory (SCT), and Technology Acceptance Model (TAM), this research uses a sample of 134 survey responses to assess the factors affecting the acceptance and use of remote work during the COVID-19 pandemic among workers of the insurance industry in Jordan. The results suggest that social trust, perceived usefulness, and perceived ease of use can help elevate employee's acceptance and use of remote work, whereas social norms have no significant effect. Considering these results, we further discuss implications and recommendations for the insurance sector.
Collapse
|
118
|
den Boer RB, Jaspers TJM, de Jongh C, Pluim JPW, van der Sommen F, Boers T, van Hillegersberg R, Van Eijnatten MAJM, Ruurda JP. Deep learning-based recognition of key anatomical structures during robot-assisted minimally invasive esophagectomy. Surg Endosc 2023:10.1007/s00464-023-09990-z. [PMID: 36947221 DOI: 10.1007/s00464-023-09990-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Accepted: 02/25/2023] [Indexed: 03/23/2023]
Abstract
OBJECTIVE To develop a deep learning algorithm for anatomy recognition in thoracoscopic video frames from robot-assisted minimally invasive esophagectomy (RAMIE) procedures using deep learning. BACKGROUND RAMIE is a complex operation with substantial perioperative morbidity and a considerable learning curve. Automatic anatomy recognition may improve surgical orientation and recognition of anatomical structures and might contribute to reducing morbidity or learning curves. Studies regarding anatomy recognition in complex surgical procedures are currently lacking. METHODS Eighty-three videos of consecutive RAMIE procedures between 2018 and 2022 were retrospectively collected at University Medical Center Utrecht. A surgical PhD candidate and an expert surgeon annotated the azygos vein and vena cava, aorta, and right lung on 1050 thoracoscopic frames. 850 frames were used for training of a convolutional neural network (CNN) to segment the anatomical structures. The remaining 200 frames of the dataset were used for testing the CNN. The Dice and 95% Hausdorff distance (95HD) were calculated to assess algorithm accuracy. RESULTS The median Dice of the algorithm was 0.79 (IQR = 0.20) for segmentation of the azygos vein and/or vena cava. A median Dice coefficient of 0.74 (IQR = 0.86) and 0.89 (IQR = 0.30) were obtained for segmentation of the aorta and lung, respectively. Inference time was 0.026 s (39 Hz). The prediction of the deep learning algorithm was compared with the expert surgeon annotations, showing an accuracy measured in median Dice of 0.70 (IQR = 0.19), 0.88 (IQR = 0.07), and 0.90 (0.10) for the vena cava and/or azygos vein, aorta, and lung, respectively. CONCLUSION This study shows that deep learning-based semantic segmentation has potential for anatomy recognition in RAMIE video frames. The inference time of the algorithm facilitated real-time anatomy recognition. Clinical applicability should be assessed in prospective clinical studies.
Collapse
Affiliation(s)
- R B den Boer
- Department of Surgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| | - T J M Jaspers
- Department of Biomedical Engineering, Eindhoven University of Technology, Groene Loper 3, 5612 AE, Eindhoven, The Netherlands
| | - C de Jongh
- Department of Surgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| | - J P W Pluim
- Department of Biomedical Engineering, Eindhoven University of Technology, Groene Loper 3, 5612 AE, Eindhoven, The Netherlands
| | - F van der Sommen
- Department of Electrical Engineering, Eindhoven University of Technology, Groene Loper 19, 5612 AP, Eindhoven, The Netherlands
| | - T Boers
- Department of Electrical Engineering, Eindhoven University of Technology, Groene Loper 19, 5612 AP, Eindhoven, The Netherlands
| | - R van Hillegersberg
- Department of Surgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| | - M A J M Van Eijnatten
- Department of Biomedical Engineering, Eindhoven University of Technology, Groene Loper 3, 5612 AE, Eindhoven, The Netherlands
| | - J P Ruurda
- Department of Surgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands.
| |
Collapse
|
119
|
McDermott MBA, Nestor B, Szolovits P. Clinical Artificial Intelligence: Design Principles and Fallacies. Clin Lab Med 2023; 43:29-46. [PMID: 36764807 DOI: 10.1016/j.cll.2022.09.004] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
Clinical artificial intelligence (AI)/machine learning (ML) is anticipated to offer new abilities in clinical decision support, diagnostic reasoning, precision medicine, clinical operational support, and clinical research, but careful concern is needed to ensure these technologies work effectively in the clinic. Here, we detail the clinical ML/AI design process, identifying several key questions and detailing several common forms of issues that arise with ML tools, as motivated by real-world examples, such that clinicians and researchers can better anticipate and correct for such issues in their own use of ML/AI techniques.
Collapse
Affiliation(s)
| | - Bret Nestor
- Department of Computer Science, University of Toronto, 40 St George St, Toronto, ON M5S 2E4, Canada
| | | |
Collapse
|
120
|
Deng Y, Lan L, You L, Chen K, Peng L, Zhao W, Song B, Wang Y, Ji Z, Zhou X. Automated CT Pancreas Segmentation for Acute Pancreatitis Patients by combining a Novel Object Detection Approach and U-Net. Biomed Signal Process Control 2023; 81:104430. [PMID: 37304128 PMCID: PMC10249746 DOI: 10.1016/j.bspc.2022.104430] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
Acute pancreatitis is an inflammatory disorder of the pancreas. Medical imaging, such as computed tomography (CT), has been widely used to detect volume changes in the pancreas for acute pancreatitis diagnosis. Many pancreas segmentation methods have been proposed but no methods for pancreas segmentation from acute pancreatitis patients. The segmentation of an inflamed pancreas is more challenging than the normal pancreas due to the following two reasons. 1) The inflamed pancreas invades surrounding organs and causes blurry boundaries. 2) The inflamed pancreas has higher shape, size, and location variability than the normal pancreas. To overcome these challenges, we propose an automated CT pancreas segmentation approach for acute pancreatitis patients by combining a novel object detection approach and U-Net. Our approach includes a detector and a segmenter. Specifically, we develop an FCN-guided region proposal network (RPN) detector to localize the pancreatitis regions. The detector first uses a fully convolutional network (FCN) to reduce the background interference of medical images and generates a fixed feature map containing the acute pancreatitis regions. Then the RPN is employed on the feature map to precisely localize the acute pancreatitis regions. After obtaining the location of pancreatitis, the U-Net segmenter is used on the cropped image according to the bounding box. The proposed approach is validated using a collected clinical dataset with 89 abdominal contrast-enhanced 3D CT scans from acute pancreatitis patients. Compared with other start-of-the-art approaches for normal pancreas segmentation, our method achieves better performance on both localization and segmentation in acute pancreatitis patients.
Collapse
Affiliation(s)
- Yang Deng
- School of Biomedical Engineering & Suzhou Institute for Advanced, University of Science and Technology of China, Suzhou 215123, China
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
| | - Lan Lan
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
- IT Center, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
- West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu 610041, China
| | - Lei You
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
| | - Kai Chen
- School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
| | - Li Peng
- Department of Radiology, West China Hospital, Sichuan University, Chengdu 610041, China
| | - Weiling Zhao
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
| | - Bin Song
- Department of Radiology, West China Hospital, Sichuan University, Chengdu 610041, China
| | - Yi Wang
- Department of Radiology, West China Hospital, Sichuan University, Chengdu 610041, China
| | - Zhiwei Ji
- College of Artificial Intelligence, Nanjing Agricultural University, Nanjing 210095, China
| | - Xiaobo Zhou
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
| |
Collapse
|
121
|
Zhao Z, Zhou F, Xu K, Zeng Z, Guan C, Zhou SK. LE-UDA: Label-Efficient Unsupervised Domain Adaptation for Medical Image Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:633-646. [PMID: 36227829 DOI: 10.1109/tmi.2022.3214766] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
While deep learning methods hitherto have achieved considerable success in medical image segmentation, they are still hampered by two limitations: (i) reliance on large-scale well-labeled datasets, which are difficult to curate due to the expert-driven and time-consuming nature of pixel-level annotations in clinical practices, and (ii) failure to generalize from one domain to another, especially when the target domain is a different modality with severe domain shifts. Recent unsupervised domain adaptation (UDA) techniques leverage abundant labeled source data together with unlabeled target data to reduce the domain gap, but these methods degrade significantly with limited source annotations. In this study, we address this underexplored UDA problem, investigating a challenging but valuable realistic scenario, where the source domain not only exhibits domain shift w.r.t. the target domain but also suffers from label scarcity. In this regard, we propose a novel and generic framework called "Label-Efficient Unsupervised Domain Adaptation" (LE-UDA). In LE-UDA, we construct self-ensembling consistency for knowledge transfer between both domains, as well as a self-ensembling adversarial learning module to achieve better feature alignment for UDA. To assess the effectiveness of our method, we conduct extensive experiments on two different tasks for cross-modality segmentation between MRI and CT images. Experimental results demonstrate that the proposed LE-UDA can efficiently leverage limited source labels to improve cross-domain segmentation performance, outperforming state-of-the-art UDA approaches in the literature.
Collapse
|
122
|
Sendra-Balcells C, Campello VM, Torrents-Barrena J, Ahmed YA, Elattar M, Ohene-Botwe B, Nyangulu P, Stones W, Ammar M, Benamer LN, Kisembo HN, Sereke SG, Wanyonyi SZ, Temmerman M, Gratacós E, Bonet E, Eixarch E, Mikolaj K, Tolsgaard MG, Lekadir K. Generalisability of fetal ultrasound deep learning models to low-resource imaging settings in five African countries. Sci Rep 2023; 13:2728. [PMID: 36792642 PMCID: PMC9932015 DOI: 10.1038/s41598-023-29490-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Accepted: 02/06/2023] [Indexed: 02/17/2023] Open
Abstract
Most artificial intelligence (AI) research and innovations have concentrated in high-income countries, where imaging data, IT infrastructures and clinical expertise are plentiful. However, slower progress has been made in limited-resource environments where medical imaging is needed. For example, in Sub-Saharan Africa, the rate of perinatal mortality is very high due to limited access to antenatal screening. In these countries, AI models could be implemented to help clinicians acquire fetal ultrasound planes for the diagnosis of fetal abnormalities. So far, deep learning models have been proposed to identify standard fetal planes, but there is no evidence of their ability to generalise in centres with low resources, i.e. with limited access to high-end ultrasound equipment and ultrasound data. This work investigates for the first time different strategies to reduce the domain-shift effect arising from a fetal plane classification model trained on one clinical centre with high-resource settings and transferred to a new centre with low-resource settings. To that end, a classifier trained with 1792 patients from Spain is first evaluated on a new centre in Denmark in optimal conditions with 1008 patients and is later optimised to reach the same performance in five African centres (Egypt, Algeria, Uganda, Ghana and Malawi) with 25 patients each. The results show that a transfer learning approach for domain adaptation can be a solution to integrate small-size African samples with existing large-scale databases in developed countries. In particular, the model can be re-aligned and optimised to boost the performance on African populations by increasing the recall to [Formula: see text] and at the same time maintaining a high precision across centres. This framework shows promise for building new AI models generalisable across clinical centres with limited data acquired in challenging and heterogeneous conditions and calls for further research to develop new solutions for the usability of AI in countries with fewer resources and, consequently, in higher need of clinical support.
Collapse
Affiliation(s)
- Carla Sendra-Balcells
- Departament de Matemàtiques i Informàtica, Universitat de Barcelona, Barcelona, Spain.
| | - Víctor M. Campello
- grid.5841.80000 0004 1937 0247Departament de Matemàtiques i Informàtica, Universitat de Barcelona, Barcelona, Spain
| | | | - Yahya Ali Ahmed
- grid.430657.30000 0004 4699 3087Obstetrics and Gynecology Department, School of Medicine, Suez University,
Suez, Egypt
| | - Mustafa Elattar
- grid.440877.80000 0004 0377 5987Medical Imaging and Image Processing, Center of Informatics Science, Nile University, Sheikh Zayed City
, Egypt ,Research and Development Division, Intixel, Cairo
, Egypt
| | - Benard Ohene-Botwe
- grid.8652.90000 0004 1937 1485Department of Radiography, School of Biomedical and Allied Health Sciences, College of Health Sciences, University of Ghana, Accra
, Ghana ,grid.4464.20000 0001 2161 2573Division of Midwifery and Radiography, School of Health and Psychological Sciences, University of London, London, UK
| | - Pempho Nyangulu
- grid.517969.5Kamuzu University of Health Sciences, Blantyre, Malawi
| | - William Stones
- grid.517969.5Kamuzu University of Health Sciences, Blantyre, Malawi
| | - Mohammed Ammar
- Department of Electrical Engineering Systems, Laboratory of Engineering System and Telecommunication, University of M’Hamed Bougara Boumerdes, Algiers, Algeria
| | - Lamya Nawal Benamer
- Obstetrics and Gynecology Department, School of Medicine, Algiers University,
Algiers, Algeria
| | | | - Senai Goitom Sereke
- grid.11194.3c0000 0004 0620 0548Department of Radiology and Radiotherapy, School of Medicine, Makerere University College of Health Sciences, Kampala, Uganda
| | - Sikolia Z. Wanyonyi
- grid.411192.e0000 0004 1756 6158Department of Obstetrics and Gynaecology, Aga Khan University Hospital, 3rd Parklands Avenue, Nairobi, Kenya
| | - Marleen Temmerman
- grid.470490.eCentre of Excellence in Women and Child Health, Aga Khan University, Nairobi, Kenya
| | - Eduard Gratacós
- grid.5841.80000 0004 1937 0247BCNatal Fetal Medicine Research Center, Hospital Clínic and Hospital Sant Joan de Déu, Universitat de Barcelona, Barcelona, Spain ,grid.10403.360000000091771775Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain ,grid.452372.50000 0004 1791 1185Centre for Biomedical Research on Rare Diseases (CIBER-ER), Barcelona, Spain
| | - Elisenda Bonet
- grid.5841.80000 0004 1937 0247BCNatal Fetal Medicine Research Center, Hospital Clínic and Hospital Sant Joan de Déu, Universitat de Barcelona, Barcelona, Spain ,grid.6835.80000 0004 1937 028XBarcelona Tech, Universitat Politècnica de Catalunya, Barcelona, Spain
| | - Elisenda Eixarch
- grid.5841.80000 0004 1937 0247BCNatal Fetal Medicine Research Center, Hospital Clínic and Hospital Sant Joan de Déu, Universitat de Barcelona, Barcelona, Spain ,grid.10403.360000000091771775Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain ,grid.452372.50000 0004 1791 1185Centre for Biomedical Research on Rare Diseases (CIBER-ER), Barcelona, Spain
| | - Kamil Mikolaj
- grid.475435.4Copenhagen Academy for Medical Education and Simulation and Department of Obstetrics, Rigshospitalet, Copenhagen, Denmark
| | - Martin Grønnebæk Tolsgaard
- grid.475435.4Copenhagen Academy for Medical Education and Simulation and Department of Obstetrics, Rigshospitalet, Copenhagen, Denmark
| | - Karim Lekadir
- grid.5841.80000 0004 1937 0247Departament de Matemàtiques i Informàtica, Universitat de Barcelona, Barcelona, Spain
| |
Collapse
|
123
|
Zhao Y, Wang X, Che T, Bao G, Li S. Multi-task deep learning for medical image computing and analysis: A review. Comput Biol Med 2023; 153:106496. [PMID: 36634599 DOI: 10.1016/j.compbiomed.2022.106496] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Revised: 12/06/2022] [Accepted: 12/27/2022] [Indexed: 12/29/2022]
Abstract
The renaissance of deep learning has provided promising solutions to various tasks. While conventional deep learning models are constructed for a single specific task, multi-task deep learning (MTDL) that is capable to simultaneously accomplish at least two tasks has attracted research attention. MTDL is a joint learning paradigm that harnesses the inherent correlation of multiple related tasks to achieve reciprocal benefits in improving performance, enhancing generalizability, and reducing the overall computational cost. This review focuses on the advanced applications of MTDL for medical image computing and analysis. We first summarize four popular MTDL network architectures (i.e., cascaded, parallel, interacted, and hybrid). Then, we review the representative MTDL-based networks for eight application areas, including the brain, eye, chest, cardiac, abdomen, musculoskeletal, pathology, and other human body regions. While MTDL-based medical image processing has been flourishing and demonstrating outstanding performance in many tasks, in the meanwhile, there are performance gaps in some tasks, and accordingly we perceive the open challenges and the perspective trends. For instance, in the 2018 Ischemic Stroke Lesion Segmentation challenge, the reported top dice score of 0.51 and top recall of 0.55 achieved by the cascaded MTDL model indicate further research efforts in high demand to escalate the performance of current models.
Collapse
Affiliation(s)
- Yan Zhao
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100083, China
| | - Xiuying Wang
- School of Computer Science, The University of Sydney, Sydney, NSW, 2008, Australia.
| | - Tongtong Che
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100083, China
| | - Guoqing Bao
- School of Computer Science, The University of Sydney, Sydney, NSW, 2008, Australia
| | - Shuyu Li
- State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|
124
|
Kunapinun A, Dailey MN, Songsaeng D, Parnichkun M, Keatmanee C, Ekpanyapong M. Improving GAN Learning Dynamics for Thyroid Nodule Segmentation. ULTRASOUND IN MEDICINE & BIOLOGY 2023; 49:416-430. [PMID: 36424307 DOI: 10.1016/j.ultrasmedbio.2022.09.010] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2022] [Revised: 06/14/2022] [Accepted: 09/14/2022] [Indexed: 06/16/2023]
Abstract
Thyroid nodules are lesions requiring diagnosis and follow-up. Tools for detecting and segmenting nodules can help physicians with this diagnosis. Besides immediate diagnosis, automated tools can also enable tracking of the probability of malignancy over time. This paper demonstrates a new algorithm for segmenting thyroid nodules in ultrasound images. The algorithm combines traditional supervised semantic segmentation with unsupervised learning using GANs. The hybrid approach has the potential to upgrade the semantic segmentation model's performance, but GANs have the well-known problems of unstable learning and mode collapse. To stabilize the training of the GAN model, we introduce the concept of closed-loop control of the gain on the loss output of the discriminator. We find gain control leads to smoother generator training and avoids the mode collapse that typically occurs when the discriminator learns too quickly relative to the generator. We also find that the combination of the supervised and unsupervised learning styles encourages both low-level accuracy and high-level consistency. As a test of the concept of controlled hybrid supervised and unsupervised semantic segmentation, we introduce a new model named the StableSeg GAN. The model uses DeeplabV3+ as the generator, Resnet18 as the discriminator, and uses PID control to stabilize the GAN learning process. The performance of the new model in terms of IoU is better than DeeplabV3+, with mean IoU of 81.26% on a challenging test set. The results of our thyroid nodule segmentation experiments show that StableSeg GANs have flexibility to segment nodules more accurately than either comparable supervised segmentation models or uncontrolled GANs.
Collapse
Affiliation(s)
- Alisa Kunapinun
- Industrial Systems Engineering Department, Asian Institute of Technology, Pathumthani, Thailand
| | - Matthew N Dailey
- Information and Communication Technologies, Asian Institute of Technology, Pathumthani, Thailand
| | - Dittapong Songsaeng
- Department of Radiology, Faculty of Medicine, Siriraj Hospital, Mahidol University, Bangkok, Thailand
| | - Manukid Parnichkun
- Industrial Systems Engineering Department, Asian Institute of Technology, Pathumthani, Thailand
| | | | - Mongkol Ekpanyapong
- Industrial Systems Engineering Department, Asian Institute of Technology, Pathumthani, Thailand.
| |
Collapse
|
125
|
Atasever S, Azginoglu N, Terzi DS, Terzi R. A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning. Clin Imaging 2023; 94:18-41. [PMID: 36462229 DOI: 10.1016/j.clinimag.2022.11.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Revised: 10/17/2022] [Accepted: 11/01/2022] [Indexed: 11/13/2022]
Abstract
This survey aims to identify commonly used methods, datasets, future trends, knowledge gaps, constraints, and limitations in the field to provide an overview of current solutions used in medical image analysis in parallel with the rapid developments in transfer learning (TL). Unlike previous studies, this survey grouped the last five years of current studies for the period between January 2017 and February 2021 according to different anatomical regions and detailed the modality, medical task, TL method, source data, target data, and public or private datasets used in medical imaging. Also, it provides readers with detailed information on technical challenges, opportunities, and future research trends. In this way, an overview of recent developments is provided to help researchers to select the most effective and efficient methods and access widely used and publicly available medical datasets, research gaps, and limitations of the available literature.
Collapse
Affiliation(s)
- Sema Atasever
- Computer Engineering Department, Nevsehir Hacı Bektas Veli University, Nevsehir, Turkey.
| | - Nuh Azginoglu
- Computer Engineering Department, Kayseri University, Kayseri, Turkey.
| | | | - Ramazan Terzi
- Computer Engineering Department, Amasya University, Amasya, Turkey.
| |
Collapse
|
126
|
Rodrigues Moreira LF, Moreira R, Travençolo BAN, Backes AR. An Artificial Intelligence-as-a-Service Architecture for deep learning model embodiment on low-cost devices: A case study of COVID-19 diagnosis. Appl Soft Comput 2023; 134:110014. [PMID: 36687763 PMCID: PMC9837155 DOI: 10.1016/j.asoc.2023.110014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Revised: 12/03/2022] [Accepted: 01/07/2023] [Indexed: 01/15/2023]
Abstract
Coronavirus Disease-2019 (COVID-19) causes Severe Acute Respiratory Syndrome-Corona Virus-2 (SARS-CoV-2) and has opened several challenges for research concerning diagnosis and treatment. Chest X-rays and computed tomography (CT) scans are effective and fast alternatives to detect and assess the damage that COVID causes to the lungs at different stages of the disease. Although the CT scan is an accurate exam, the chest X-ray is still helpful due to the cheaper, faster, lower radiation exposure, and is available in low-incoming countries. Computer-aided diagnostic systems based on Artificial Intelligence (AI) and computer vision are an alternative to extract features from X-ray images, providing an accurate COVID-19 diagnosis. However, specialized and expensive computational resources come across as challenging. Also, it needs to be better understood how low-cost devices and smartphones can hold AI models to predict diseases timely. Even using deep learning to support image-based medical diagnosis, challenges still need to be addressed once the known techniques use centralized intelligence on high-performance servers, making it difficult to embed these models in low-cost devices. This paper sheds light on these questions by proposing the Artificial Intelligence as a Service Architecture (AIaaS), a hybrid AI support operation, both centralized and distributed, with the purpose of enabling the embedding of already-trained models on low-cost devices or smartphones. We demonstrated the suitability of our architecture through a case study of COVID-19 diagnosis using a low-cost device. Among the main findings of this paper, we point out the performance evaluation of low-cost devices to handle COVID-19 predicting tasks timely and accurately and the quantitative performance evaluation of CNN models embodiment on low-cost devices.
Collapse
Affiliation(s)
| | - Rodrigo Moreira
- Institute of Exacts and Technological Sciences (IEP), Federal University of Viçosa, Rio Paranaíba, Minas Gerais, Brazil
| | | | - André Ricardo Backes
- Department of Computing (DC), Federal University of São Carlos, São Carlos, São Paulo, Brazil
| |
Collapse
|
127
|
Hadjiiski L, Cha K, Chan HP, Drukker K, Morra L, Näppi JJ, Sahiner B, Yoshida H, Chen Q, Deserno TM, Greenspan H, Huisman H, Huo Z, Mazurchuk R, Petrick N, Regge D, Samala R, Summers RM, Suzuki K, Tourassi G, Vergara D, Armato SG. AAPM task group report 273: Recommendations on best practices for AI and machine learning for computer-aided diagnosis in medical imaging. Med Phys 2023; 50:e1-e24. [PMID: 36565447 DOI: 10.1002/mp.16188] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Revised: 11/13/2022] [Accepted: 11/22/2022] [Indexed: 12/25/2022] Open
Abstract
Rapid advances in artificial intelligence (AI) and machine learning, and specifically in deep learning (DL) techniques, have enabled broad application of these methods in health care. The promise of the DL approach has spurred further interest in computer-aided diagnosis (CAD) development and applications using both "traditional" machine learning methods and newer DL-based methods. We use the term CAD-AI to refer to this expanded clinical decision support environment that uses traditional and DL-based AI methods. Numerous studies have been published to date on the development of machine learning tools for computer-aided, or AI-assisted, clinical tasks. However, most of these machine learning models are not ready for clinical deployment. It is of paramount importance to ensure that a clinical decision support tool undergoes proper training and rigorous validation of its generalizability and robustness before adoption for patient care in the clinic. To address these important issues, the American Association of Physicists in Medicine (AAPM) Computer-Aided Image Analysis Subcommittee (CADSC) is charged, in part, to develop recommendations on practices and standards for the development and performance assessment of computer-aided decision support systems. The committee has previously published two opinion papers on the evaluation of CAD systems and issues associated with user training and quality assurance of these systems in the clinic. With machine learning techniques continuing to evolve and CAD applications expanding to new stages of the patient care process, the current task group report considers the broader issues common to the development of most, if not all, CAD-AI applications and their translation from the bench to the clinic. The goal is to bring attention to the proper training and validation of machine learning algorithms that may improve their generalizability and reliability and accelerate the adoption of CAD-AI systems for clinical decision support.
Collapse
Affiliation(s)
- Lubomir Hadjiiski
- Department of Radiology, University of Michigan, Ann Arbor, Michigan, USA
| | - Kenny Cha
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | - Heang-Ping Chan
- Department of Radiology, University of Michigan, Ann Arbor, Michigan, USA
| | - Karen Drukker
- Department of Radiology, University of Chicago, Chicago, Illinois, USA
| | - Lia Morra
- Department of Control and Computer Engineering, Politecnico di Torino, Torino, Italy
| | - Janne J Näppi
- 3D Imaging Research, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Berkman Sahiner
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | - Hiroyuki Yoshida
- 3D Imaging Research, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Quan Chen
- Department of Radiation Medicine, University of Kentucky, Lexington, Kentucky, USA
| | - Thomas M Deserno
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Braunschweig, Germany
| | - Hayit Greenspan
- Department of Biomedical Engineering, Faculty of Engineering, Tel Aviv, Israel & Department of Radiology, Ichan School of Medicine, Tel Aviv University, Mt Sinai, New York, New York, USA
| | - Henkjan Huisman
- Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Zhimin Huo
- Tencent America, Palo Alto, California, USA
| | - Richard Mazurchuk
- Division of Cancer Prevention, National Cancer Institute, National Institutes of Health, Bethesda, Maryland, USA
| | | | - Daniele Regge
- Radiology Unit, Candiolo Cancer Institute, FPO-IRCCS, Candiolo, Italy.,Department of Surgical Sciences, University of Turin, Turin, Italy
| | - Ravi Samala
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | - Ronald M Summers
- Radiology and Imaging Sciences, National Institutes of Health Clinical Center, Bethesda, Maryland, USA
| | - Kenji Suzuki
- Institute of Innovative Research, Tokyo Institute of Technology, Tokyo, Japan
| | | | - Daniel Vergara
- Department of Radiology, Yale New Haven Hospital, New Haven, Connecticut, USA
| | - Samuel G Armato
- Department of Radiology, University of Chicago, Chicago, Illinois, USA
| |
Collapse
|
128
|
Uncertain-CAM: Uncertainty-Based Ensemble Machine Voting for Improved COVID-19 CXR Classification and Explainability. Diagnostics (Basel) 2023; 13:diagnostics13030441. [PMID: 36766546 PMCID: PMC9914375 DOI: 10.3390/diagnostics13030441] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2022] [Revised: 01/08/2023] [Accepted: 01/20/2023] [Indexed: 01/27/2023] Open
Abstract
The ongoing coronavirus disease 2019 (COVID-19) pandemic has had a significant impact on patients and healthcare systems across the world. Distinguishing non-COVID-19 patients from COVID-19 patients at the lowest possible cost and in the earliest stages of the disease is a major issue. Additionally, the implementation of explainable deep learning decisions is another issue, especially in critical fields such as medicine. The study presents a method to train deep learning models and apply an uncertainty-based ensemble voting policy to achieve 99% accuracy in classifying COVID-19 chest X-rays from normal and pneumonia-related infections. We further present a training scheme that integrates the cyclic cosine annealing approach with cross-validation and uncertainty quantification that is measured using prediction interval coverage probability (PICP) as final ensemble voting weights. We also propose the Uncertain-CAM technique, which improves deep learning explainability and provides a more reliable COVID-19 classification system. We introduce a new image processing technique to measure the explainability based on ground-truth, and we compared it with the widely adopted Grad-CAM method.
Collapse
|
129
|
Snider EJ, Hernandez-Torres SI, Hennessey R. Using Ultrasound Image Augmentation and Ensemble Predictions to Prevent Machine-Learning Model Overfitting. Diagnostics (Basel) 2023; 13:diagnostics13030417. [PMID: 36766522 PMCID: PMC9914871 DOI: 10.3390/diagnostics13030417] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2022] [Revised: 01/07/2023] [Accepted: 01/18/2023] [Indexed: 01/26/2023] Open
Abstract
Deep learning predictive models have the potential to simplify and automate medical imaging diagnostics by lowering the skill threshold for image interpretation. However, this requires predictive models that are generalized to handle subject variability as seen clinically. Here, we highlight methods to improve test accuracy of an image classifier model for shrapnel identification using tissue phantom image sets. Using a previously developed image classifier neural network-termed ShrapML-blind test accuracy was less than 70% and was variable depending on the training/test data setup, as determined by a leave one subject out (LOSO) holdout methodology. Introduction of affine transformations for image augmentation or MixUp methodologies to generate additional training sets improved model performance and overall accuracy improved to 75%. Further improvements were made by aggregating predictions across five LOSO holdouts. This was done by bagging confidences or predictions from all LOSOs or the top-3 LOSO confidence models for each image prediction. Top-3 LOSO confidence bagging performed best, with test accuracy improved to greater than 85% accuracy for two different blind tissue phantoms. This was confirmed by gradient-weighted class activation mapping to highlight that the image classifier was tracking shrapnel in the image sets. Overall, data augmentation and ensemble prediction approaches were suitable for creating more generalized predictive models for ultrasound image analysis, a critical step for real-time diagnostic deployment.
Collapse
|
130
|
Jonsson T. Micro-CT and deep learning: Modern techniques and applications in insect morphology and neuroscience. FRONTIERS IN INSECT SCIENCE 2023; 3:1016277. [PMID: 38469492 PMCID: PMC10926430 DOI: 10.3389/finsc.2023.1016277] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/10/2022] [Accepted: 01/06/2023] [Indexed: 03/13/2024]
Abstract
Advances in modern imaging and computer technologies have led to a steady rise in the use of micro-computed tomography (µCT) in many biological areas. In zoological research, this fast and non-destructive method for producing high-resolution, two- and three-dimensional images is increasingly being used for the functional analysis of the external and internal anatomy of animals. µCT is hereby no longer limited to the analysis of specific biological tissues in a medical or preclinical context but can be combined with a variety of contrast agents to study form and function of all kinds of tissues and species, from mammals and reptiles to fish and microscopic invertebrates. Concurrently, advances in the field of artificial intelligence, especially in deep learning, have revolutionised computer vision and facilitated the automatic, fast and ever more accurate analysis of two- and three-dimensional image datasets. Here, I want to give a brief overview of both micro-computed tomography and deep learning and present their recent applications, especially within the field of insect science. Furthermore, the combination of both approaches to investigate neural tissues and the resulting potential for the analysis of insect sensory systems, from receptor structures via neuronal pathways to the brain, are discussed.
Collapse
Affiliation(s)
- Thorin Jonsson
- Institute of Biology, Karl-Franzens-University Graz, Graz, Austria
| |
Collapse
|
131
|
Rahman Siddiquee MM, Shah J, Chong C, Nikolova S, Dumkrieger G, Li B, Wu T, Schwedt TJ. Headache classification and automatic biomarker extraction from structural MRIs using deep learning. Brain Commun 2023; 5:fcac311. [PMID: 36751567 PMCID: PMC9897182 DOI: 10.1093/braincomms/fcac311] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2022] [Revised: 08/24/2022] [Accepted: 11/24/2022] [Indexed: 11/29/2022] Open
Abstract
Data-driven machine-learning methods on neuroimaging (e.g. MRI) are of great interest for the investigation and classification of neurological diseases. However, traditional machine learning requires domain knowledge to delineate the brain regions first, followed by feature extraction from the regions. Compared with this semi-automated approach, recently developed deep learning methods have advantages since they do not require such prior knowledge; instead, deep learning methods can automatically find features that differentiate MRIs from different cohorts. In the present study, we developed a deep learning-based classification pipeline distinguishing brain MRIs of individuals with one of three types of headaches [migraine (n = 95), acute post-traumatic headache (n = 48) and persistent post-traumatic headache (n = 49)] from those of healthy controls (n = 532) and identified the brain regions that most contributed to each classification task. Our pipeline included: (i) data preprocessing; (ii) binary classification of healthy controls versus headache type using a 3D ResNet-18; and (iii) biomarker extraction from the trained 3D ResNet-18. During the classification at the second step of our pipeline, we resolved two common issues in deep learning methods, limited training data and imbalanced samples from different categories, by incorporating a large public data set and resampling among the headache cohorts. Our method achieved the following classification accuracies when tested on independent test sets: (i) migraine versus healthy controls-75% accuracy, 66.7% sensitivity and 83.3% specificity; (2) acute post-traumatic headache versus healthy controls-75% accuracy, 66.7% sensitivity and 83.3% specificity; and (3) persistent post-traumatic headache versus healthy controls-91.7% accuracy, 100% sensitivity and 83.3% specificity. The most significant biomarkers identified by the classifier for migraine were caudate, caudal anterior cingulate, superior frontal, thalamus and ventral diencephalon. For acute post-traumatic headache, lateral occipital, cuneus, lingual, pericalcarine and superior parietal regions were identified as most significant biomarkers. Finally, for persistent post-traumatic headache, the most significant biomarkers were cerebellum, middle temporal, inferior temporal, inferior parietal and superior parietal. In conclusion, our study shows that the deep learning methods can automatically detect aberrations in the brain regions associated with different headache types. It does not require any human knowledge as input which significantly reduces human effort. It uncovers the great potential of deep learning methods for classification and automatic extraction of brain imaging-based biomarkers for these headache types.
Collapse
Affiliation(s)
- Md Mahfuzur Rahman Siddiquee
- School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA.,ASU-Mayo Center for Innovative Imaging, Tempe, AZ, USA
| | - Jay Shah
- School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA.,ASU-Mayo Center for Innovative Imaging, Tempe, AZ, USA
| | - Catherine Chong
- ASU-Mayo Center for Innovative Imaging, Tempe, AZ, USA.,Department of Neurology, Mayo Clinic, Phoenix, AZ, USA
| | | | | | - Baoxin Li
- School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA.,ASU-Mayo Center for Innovative Imaging, Tempe, AZ, USA
| | - Teresa Wu
- School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA.,ASU-Mayo Center for Innovative Imaging, Tempe, AZ, USA
| | - Todd J Schwedt
- ASU-Mayo Center for Innovative Imaging, Tempe, AZ, USA.,Department of Neurology, Mayo Clinic, Phoenix, AZ, USA
| |
Collapse
|
132
|
Gao Z, Hong B, Li Y, Zhang X, Wu J, Wang C, Zhang X, Gong T, Zheng Y, Meng D, Li C. A semi-supervised multi-task learning framework for cancer classification with weak annotation in whole-slide images. Med Image Anal 2023; 83:102652. [PMID: 36327654 DOI: 10.1016/j.media.2022.102652] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2021] [Revised: 09/15/2022] [Accepted: 10/08/2022] [Indexed: 11/06/2022]
Abstract
Cancer region detection (CRD) and subtyping are two fundamental tasks in digital pathology image analysis. The development of data-driven models for CRD and subtyping on whole-slide images (WSIs) would mitigate the burden of pathologists and improve their accuracy in diagnosis. However, the existing models are facing two major limitations. Firstly, they typically require large-scale datasets with precise annotations, which contradicts with the original intention of reducing labor effort. Secondly, for the subtyping task, the non-cancerous regions are treated as the same as cancerous regions within a WSI, which confuses a subtyping model in its training process. To tackle the latter limitation, the previous research proposed to perform CRD first for ruling out the non-cancerous region, then train a subtyping model based on the remaining cancerous patches. However, separately training ignores the interaction of these two tasks, also leads to propagating the error of the CRD task to the subtyping task. To address these issues and concurrently improve the performance on both CRD and subtyping tasks, we propose a semi-supervised multi-task learning (MTL) framework for cancer classification. Our framework consists of a backbone feature extractor, two task-specific classifiers, and a weight control mechanism. The backbone feature extractor is shared by two task-specific classifiers, such that the interaction of CRD and subtyping tasks can be captured. The weight control mechanism preserves the sequential relationship of these two tasks and guarantees the error back-propagation from the subtyping task to the CRD task under the MTL framework. We train the overall framework in a semi-supervised setting, where datasets only involve small quantities of annotations produced by our minimal point-based (min-point) annotation strategy. Extensive experiments on four large datasets with different cancer types demonstrate the effectiveness of the proposed framework in both accuracy and generalization.
Collapse
Affiliation(s)
- Zeyu Gao
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, Xi'an 710049, China
| | - Bangyang Hong
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, Xi'an 710049, China
| | - Yang Li
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, Xi'an 710049, China
| | - Xianli Zhang
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, Xi'an 710049, China
| | - Jialun Wu
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, Xi'an 710049, China
| | - Chunbao Wang
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Department of Pathology, The First Affiliated Hospital of Xi'an Jiaotong University, Xi'an 710061, China
| | - Xiangrong Zhang
- School of Artificial Intelligence, Xidian University, Xi'an 710071, China
| | - Tieliang Gong
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, Xi'an 710049, China
| | - Yefeng Zheng
- Tencent Jarvis Lab, Shenzhen, Guangdong 518075, China
| | - Deyu Meng
- School of Mathematics and Statistics, Xi'an Jiaotong University, Xi'an 710049, China
| | - Chen Li
- School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China; Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, Xi'an 710049, China.
| |
Collapse
|
133
|
Saad MH, Hashima S, Sayed W, El-Shazly EH, Madian AH, Fouda MM. Early Diagnosis of COVID-19 Images Using Optimal CNN Hyperparameters. Diagnostics (Basel) 2022; 13:diagnostics13010076. [PMID: 36611368 PMCID: PMC9818649 DOI: 10.3390/diagnostics13010076] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2022] [Revised: 12/13/2022] [Accepted: 12/19/2022] [Indexed: 12/29/2022] Open
Abstract
Coronavirus disease (COVID-19) is a worldwide epidemic that poses substantial health hazards. However, COVID-19 diagnostic test sensitivity is still restricted due to abnormalities in specimen processing. Meanwhile, optimizing the highly defined number of convolutional neural network (CNN) hyperparameters (hundreds to thousands) is a useful direction to improve its overall performance and overcome its cons. Hence, this paper proposes an optimization strategy for obtaining the optimal learning rate and momentum of a CNN's hyperparameters using the grid search method to improve the network performance. Therefore, three alternative CNN architectures (GoogleNet, VGG16, and ResNet) were used to optimize hyperparameters utilizing two different COVID-19 radiography data sets (Kaggle (X-ray) and China national center for bio-information (CT)). These architectures were tested with/without optimizing the hyperparameters. The results confirm effective disease classification using the CNN structures with optimized hyperparameters. Experimental findings indicate that the new technique outperformed the previous in terms of accuracy, sensitivity, specificity, recall, F-score, false positive and negative rates, and error rate. At epoch 25, the optimized Resnet obtained high classification accuracy, reaching 98.98% for X-ray images and 98.78% for CT images.
Collapse
Affiliation(s)
- Mohamed H. Saad
- Radiation Engineering Department, National Center for Radiation Research and Technology (NCRRT), Egyptian Atomic Energy Authority, Cairo 11787, Egypt
| | - Sherief Hashima
- Engineering Department, Nuclear Research Center (NRC), Egyptian Atomic Energy Authority, Cairo 13759, Egypt
- Correspondence: ; Tel.: +20-10-94230077
| | - Wessam Sayed
- Radiation Engineering Department, National Center for Radiation Research and Technology (NCRRT), Egyptian Atomic Energy Authority, Cairo 11787, Egypt
| | - Ehab H. El-Shazly
- Radiation Engineering Department, National Center for Radiation Research and Technology (NCRRT), Egyptian Atomic Energy Authority, Cairo 11787, Egypt
| | - Ahmed H. Madian
- Radiation Engineering Department, National Center for Radiation Research and Technology (NCRRT), Egyptian Atomic Energy Authority, Cairo 11787, Egypt
| | - Mostafa M. Fouda
- Department of Electrical and Computer Engineering, Idaho State University, Pocatello, ID 83209, USA
- Department of Electrical Engineering, Faculty of Engineering at Shoubra, Benha University, Cairo 11672, Egypt
| |
Collapse
|
134
|
Convolutional neural network for automated segmentation of the liver and its vessels on non-contrast T1 vibe Dixon acquisitions. Sci Rep 2022; 12:22059. [PMID: 36543852 PMCID: PMC9772168 DOI: 10.1038/s41598-022-26328-2] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Accepted: 12/13/2022] [Indexed: 12/24/2022] Open
Abstract
We evaluated the effectiveness of automated segmentation of the liver and its vessels with a convolutional neural network on non-contrast T1 vibe Dixon acquisitions. A dataset of non-contrast T1 vibe Dixon liver magnetic resonance images was labelled slice-by-slice for the outer liver border, portal, and hepatic veins by an expert. A 3D U-Net convolutional neural network was trained with different combinations of Dixon in-phase, opposed-phase, water, and fat reconstructions. The neural network trained with the single-modal in-phase reconstructions achieved a high performance for liver parenchyma (Dice 0.936 ± 0.02), portal veins (0.634 ± 0.09), and hepatic veins (0.532 ± 0.12) segmentation. No benefit of using multi-modal input was observed (p = 1.0 for all experiments), combining in-phase, opposed-phase, fat, and water reconstruction. Accuracy for differentiation between portal and hepatic veins was 99% for portal veins and 97% for hepatic veins in the central region and slightly lower in the peripheral region (91% for portal veins, 80% for hepatic veins). In conclusion, deep learning-based automated segmentation of the liver and its vessels on non-contrast T1 vibe Dixon was highly effective. The single-modal in-phase input achieved the best performance in segmentation and differentiation between portal and hepatic veins.
Collapse
|
135
|
Deep Learning Model for Computer-Aided Diagnosis of Urolithiasis Detection from Kidney-Ureter-Bladder Images. BIOENGINEERING (BASEL, SWITZERLAND) 2022; 9:bioengineering9120811. [PMID: 36551017 PMCID: PMC9774756 DOI: 10.3390/bioengineering9120811] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Revised: 12/08/2022] [Accepted: 12/12/2022] [Indexed: 12/24/2022]
Abstract
Kidney-ureter-bladder (KUB) imaging is a radiological examination with a low cost, low radiation, and convenience. Although emergency room clinicians can arrange KUB images easily as a first-line examination for patients with suspicious urolithiasis, interpreting the KUB images correctly is difficult for inexperienced clinicians. Obtaining a formal radiology report immediately after a KUB imaging examination can also be challenging. Recently, artificial-intelligence-based computer-aided diagnosis (CAD) systems have been developed to help clinicians who are not experts make correct diagnoses for further treatment more effectively. Therefore, in this study, we proposed a CAD system for KUB imaging based on a deep learning model designed to help first-line emergency room clinicians diagnose urolithiasis accurately. A total of 355 KUB images were retrospectively collected from 104 patients who were diagnosed with urolithiasis at Kaohsiung Chang Gung Memorial Hospital. Then, we trained a deep learning model with a ResNet architecture to classify KUB images in terms of the presence or absence of kidney stones with this dataset of pre-processed images. Finally, we tuned the parameters and tested the model experimentally. The results show that the accuracy, sensitivity, specificity, and F1-measure of the model were 0.977, 0.953, 1, and 0.976 on the validation set and 0.982, 0.964, 1, and 0.982 on the testing set, respectively. Moreover, the results demonstrate that the proposed model performed well compared to the existing CNN-based methods and was able to detect urolithiasis in KUB images successfully. We expect the proposed approach to help emergency room clinicians make accurate diagnoses and reduce unnecessary radiation exposure from computed tomography (CT) scans, along with the associated medical costs.
Collapse
|
136
|
Papanastasiou G, García Seco de Herrera A, Wang C, Zhang H, Yang G, Wang G. Focus on machine learning models in medical imaging. Phys Med Biol 2022; 68:010301. [PMID: 36594883 DOI: 10.1088/1361-6560/aca069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Accepted: 11/04/2022] [Indexed: 12/23/2022]
Affiliation(s)
| | | | | | - Heye Zhang
- Sun Yat-sen University, People's Republic of China
| | | | - Ge Wang
- Rensselaer Polytechnic Institute, United States of America
| |
Collapse
|
137
|
Ma Z, Feng D, Wang J, Ma H. Retinal OCTA Image Segmentation Based on Global Contrastive Learning. SENSORS (BASEL, SWITZERLAND) 2022; 22:9847. [PMID: 36560216 PMCID: PMC9781437 DOI: 10.3390/s22249847] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Revised: 12/11/2022] [Accepted: 12/12/2022] [Indexed: 06/17/2023]
Abstract
The automatic segmentation of retinal vessels is of great significance for the analysis and diagnosis of retinal related diseases. However, the imbalanced data in retinal vascular images remain a great challenge. Current image segmentation methods based on deep learning almost always focus on local information in a single image while ignoring the global information of the entire dataset. To solve the problem of data imbalance in optical coherence tomography angiography (OCTA) datasets, this paper proposes a medical image segmentation method (contrastive OCTA segmentation net, COSNet) based on global contrastive learning. First, the feature extraction module extracts the features of OCTA image input and maps them to the segment head and the multilayer perceptron (MLP) head, respectively. Second, a contrastive learning module saves the pixel queue and pixel embedding of each category in the feature map into the memory bank, generates sample pairs through a mixed sampling strategy to construct a new contrastive loss function, and forces the network to learn local information and global information simultaneously. Finally, the segmented image is fine tuned to restore positional information of deep vessels. The experimental results show the proposed method can improve the accuracy (ACC), the area under the curve (AUC), and other evaluation indexes of image segmentation compared with the existing methods. This method could accomplish segmentation tasks in imbalanced data and extend to other segmentation tasks.
Collapse
Affiliation(s)
- Ziping Ma
- College of Mathematics and Information Science, North Minzu University, Yinchuan 750021, China
| | - Dongxiu Feng
- College of Computer Science and Engineering, North Minzu University, Yinchuan 750021, China
| | - Jingyu Wang
- College of Mathematics and Information Science, North Minzu University, Yinchuan 750021, China
| | - Hu Ma
- College of Mathematics and Information Science, North Minzu University, Yinchuan 750021, China
| |
Collapse
|
138
|
Cohen Hochberg D, Greenspan H, Giryes R. A Self Supervised StyleGAN for Image Annotation and Classification With Extremely Limited Labels. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:3509-3519. [PMID: 35767509 DOI: 10.1109/tmi.2022.3187170] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2023]
Abstract
The recent success of learning-based algorithms can be greatly attributed to the immense amount of annotated data used for training. Yet, many datasets lack annotations due to the high costs associated with labeling, resulting in degraded performances of deep learning methods. Self-supervised learning is frequently adopted to mitigate the reliance on massive labeled datasets since it exploits unlabeled data to learn relevant feature representations. In this work, we propose SS-StyleGAN, a self-supervised approach for image annotation and classification suitable for extremely small annotated datasets. This novel framework adds self-supervision to the StyleGAN architecture by integrating an encoder that learns the embedding to the StyleGAN latent space, which is well-known for its disentangled properties. The learned latent space enables the smart selection of representatives from the data to be labeled for improved classification performance. We show that the proposed method attains strong classification results using small labeled datasets of sizes 50 and even 10. We demonstrate the superiority of our approach for the tasks of COVID-19 and liver tumor pathology identification.
Collapse
|
139
|
Hsu ST, Su YJ, Hung CH, Chen MJ, Lu CH, Kuo CE. Automatic ovarian tumors recognition system based on ensemble convolutional neural network with ultrasound imaging. BMC Med Inform Decis Mak 2022; 22:298. [PMID: 36397100 PMCID: PMC9673368 DOI: 10.1186/s12911-022-02047-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2022] [Accepted: 11/14/2022] [Indexed: 11/18/2022] Open
Abstract
Background Upon the discovery of ovarian cysts, obstetricians, gynecologists, and ultrasound examiners must address the common clinical challenge of distinguishing between benign and malignant ovarian tumors. Numerous types of ovarian tumors exist, many of which exhibit similar characteristics that increase the ambiguity in clinical diagnosis. Using deep learning technology, we aimed to develop a method that rapidly and accurately assists the different diagnosis of ovarian tumors in ultrasound images. Methods Based on deep learning method, we used ten well-known convolutional neural network models (e.g., Alexnet, GoogleNet, and ResNet) for training of transfer learning. To ensure method stability and robustness, we repeated the random sampling of the training and validation data ten times. The mean of the ten test results was set as the final assessment data. After the training process was completed, the three models with the highest ratio of calculation accuracy to time required for classification were used for ensemble learning pertaining. Finally, the interpretation results of the ensemble classifier were used as the final results. We also applied ensemble gradient-weighted class activation mapping (Grad-CAM) technology to visualize the decision-making results of the models. Results The highest mean accuracy, mean sensitivity, and mean specificity of ten single CNN models were 90.51 ± 4.36%, 89.77 ± 4.16%, and 92.00 ± 5.95%, respectively. The mean accuracy, mean sensitivity, and mean specificity of the ensemble classifier method were 92.15 ± 2.84%, 91.37 ± 3.60%, and 92.92 ± 4.00%, respectively. The performance of the ensemble classifier is better than that of a single classifier in three evaluation metrics. Moreover, the standard deviation is also better which means the ensemble classifier is more stable and robust. Conclusion From the comprehensive perspective of data quantity, data diversity, robustness of validation strategy, and overall accuracy, the proposed method outperformed the methods used in previous studies. In future studies, we will continue to increase the number of authenticated images and apply our proposed method in clinical settings to increase its robustness and reliability.
Collapse
|
140
|
Li Y, Xu C, Han J, An Z, Wang D, Ma H, Liu C. MHAU-Net: Skin Lesion Segmentation Based on Multi-Scale Hybrid Residual Attention Network. SENSORS (BASEL, SWITZERLAND) 2022; 22:8701. [PMID: 36433298 PMCID: PMC9695536 DOI: 10.3390/s22228701] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/01/2022] [Revised: 11/03/2022] [Accepted: 11/07/2022] [Indexed: 06/16/2023]
Abstract
Melanoma is a main factor that leads to skin cancer, and early diagnosis and treatment can significantly reduce the mortality of patients. Skin lesion boundary segmentation is a key to accurately localizing a lesion in dermoscopic images. However, the irregular shape and size of the lesions and the blurred boundary of the lesions pose significant challenges for researchers. In recent years, pixel-level semantic segmentation strategies based on convolutional neural networks have been widely used, but many methods still suffer from the inaccurate segmentation of fuzzy boundaries. In this paper, we proposed a multi-scale hybrid attentional convolutional neural network (MHAU-Net) for the precise localization and segmentation of skin lesions. MHAU-Net has four main components: multi-scale resolution input, hybrid residual attention (HRA), dilated convolution, and atrous spatial pyramid pooling. Multi-scale resolution inputs provide richer visual information, and HRA solves the problem of blurred boundaries and enhances the segmentation results. The Dice, mIoU, average specificity, and sensitivity on the ISIC2018 task 1 validation set were 93.69%, 90.02%, 92.7% and 93.9%, respectively. The segmentation metrics are significantly better than the latest DCSAU-Net, UNeXt, and U-Net, and excellent segmentation results are achieved on different datasets. We performed model robustness validations on the Kvasir-SEG dataset with an overall sensitivity and average specificity of 95.91% and 96.28%, respectively.
Collapse
Affiliation(s)
- Yingjie Li
- School of Integrated Circuits, Anhui University, Hefei 230601, China
- Anhui Engineering Laboratory of Agro-Ecological Big Data, Hefei 230601, China
| | - Chao Xu
- School of Integrated Circuits, Anhui University, Hefei 230601, China
- Anhui Engineering Laboratory of Agro-Ecological Big Data, Hefei 230601, China
| | - Jubao Han
- School of Integrated Circuits, Anhui University, Hefei 230601, China
- Anhui Engineering Laboratory of Agro-Ecological Big Data, Hefei 230601, China
| | - Ziheng An
- School of Integrated Circuits, Anhui University, Hefei 230601, China
- Anhui Engineering Laboratory of Agro-Ecological Big Data, Hefei 230601, China
| | - Deyu Wang
- School of Integrated Circuits, Anhui University, Hefei 230601, China
- Anhui Engineering Laboratory of Agro-Ecological Big Data, Hefei 230601, China
| | - Haichao Ma
- School of Integrated Circuits, Anhui University, Hefei 230601, China
- Anhui Engineering Laboratory of Agro-Ecological Big Data, Hefei 230601, China
| | - Chuanxu Liu
- School of Integrated Circuits, Anhui University, Hefei 230601, China
- Anhui Engineering Laboratory of Agro-Ecological Big Data, Hefei 230601, China
| |
Collapse
|
141
|
Udupa JK, Liu T, Jin C, Zhao L, Odhner D, Tong Y, Agrawal V, Pednekar G, Nag S, Kotia T, Goodman M, Wileyto EP, Mihailidis D, Lukens JN, Berman AT, Stambaugh J, Lim T, Chowdary R, Jalluri D, Jabbour SK, Kim S, Reyhan M, Robinson CG, Thorstad WL, Choi JI, Press R, Simone CB, Camaratta J, Owens S, Torigian DA. Combining natural and artificial intelligence for robust automatic anatomy segmentation: Application in neck and thorax auto-contouring. Med Phys 2022; 49:7118-7149. [PMID: 35833287 PMCID: PMC10087050 DOI: 10.1002/mp.15854] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2022] [Revised: 06/20/2022] [Accepted: 06/30/2022] [Indexed: 01/01/2023] Open
Abstract
BACKGROUND Automatic segmentation of 3D objects in computed tomography (CT) is challenging. Current methods, based mainly on artificial intelligence (AI) and end-to-end deep learning (DL) networks, are weak in garnering high-level anatomic information, which leads to compromised efficiency and robustness. This can be overcome by incorporating natural intelligence (NI) into AI methods via computational models of human anatomic knowledge. PURPOSE We formulate a hybrid intelligence (HI) approach that integrates the complementary strengths of NI and AI for organ segmentation in CT images and illustrate performance in the application of radiation therapy (RT) planning via multisite clinical evaluation. METHODS The system employs five modules: (i) body region recognition, which automatically trims a given image to a precisely defined target body region; (ii) NI-based automatic anatomy recognition object recognition (AAR-R), which performs object recognition in the trimmed image without DL and outputs a localized fuzzy model for each object; (iii) DL-based recognition (DL-R), which refines the coarse recognition results of AAR-R and outputs a stack of 2D bounding boxes (BBs) for each object; (iv) model morphing (MM), which deforms the AAR-R fuzzy model of each object guided by the BBs output by DL-R; and (v) DL-based delineation (DL-D), which employs the object containment information provided by MM to delineate each object. NI from (ii), AI from (i), (iii), and (v), and their combination from (iv) facilitate the HI system. RESULTS The HI system was tested on 26 organs in neck and thorax body regions on CT images obtained prospectively from 464 patients in a study involving four RT centers. Data sets from one separate independent institution involving 125 patients were employed in training/model building for each of the two body regions, whereas 104 and 110 data sets from the 4 RT centers were utilized for testing on neck and thorax, respectively. In the testing data sets, 83% of the images had limitations such as streak artifacts, poor contrast, shape distortion, pathology, or implants. The contours output by the HI system were compared to contours drawn in clinical practice at the four RT centers by utilizing an independently established ground-truth set of contours as reference. Three sets of measures were employed: accuracy via Dice coefficient (DC) and Hausdorff boundary distance (HD), subjective clinical acceptability via a blinded reader study, and efficiency by measuring human time saved in contouring by the HI system. Overall, the HI system achieved a mean DC of 0.78 and 0.87 and a mean HD of 2.22 and 4.53 mm for neck and thorax, respectively. It significantly outperformed clinical contouring in accuracy and saved overall 70% of human time over clinical contouring time, whereas acceptability scores varied significantly from site to site for both auto-contours and clinically drawn contours. CONCLUSIONS The HI system is observed to behave like an expert human in robustness in the contouring task but vastly more efficiently. It seems to use NI help where image information alone will not suffice to decide, first for the correct localization of the object and then for the precise delineation of the boundary.
Collapse
Affiliation(s)
- Jayaram K. Udupa
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Tiange Liu
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- School of Information Science and EngineeringYanshan UniversityQinhuangdaoChina
| | - Chao Jin
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Liming Zhao
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Dewey Odhner
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Yubing Tong
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Vibhu Agrawal
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Gargi Pednekar
- Quantitative Radiology SolutionsPhiladelphiaPennsylvaniaUSA
| | - Sanghita Nag
- Quantitative Radiology SolutionsPhiladelphiaPennsylvaniaUSA
| | - Tarun Kotia
- Quantitative Radiology SolutionsPhiladelphiaPennsylvaniaUSA
| | | | - E. Paul Wileyto
- Department of Biostatistics and EpidemiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Dimitris Mihailidis
- Department of Radiation OncologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - John Nicholas Lukens
- Department of Radiation OncologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Abigail T. Berman
- Department of Radiation OncologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Joann Stambaugh
- Department of Radiation OncologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Tristan Lim
- Department of Radiation OncologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Rupa Chowdary
- Department of MedicineUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Dheeraj Jalluri
- Department of MedicineUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Salma K. Jabbour
- Department of Radiation OncologyRutgers UniversityNew BrunswickNew JerseyUSA
| | - Sung Kim
- Department of Radiation OncologyRutgers UniversityNew BrunswickNew JerseyUSA
| | - Meral Reyhan
- Department of Radiation OncologyRutgers UniversityNew BrunswickNew JerseyUSA
| | | | - Wade L. Thorstad
- Department of Radiation OncologyWashington UniversitySt. LouisMissouriUSA
| | | | | | | | - Joe Camaratta
- Quantitative Radiology SolutionsPhiladelphiaPennsylvaniaUSA
| | - Steve Owens
- Quantitative Radiology SolutionsPhiladelphiaPennsylvaniaUSA
| | - Drew A. Torigian
- Medical Image Processing GroupDepartment of RadiologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| |
Collapse
|
142
|
Yu H, Sharifai N, Jiang K, Wang F, Teodoro G, Farris AB, Kong J. Artificial intelligence based liver portal tract region identification and quantification with transplant biopsy whole-slide images. Comput Biol Med 2022; 150:106089. [PMID: 36137315 DOI: 10.1016/j.compbiomed.2022.106089] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2022] [Revised: 08/11/2022] [Accepted: 09/03/2022] [Indexed: 11/24/2022]
Abstract
Liver fibrosis staging is clinically important for liver disease progression prediction. As the portal tract fibrotic quantity and size in a liver biopsy correlate with the fibrosis stage, an accurate analysis of portal tract regions is clinically critical. Manual annotations of portal tract regions, however, are time-consuming and subject to large inter- and intra-observer variability. To address such a challenge, we develop a Multiple Up-sampling and Spatial Attention guided UNet model (MUSA-UNet) to segment liver portal tract regions in whole-slide images of liver tissue slides. To enhance the segmentation performance, we propose to use depth-wise separable convolution, the spatial attention mechanism, the residual connection, and multiple up-sampling paths in the developed model. This study includes 53 histopathology whole slide images from patients who received liver transplantation. In total, 6,012 patches derived from 30 images are used for our deep learning model training and validation. The remaining 23 whole slide images are utilized for the model testing. The average liver portal tract segmentation performance of the developed MUSA-UNet is 0.94 (Precision), 0.85 (Recall), 0.89 (F1 Score), 0.89 (Accuracy), 0.80 (Jaccard Index), and 0.91 (Fowlkes-Mallows Index), respectively. The clinical Scheuer fibrosis stage presents a strong correlation with the resulting average portal tract fibrotic area (R = 0.681, p<0.001) and portal tract percentage (R = 0.335, p = 0.02) computed from the MUSA-UNet segmentation results. In conclusion, our developed deep learning model MUSA-UNet can accurately segment portal tract regions from whole-slide images of liver tissue biopsies, presenting its promising potential to assist liver disease diagnosis in a computational manner.
Collapse
Affiliation(s)
- Hanyi Yu
- Department of Computer Science, Emory University, Atlanta, 30322, GA, USA.
| | - Nima Sharifai
- Department of Pathology, University of Maryland School of Medicine, Baltimore, 21201, MD, USA.
| | - Kun Jiang
- Department of Pathology, Moffitt Cancer Center, Tampa, 33612, FL, USA.
| | - Fusheng Wang
- Department of Computer Science and Department of Biomedical Informatics, Stony Brook University, Stony Brook, 11794, NY, USA.
| | - George Teodoro
- Department of Computer Science, Federal University of Minas Gerais, Belo Horizonte, 31270, Minas Gerais, Brazil.
| | - Alton B Farris
- Department of Pathology and Laboratory Medicine, Emory University, Atlanta, 30322, GA, USA.
| | - Jun Kong
- Department of Computer Science, Emory University, Atlanta, 30322, GA, USA; Department of Mathematics and Statistics, Georgia State University, Atlanta, 30303, GA, USA; Winship Cancer Institute, Emory University, Atlanta, 30322, GA, USA.
| |
Collapse
|
143
|
X-Ray Lung Image Classification Using a Canny Edge Detector. JOURNAL OF ELECTRICAL AND COMPUTER ENGINEERING 2022. [DOI: 10.1155/2022/3081584] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
The medical imaging technique is used in order to obtain a tissue image of a specific part of the human body without any surgical intervention. The presence of differences in the clinical experiences of a section of doctors or doctors in general can lead to discrepancies in the analysis and understanding of medical images and thus affects the accuracy of the diagnosis for the patient’s condition. The use of a medical imaging system for reliable diagnosis through the use of the computer will lead to high accuracy in diagnosis. For this reason, the need to improve the special performance of systems that perform computer-aided diagnosis used in the medical imaging process has increased special performance of the computer-aided diagnostic systems used in the medical imaging process. The medical image classification technique has the ability to perform a preliminary analysis as well as an understanding of medical images and also can identify the affected parts of the human body, which leads to helping doctors in the process of optimal diagnosis. The process of classifying medical images needs to extract the features of the image so that the classification process is carried out with high accuracy, and one of these features is detecting the edges of the image using a Canny edge detector. This is what the author performed in the research, and the experimental results show the effectiveness and goodness of this method.
Collapse
|
144
|
You S, Reyes M. Influence of contrast and texture based image modifications on the performance and attention shift of U-Net models for brain tissue segmentation. FRONTIERS IN NEUROIMAGING 2022; 1:1012639. [PMID: 37555149 PMCID: PMC10406260 DOI: 10.3389/fnimg.2022.1012639] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/05/2022] [Accepted: 10/12/2022] [Indexed: 08/10/2023]
Abstract
Contrast and texture modifications applied during training or test-time have recently shown promising results to enhance the generalization performance of deep learning segmentation methods in medical image analysis. However, a deeper understanding of this phenomenon has not been investigated. In this study, we investigated this phenomenon using a controlled experimental setting, using datasets from the Human Connectome Project and a large set of simulated MR protocols, in order to mitigate data confounders and investigate possible explanations as to why model performance changes when applying different levels of contrast and texture-based modifications. Our experiments confirm previous findings regarding the improved performance of models subjected to contrast and texture modifications employed during training and/or testing time, but further show the interplay when these operations are combined, as well as the regimes of model improvement/worsening across scanning parameters. Furthermore, our findings demonstrate a spatial attention shift phenomenon of trained models, occurring for different levels of model performance, and varying in relation to the type of applied image modification.
Collapse
Affiliation(s)
- Suhang You
- Medical Image Analysis Group, ARTORG, Graduate School for Cellular and Biomedical Sciences, University of Bern, Bern, Switzerland
| | | |
Collapse
|
145
|
Sheu RK, Pardeshi MS. A Survey on Medical Explainable AI (XAI): Recent Progress, Explainability Approach, Human Interaction and Scoring System. SENSORS (BASEL, SWITZERLAND) 2022; 22:8068. [PMID: 36298417 PMCID: PMC9609212 DOI: 10.3390/s22208068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/23/2022] [Revised: 10/18/2022] [Accepted: 10/18/2022] [Indexed: 06/16/2023]
Abstract
The emerging field of eXplainable AI (XAI) in the medical domain is considered to be of utmost importance. Meanwhile, incorporating explanations in the medical domain with respect to legal and ethical AI is necessary to understand detailed decisions, results, and current status of the patient's conditions. Successively, we will be presenting a detailed survey for the medical XAI with the model enhancements, evaluation methods, significant overview of case studies with open box architecture, medical open datasets, and future improvements. Potential differences in AI and XAI methods are provided with the recent XAI methods stated as (i) local and global methods for preprocessing, (ii) knowledge base and distillation algorithms, and (iii) interpretable machine learning. XAI characteristics details with future healthcare explainability is included prominently, whereas the pre-requisite provides insights for the brainstorming sessions before beginning a medical XAI project. Practical case study determines the recent XAI progress leading to the advance developments within the medical field. Ultimately, this survey proposes critical ideas surrounding a user-in-the-loop approach, with an emphasis on human-machine collaboration, to better produce explainable solutions. The surrounding details of the XAI feedback system for human rating-based evaluation provides intelligible insights into a constructive method to produce human enforced explanation feedback. For a long time, XAI limitations of the ratings, scores and grading are present. Therefore, a novel XAI recommendation system and XAI scoring system are designed and approached from this work. Additionally, this paper encourages the importance of implementing explainable solutions into the high impact medical field.
Collapse
Affiliation(s)
- Ruey-Kai Sheu
- Department of Computer Science, Tunghai University, No. 1727, Section 4, Taiwan Blvd, Xitun District, Taichung 407224, Taiwan
| | - Mayuresh Sunil Pardeshi
- AI Center, Tunghai University, No. 1727, Section 4, Taiwan Blvd, Xitun District, Taichung 407224, Taiwan
| |
Collapse
|
146
|
Ji L, Mao R, Wu J, Ge C, Xiao F, Xu X, Xie L, Gu X. Deep Convolutional Neural Network for Nasopharyngeal Carcinoma Discrimination on MRI by Comparison of Hierarchical and Simple Layered Convolutional Neural Networks. Diagnostics (Basel) 2022; 12:diagnostics12102478. [PMID: 36292167 PMCID: PMC9601165 DOI: 10.3390/diagnostics12102478] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Revised: 10/03/2022] [Accepted: 10/09/2022] [Indexed: 12/24/2022] Open
Abstract
Nasopharyngeal carcinoma (NPC) is one of the most common head and neck cancers. Early diagnosis plays a critical role in the treatment of NPC. To aid diagnosis, deep learning methods can provide interpretable clues for identifying NPC from magnetic resonance images (MRI). To identify the optimal models, we compared the discrimination performance of hierarchical and simple layered convolutional neural networks (CNN). Retrospectively, we collected the MRI images of patients and manually built the tailored NPC image dataset. We examined the performance of the representative CNN models including shallow CNN, ResNet50, ResNet101, and EfficientNet-B7. By fine-tuning, shallow CNN, ResNet50, ResNet101, and EfficientNet-B7 achieved the precision of 72.2%, 94.4%, 92.6%, and 88.4%, displaying the superiority of deep hierarchical neural networks. Among the examined models, ResNet50 with pre-trained weights demonstrated the best classification performance over other types of CNN with accuracy, precision, and an F1-score of 0.93, 0.94, and 0.93, respectively. The fine-tuned ResNet50 achieved the highest prediction performance and can be used as a potential tool for aiding the diagnosis of NPC tumors.
Collapse
Affiliation(s)
- Li Ji
- Department of Otorhinolaryngology, The Second People’s Hospital of Changzhou Affiliated to Nanjing Medical University, Changzhou 213003, China
| | - Rongzhi Mao
- Institute of Bioinformatics and Medical Engineering, School of Electrical and Information Engineering, Jiangsu University of Technology, Changzhou 213001, China
| | - Jian Wu
- Department of Otorhinolaryngology, The Second People’s Hospital of Changzhou Affiliated to Nanjing Medical University, Changzhou 213003, China
| | - Cheng Ge
- Institute of Bioinformatics and Medical Engineering, School of Electrical and Information Engineering, Jiangsu University of Technology, Changzhou 213001, China
| | - Feng Xiao
- Department of Otorhinolaryngology, The Second People’s Hospital of Changzhou Affiliated to Nanjing Medical University, Changzhou 213003, China
| | - Xiaojun Xu
- Institute of Bioinformatics and Medical Engineering, School of Electrical and Information Engineering, Jiangsu University of Technology, Changzhou 213001, China
- Correspondence: (X.X.); (L.X.); (X.G.)
| | - Liangxu Xie
- Institute of Bioinformatics and Medical Engineering, School of Electrical and Information Engineering, Jiangsu University of Technology, Changzhou 213001, China
- Correspondence: (X.X.); (L.X.); (X.G.)
| | - Xiaofeng Gu
- Department of Otorhinolaryngology, The Second People’s Hospital of Changzhou Affiliated to Nanjing Medical University, Changzhou 213003, China
- Correspondence: (X.X.); (L.X.); (X.G.)
| |
Collapse
|
147
|
Zhao H, Zhou Z, Wu F, Xiang D, Zhao H, Zhang W, Li L, Li Z, Huang J, Hu H, Liu C, Wang T, Liu W, Ma J, Yang F, Wang X, Zheng C. Self-supervised learning enables 3D digital subtraction angiography reconstruction from ultra-sparse 2D projection views: A multicenter study. Cell Rep Med 2022; 3:100775. [PMID: 36208630 PMCID: PMC9589028 DOI: 10.1016/j.xcrm.2022.100775] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2022] [Revised: 08/04/2022] [Accepted: 09/17/2022] [Indexed: 11/04/2022]
Abstract
3D digital subtraction angiography (DSA) reconstruction from rotational 2D projection X-ray angiography is an important basis for diagnosis and treatment of intracranial aneurysms (IAs). The gold standard requires approximately 133 different projection views for 3D reconstruction. A method to significantly reduce the radiation dosage while ensuring the reconstruction quality is yet to be developed. We propose a self-supervised learning method to realize 3D-DSA reconstruction using ultra-sparse 2D projections. 202 cases (100 from one hospital for training and testing, 102 from two other hospitals for external validation) suspected to be suffering from IAs were conducted to analyze the reconstructed images. Two radiologists scored the reconstructed images from internal and external datasets using eight projections and identified all 82 lesions with high diagnostic confidence. The radiation dosages are approximately 1/16.7 compared with the gold standard method. Our proposed method can help develop a revolutionary 3D-DSA reconstruction method for use in clinic.
Collapse
Affiliation(s)
- Huangxuan Zhao
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China
| | - Zhenghong Zhou
- School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China
| | - Feihong Wu
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China
| | - Dongqiao Xiang
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China
| | - Hui Zhao
- Department of Interventional Radiology, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Wei Zhang
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China
| | - Lin Li
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China
| | - Zhong Li
- Department of Interventional Radiology, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Jia Huang
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China
| | - Hongyao Hu
- Department of Interventional Radiology, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Chengbo Liu
- Research Laboratory for Biomedical Optics and Molecular Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
| | - Tao Wang
- Department of Respiratory and Critical Care Medicine, University of Chinese Academy of Sciences Shenzhen Hospital, Shenzhen 518107, China
| | - Wenyu Liu
- School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China
| | - Jinqiang Ma
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China,Corresponding author
| | - Fan Yang
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China,Corresponding author
| | - Xinggang Wang
- School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China,Corresponding author
| | - Chuansheng Zheng
- Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China,Hubei Province Key Laboratory of Molecular Imaging, Wuhan 430022, China,Corresponding author
| |
Collapse
|
148
|
Valencia L, Clèrigues A, Valverde S, Salem M, Oliver A, Rovira À, Lladó X. Evaluating the use of synthetic T1-w images in new T2 lesion detection in multiple sclerosis. Front Neurosci 2022; 16:954662. [PMID: 36248650 PMCID: PMC9558286 DOI: 10.3389/fnins.2022.954662] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Accepted: 08/30/2022] [Indexed: 11/23/2022] Open
Abstract
The assessment of disease activity using serial brain MRI scans is one of the most valuable strategies for monitoring treatment response in patients with multiple sclerosis (MS) receiving disease-modifying treatments. Recently, several deep learning approaches have been proposed to improve this analysis, obtaining a good trade-off between sensitivity and specificity, especially when using T1-w and T2-FLAIR images as inputs. However, the need to acquire two different types of images is time-consuming, costly and not always available in clinical practice. In this paper, we investigate an approach to generate synthetic T1-w images from T2-FLAIR images and subsequently analyse the impact of using original and synthetic T1-w images on the performance of a state-of-the-art approach for longitudinal MS lesion detection. We evaluate our approach on a dataset containing 136 images from MS patients, and 73 images with lesion activity (the appearance of new T2 lesions in follow-up scans). To evaluate the synthesis of the images, we analyse the structural similarity index metric and the median absolute error and obtain consistent results. To study the impact of synthetic T1-w images, we evaluate the performance of the new lesion detection approach when using (1) both T2-FLAIR and T1-w original images, (2) only T2-FLAIR images, and (3) both T2-FLAIR and synthetic T1-w images. Sensitivities of 0.75, 0.63, and 0.81, respectively, were obtained at the same false-positive rate (0.14) for all experiments. In addition, we also present the results obtained when using the data from the international MSSEG-2 challenge, showing also an improvement when including synthetic T1-w images. In conclusion, we show that the use of synthetic images can support the lack of data or even be used instead of the original image to homogenize the contrast of the different acquisitions in new T2 lesions detection algorithms.
Collapse
Affiliation(s)
- Liliana Valencia
- Research Institute of Computer Vision and Robotics, University of Girona, Girona, Spain
- *Correspondence: Liliana Valencia
| | - Albert Clèrigues
- Research Institute of Computer Vision and Robotics, University of Girona, Girona, Spain
| | | | - Mostafa Salem
- Research Institute of Computer Vision and Robotics, University of Girona, Girona, Spain
- Department of Computer Science, Faculty of Computers and Information, Assiut University, Asyut, Egypt
| | - Arnau Oliver
- Research Institute of Computer Vision and Robotics, University of Girona, Girona, Spain
| | - Àlex Rovira
- Magnetic Resonance Unit, Department of Radiology, Vall d'Hebron University Hospital, Barcelona, Spain
| | - Xavier Lladó
- Research Institute of Computer Vision and Robotics, University of Girona, Girona, Spain
| |
Collapse
|
149
|
Rouzrokh P, Khosravi B, Johnson QJ, Faghani S, Vera Garcia DV, Erickson BJ, Maradit Kremers H, Taunton MJ, Wyles CC. Applying Deep Learning to Establish a Total Hip Arthroplasty Radiography Registry: A Stepwise Approach. J Bone Joint Surg Am 2022; 104:1649-1658. [PMID: 35866648 PMCID: PMC9617138 DOI: 10.2106/jbjs.21.01229] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Abstract
BACKGROUND Establishing imaging registries for large patient cohorts is challenging because manual labeling is tedious and relying solely on DICOM (digital imaging and communications in medicine) metadata can result in errors. We endeavored to establish an automated hip and pelvic radiography registry of total hip arthroplasty (THA) patients by utilizing deep-learning pipelines. The aims of the study were (1) to utilize these automated pipelines to identify all pelvic and hip radiographs with appropriate annotation of laterality and presence or absence of implants, and (2) to automatically measure acetabular component inclination and version for THA images. METHODS We retrospectively retrieved 846,988 hip and pelvic radiography DICOM files from 20,378 patients who underwent primary or revision THA performed at our institution from 2000 to 2020. Metadata for the files were screened followed by extraction of imaging data. Two deep-learning algorithms (an EfficientNetB3 classifier and a YOLOv5 object detector) were developed to automatically determine the radiographic appearance of all files. Additional deep-learning algorithms were utilized to automatically measure the acetabular angles on anteroposterior pelvic and lateral hip radiographs. Algorithm performance was compared with that of human annotators on a random test sample of 5,000 radiographs. RESULTS Deep-learning algorithms enabled appropriate exclusion of 209,332 DICOM files (24.7%) as misclassified non-hip/pelvic radiographs or having corrupted pixel data. The final registry was automatically curated and annotated in <8 hours and included 168,551 anteroposterior pelvic, 176,890 anteroposterior hip, 174,637 lateral hip, and 117,578 oblique hip radiographs. The algorithms achieved 99.9% accuracy, 99.6% precision, 99.5% recall, and a 99.6% F1 score in determining the radiograph appearance. CONCLUSIONS We developed a highly accurate series of deep-learning algorithms to rapidly curate and annotate THA patient radiographs. This efficient pipeline can be utilized by other institutions or registries to construct radiography databases for patient care, longitudinal surveillance, and large-scale research. The stepwise approach for establishing a radiography registry can further be utilized as a workflow guide for other anatomic areas. LEVEL OF EVIDENCE Diagnostic Level IV . See Instructions for Authors for a complete description of levels of evidence.
Collapse
Affiliation(s)
- Pouria Rouzrokh
- Radiology Informatics Laboratory, Department of Radiology, Mayo Clinic, Rochester, Minnesota
- Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Mayo Clinic, Rochester, Minnesota
| | - Bardia Khosravi
- Radiology Informatics Laboratory, Department of Radiology, Mayo Clinic, Rochester, Minnesota
- Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Mayo Clinic, Rochester, Minnesota
| | - Quinn J Johnson
- Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Mayo Clinic, Rochester, Minnesota
- Mayo Clinic Alix School of Medicine, Mayo Clinic, Rochester, Minnesota
| | - Shahriar Faghani
- Radiology Informatics Laboratory, Department of Radiology, Mayo Clinic, Rochester, Minnesota
| | - Diana V Vera Garcia
- Radiology Informatics Laboratory, Department of Radiology, Mayo Clinic, Rochester, Minnesota
- Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Mayo Clinic, Rochester, Minnesota
| | - Bradley J Erickson
- Radiology Informatics Laboratory, Department of Radiology, Mayo Clinic, Rochester, Minnesota
| | - Hilal Maradit Kremers
- Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Mayo Clinic, Rochester, Minnesota
- Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, Minnesota
| | - Michael J Taunton
- Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Mayo Clinic, Rochester, Minnesota
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, Minnesota
| | - Cody C Wyles
- Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Mayo Clinic, Rochester, Minnesota
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, Minnesota
- Department of Clinical Anatomy, Mayo Clinic, Rochester, Minnesota
| |
Collapse
|
150
|
Deformation Analysis and Research of Building Envelope by Deep Learning Technology under the Reinforcement of the Diaphragm Wall. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:9489445. [PMID: 36156955 PMCID: PMC9492380 DOI: 10.1155/2022/9489445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Revised: 08/17/2022] [Accepted: 08/25/2022] [Indexed: 11/17/2022]
Abstract
The safety analysis of underground buildings is the most crucial problem in the construction industry. This work aims to optimize the safety analysis results of the underground building envelope and comprehensively improve the safety of the underground building. Long short-term memory (LSTM) can make long-term and short-term predictions, thus reducing the model’s prediction error. Applying it to the deformation analysis, data prediction of the underground building envelope can improve the accuracy of the deformation prediction of the envelope. This work deeply discusses deep learning technology and the principle of the LSTM model. Based on the safety analysis concept of the underground building envelope, LSTM underground building envelope deformation’s prediction model is established and comprehensively evaluated. The results show that in the prediction of horizontal displacement of foundation pit pile of diaphragm wall, the mean relative error (MRE) of the prediction results of the designed model range in 10%–18%, and the calculation time ranges 15–36 s. In the settlement displacement prediction, the model’s MRE is within the range of 5%–7%, and the calculation time is within the range of 17–40 s. With the increase of training times, the prediction accuracy of the model increases, and the calculation time becomes relatively stable. Compared with other models, the relative error of prediction results is about 5.4% at the highest and 1.8% at the lowest. This work provides technical support for improving the safety prediction accuracy of the underground building envelope and provides some reference value for the comprehensive development of the underground building industry.
Collapse
|