1
|
Khan RF, Lee BD, Lee MS. Transformers in medical image segmentation: a narrative review. Quant Imaging Med Surg 2023; 13:8747-8767. [PMID: 38106306 PMCID: PMC10722011 DOI: 10.21037/qims-23-542] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Accepted: 09/14/2023] [Indexed: 12/19/2023]
Abstract
Background and Objective Transformers, which have been widely recognized as state-of-the-art tools in natural language processing (NLP), have also come to be recognized for their value in computer vision tasks. With this increasing popularity, they have also been extensively researched in the more complex medical imaging domain. The associated developments have resulted in transformers being on par with sought-after convolution neural networks, particularly for medical image segmentation. Methods combining both types of networks have proven to be especially successful in capturing local and global contexts, thereby significantly boosting their performances in various segmentation problems. Motivated by this success, we have attempted to survey the consequential research focused on innovative transformer networks, specifically those designed to cater to medical image segmentation in an efficient manner. Methods Databases like Google Scholar, arxiv, ResearchGate, Microsoft Academic, and Semantic Scholar have been utilized to find recent developments in this field. Specifically, research in the English language from 2021 to 2023 was considered. Key Content and Findings In this survey, we look into the different types of architectures and attention mechanisms that uniquely improve performance and the structures that are in place to handle complex medical data. Through this survey, we summarize the popular and unconventional transformer-based research as seen through different key angles and analyze quantitatively the strategies that have proven more advanced. Conclusions We have also attempted to discern existing gaps and challenges within current research, notably highlighting the deficiency of annotated medical data for precise deep learning model training. Furthermore, potential future directions for enhancing transformers' utility in healthcare are outlined, encompassing strategies such as transfer learning and exploiting foundation models for specialized medical image segmentation.
Collapse
Affiliation(s)
- Rabeea Fatma Khan
- Department of Computer Science, Graduate School, Kyonggi University, Suwon, Republic of Korea
| | - Byoung-Dai Lee
- Department of Computer Science, Graduate School, Kyonggi University, Suwon, Republic of Korea
| | - Mu Sook Lee
- Department of Radiology, Keimyung University Dongsan Hospital, Daegu, Republic of Korea
| |
Collapse
|
2
|
Li L, Tan J, Yu L, Li C, Nan H, Zheng S. LSAM: L2-norm self-attention and latent space feature interaction for automatic 3D multi-modal head and neck tumor segmentation. Phys Med Biol 2023; 68:225004. [PMID: 37852283 DOI: 10.1088/1361-6560/ad04a8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Accepted: 10/18/2023] [Indexed: 10/20/2023]
Abstract
Objective.Head and neck (H&N) cancers are prevalent globally, and early and accurate detection is absolutely crucial for timely and effective treatment. However, the segmentation of H&N tumors is challenging due to the similar density of the tumors and surrounding tissues in CT images. While positron emission computed tomography (PET) images provide information about the metabolic activity of the tissue and can distinguish between lesion regions and normal tissue. But they are limited by their low spatial resolution. To fully leverage the complementary information from PET and CT images, we propose a novel and innovative multi-modal tumor segmentation method specifically designed for H&N tumor segmentation.Approach.The proposed novel and innovative multi-modal tumor segmentation network (LSAM) consists of two key learning modules, namely L2-Norm self-attention and latent space feature interaction, which exploit the high sensitivity of PET images and the anatomical information of CT images. These two advanced modules contribute to a powerful 3D segmentation network based on a U-shaped structure. The well-designed segmentation method can integrate complementary features from different modalities at multiple scales, thereby improving the feature interaction between modalities.Main results.We evaluated the proposed method on the public HECKTOR PET-CT dataset, and the experimental results demonstrate that the proposed method convincingly outperforms existing H&N tumor segmentation methods in terms of key evaluation metrics, including DSC (0.8457), Jaccard (0.7756), RVD (0.0938), and HD95 (11.75).Significance.The innovative Self-Attention mechanism based on L2-Norm offers scalability and is effective in reducing the impact of outliers on the performance of the model. And the novel method for multi-scale feature interaction based on Latent Space utilizes the learning process in the encoder phase to achieve the best complementary effects among different modalities.
Collapse
Affiliation(s)
- Laquan Li
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
- School of Science, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Jiaxin Tan
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Lei Yu
- Emergency Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, People's Republic of China
| | - Chunwen Li
- Emergency Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, People's Republic of China
| | - Hai Nan
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing, People's Republic of China
| | - Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| |
Collapse
|
3
|
Liu Z, Lv Q, Yang Z, Li Y, Lee CH, Shen L. Recent progress in transformer-based medical image analysis. Comput Biol Med 2023; 164:107268. [PMID: 37494821 DOI: 10.1016/j.compbiomed.2023.107268] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 05/30/2023] [Accepted: 07/16/2023] [Indexed: 07/28/2023]
Abstract
The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.
Collapse
Affiliation(s)
- Zhaoshan Liu
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| | - Qiujie Lv
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore; School of Intelligent Systems Engineering, Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, 518107, China.
| | - Ziduo Yang
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore; School of Intelligent Systems Engineering, Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, 518107, China.
| | - Yifan Li
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| | - Chau Hung Lee
- Department of Radiology, Tan Tock Seng Hospital, 11 Jalan Tan Tock Seng, Singapore, 308433, Singapore.
| | - Lei Shen
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| |
Collapse
|
4
|
Xiao H, Li L, Liu Q, Zhu X, Zhang Q. Transformers in medical image segmentation: A review. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104791] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/09/2023]
|
5
|
Li Y, Sun X, Wang S, Li X, Qin Y, Pan J, Chen P. MDST: multi-domain sparse-view CT reconstruction based on convolution and swin transformer. Phys Med Biol 2023; 68:095019. [PMID: 36889004 DOI: 10.1088/1361-6560/acc2ab] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Accepted: 03/08/2023] [Indexed: 03/10/2023]
Abstract
Objective.Sparse-view computed tomography (SVCT), which can reduce the radiation doses administered to patients and hasten data acquisition, has become an area of particular interest to researchers. Most existing deep learning-based image reconstruction methods are based on convolutional neural networks (CNNs). Due to the locality of convolution and continuous sampling operations, existing approaches cannot fully model global context feature dependencies, which makes the CNN-based approaches less efficient in modeling the computed tomography (CT) images with various structural information.Approach.To overcome the above challenges, this paper develops a novel multi-domain optimization network based on convolution and swin transformer (MDST). MDST uses swin transformer block as the main building block in both projection (residual) domain and image (residual) domain sub-networks, which models global and local features of the projections and reconstructed images. MDST consists of two modules for initial reconstruction and residual-assisted reconstruction, respectively. The sparse sinogram is first expanded in the initial reconstruction module with a projection domain sub-network. Then, the sparse-view artifacts are effectively suppressed by an image domain sub-network. Finally, the residual assisted reconstruction module to correct the inconsistency of the initial reconstruction, further preserving image details.Main results. Extensive experiments on CT lymph node datasets and real walnut datasets show that MDST can effectively alleviate the loss of fine details caused by information attenuation and improve the reconstruction quality of medical images.Significance.MDST network is robust and can effectively reconstruct images with different noise level projections. Different from the current prevalent CNN-based networks, MDST uses transformer as the main backbone, which proves the potential of transformer in SVCT reconstruction.
Collapse
Affiliation(s)
- Yu Li
- Department of Information and Communication Engineering, North University of China, Taiyuan, People's Republic of China
- The State Key Lab for Electronic Testing Technology, North University of China, People's Republic of China
| | - XueQin Sun
- Department of Information and Communication Engineering, North University of China, Taiyuan, People's Republic of China
- The State Key Lab for Electronic Testing Technology, North University of China, People's Republic of China
| | - SuKai Wang
- Department of Information and Communication Engineering, North University of China, Taiyuan, People's Republic of China
- The State Key Lab for Electronic Testing Technology, North University of China, People's Republic of China
| | - XuRu Li
- Department of Information and Communication Engineering, North University of China, Taiyuan, People's Republic of China
- The State Key Lab for Electronic Testing Technology, North University of China, People's Republic of China
| | - YingWei Qin
- Department of Information and Communication Engineering, North University of China, Taiyuan, People's Republic of China
- The State Key Lab for Electronic Testing Technology, North University of China, People's Republic of China
| | - JinXiao Pan
- Department of Information and Communication Engineering, North University of China, Taiyuan, People's Republic of China
- The State Key Lab for Electronic Testing Technology, North University of China, People's Republic of China
| | - Ping Chen
- Department of Information and Communication Engineering, North University of China, Taiyuan, People's Republic of China
- The State Key Lab for Electronic Testing Technology, North University of China, People's Republic of China
| |
Collapse
|
6
|
Li J, Chen J, Tang Y, Wang C, Landman BA, Zhou SK. Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives. Med Image Anal 2023; 85:102762. [PMID: 36738650 PMCID: PMC10010286 DOI: 10.1016/j.media.2023.102762] [Citation(s) in RCA: 27] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 01/18/2023] [Accepted: 01/27/2023] [Indexed: 02/01/2023]
Abstract
Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.
Collapse
Affiliation(s)
- Jun Li
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Junyu Chen
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD, USA
| | - Yucheng Tang
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - Ce Wang
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Bennett A Landman
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - S Kevin Zhou
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China; School of Biomedical Engineering & Suzhou Institute for Advanced Research, Center for Medical Imaging, Robotics, and Analytic Computing & Learning (MIRACLE), University of Science and Technology of China, Suzhou 215123, China.
| |
Collapse
|
7
|
Zheng S, Tan J, Jiang C, Li L. Automated multi-modal Transformer network (AMTNet) for 3D medical images segmentation. Phys Med Biol 2023; 68. [PMID: 36595252 DOI: 10.1088/1361-6560/aca74c] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2022] [Accepted: 11/29/2022] [Indexed: 12/05/2022]
Abstract
Objective.Over the past years, convolutional neural networks based methods have dominated the field of medical image segmentation. But the main drawback of these methods is that they have difficulty representing long-range dependencies. Recently, the Transformer has demonstrated super performance in computer vision and has also been successfully applied to medical image segmentation because of the self-attention mechanism and long-range dependencies encoding on images. To the best of our knowledge, only a few works focus on cross-modalities of image segmentation using the Transformer. Hence, the main objective of this study was to design, propose and validate a deep learning method to extend the application of Transformer to multi-modality medical image segmentation.Approach.This paper proposes a novel automated multi-modal Transformer network termed AMTNet for 3D medical image segmentation. Especially, the network is a well-modeled U-shaped network architecture where many effective and significant changes have been made in the feature encoding, fusion, and decoding parts. The encoding part comprises 3D embedding, 3D multi-modal Transformer, and 3D Co-learn down-sampling blocks. Symmetrically, the 3D Transformer block, upsampling block, and 3D-expanding blocks are included in the decoding part. In addition, a Transformer-based adaptive channel interleaved Transformer feature fusion module is designed to fully fuse features of different modalities.Main results.We provide a comprehensive experimental analysis of the Prostate and BraTS2021 datasets. The results show that our method achieves an average DSC of 0.907 and 0.851 (0.734 for ET, 0.895 for TC, and 0.924 for WT) on these two datasets, respectively. These values show that AMTNet yielded significant improvements over the state-of-the-art segmentation networks.Significance.The proposed 3D segmentation network exploits complementary features of different modalities during the feature extraction process at multiple scales to increase the 3D feature representations and improve the segmentation efficiency. This powerful network enriches the research of the Transformer to multi-modal medical image segmentation.
Collapse
Affiliation(s)
- Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China.,Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| | - Jiaxin Tan
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| | - Chuangbo Jiang
- School of Science, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| | - Laquan Li
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China.,School of Science, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| |
Collapse
|
8
|
Rao VM, Wan Z, Arabshahi S, Ma DJ, Lee PY, Tian Y, Zhang X, Laine AF, Guo J. Improving across-dataset brain tissue segmentation for MRI imaging using transformer. FRONTIERS IN NEUROIMAGING 2022; 1:1023481. [PMID: 37555170 PMCID: PMC10406272 DOI: 10.3389/fnimg.2022.1023481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/19/2022] [Accepted: 10/24/2022] [Indexed: 08/10/2023]
Abstract
Brain tissue segmentation has demonstrated great utility in quantifying MRI data by serving as a precursor to further post-processing analysis. However, manual segmentation is highly labor-intensive, and automated approaches, including convolutional neural networks (CNNs), have struggled to generalize well due to properties inherent to MRI acquisition, leaving a great need for an effective segmentation tool. This study introduces a novel CNN-Transformer hybrid architecture designed to improve brain tissue segmentation by taking advantage of the increased performance and generality conferred by Transformers for 3D medical image segmentation tasks. We first demonstrate the superior performance of our model on various T1w MRI datasets. Then, we rigorously validate our model's generality applied across four multi-site T1w MRI datasets, covering different vendors, field strengths, scan parameters, and neuropsychiatric conditions. Finally, we highlight the reliability of our model on test-retest scans taken in different time points. In all situations, our model achieved the greatest generality and reliability compared to the benchmarks. As such, our method is inherently robust and can serve as a valuable tool for brain related T1w MRI studies. The code for the TABS network is available at: https://github.com/raovish6/TABS.
Collapse
Affiliation(s)
- Vishwanatha M. Rao
- Department of Biomedical Engineering, Columbia University, New York, NY, United States
| | - Zihan Wan
- Department of Applied Mathematics, Columbia University, New York, NY, United States
| | - Soroush Arabshahi
- Department of Biomedical Engineering, Columbia University, New York, NY, United States
| | - David J. Ma
- Department of Biomedical Engineering, Columbia University, New York, NY, United States
| | - Pin-Yu Lee
- Department of Biomedical Engineering, Columbia University, New York, NY, United States
| | - Ye Tian
- Department of Biomedical Engineering, Columbia University, New York, NY, United States
| | - Xuzhe Zhang
- Department of Biomedical Engineering, Columbia University, New York, NY, United States
| | - Andrew F. Laine
- Department of Biomedical Engineering, Columbia University, New York, NY, United States
| | - Jia Guo
- Department of Psychiatry, Columbia University, New York, NY, United States
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, United States
| |
Collapse
|
9
|
Xing W, Zhu Z, Hou D, Yue Y, Dai F, Li Y, Tong L, Song Y, Ta D. CM-SegNet: A deep learning-based automatic segmentation approach for medical images by combining convolution and multilayer perceptron. Comput Biol Med 2022; 147:105797. [PMID: 35780603 DOI: 10.1016/j.compbiomed.2022.105797] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2022] [Revised: 06/21/2022] [Accepted: 06/26/2022] [Indexed: 11/30/2022]
Abstract
Accurate segmentation of lesions in medical images is of great significance for clinical diagnosis and evaluation. The low contrast between lesions and surrounding tissues increases the difficulty of automatic segmentation, while the efficiency of manual segmentation is low. In order to increase the generalization performance of segmentation model, we proposed a deep learning-based automatic segmentation model called CM-SegNet for segmenting medical images of different modalities. It was designed according to the multiscale input and encoding-decoding thoughts, and composed of multilayer perceptron and convolution modules. This model achieved communication of different channels and different spatial locations of each patch, and considers the edge related feature information between adjacent patches. Thus, it could fully extract global and local image information for the segmentation task. Meanwhile, this model met the effective segmentation of different structural lesion regions in different slices of three-dimensional medical images. In this experiment, the proposed CM-SegNet was trained, validated, and tested using six medical image datasets of different modalities and 5-fold cross validation method. The results showed that the CM-SegNet model had better segmentation performance and shorter training time for different medical images than the previous methods, suggesting it is faster and more accurate in automatic segmentation and has great potential application in clinic.
Collapse
Affiliation(s)
- Wenyu Xing
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, 200438, China; Human Phenome Institute, Fudan University, Shanghai, 200438, China
| | - Zhibin Zhu
- School of Physics and Electromechanical Engineering, Hexi University, Zhangye, 734000, China
| | - Dongni Hou
- Department of Pulmonary and Critical Care Medicine, Zhongshan Hospital, Fudan University, Shanghai, 200032, China; Shanghai Key Laboratory of Lung Inflammation and Injury, Shanghai, 200032, China.
| | - Yaoting Yue
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, 200438, China; Human Phenome Institute, Fudan University, Shanghai, 200438, China
| | - Fei Dai
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, 200438, China
| | - Yifang Li
- Academy for Engineering and Technology, Fudan University, Shanghai, 200433, China
| | - Lin Tong
- Department of Pulmonary and Critical Care Medicine, Zhongshan Hospital, Fudan University, Shanghai, 200032, China; Shanghai Key Laboratory of Lung Inflammation and Injury, Shanghai, 200032, China
| | - Yuanlin Song
- Department of Pulmonary and Critical Care Medicine, Zhongshan Hospital, Fudan University, Shanghai, 200032, China; Shanghai Key Laboratory of Lung Inflammation and Injury, Shanghai, 200032, China
| | - Dean Ta
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, 200438, China; Department of Rehabilitation Medicine, Huashan Hospital, Fudan University, Shanghai, 200040, China.
| |
Collapse
|