1
|
Yin Y, Tang Z, Weng H. Application of visual transformer in renal image analysis. Biomed Eng Online 2024; 23:27. [PMID: 38439100 PMCID: PMC10913284 DOI: 10.1186/s12938-024-01209-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Accepted: 01/22/2024] [Indexed: 03/06/2024] Open
Abstract
Deep Self-Attention Network (Transformer) is an encoder-decoder architectural model that excels in establishing long-distance dependencies and is first applied in natural language processing. Due to its complementary nature with the inductive bias of convolutional neural network (CNN), Transformer has been gradually applied to medical image processing, including kidney image processing. It has become a hot research topic in recent years. To further explore new ideas and directions in the field of renal image processing, this paper outlines the characteristics of the Transformer network model and summarizes the application of the Transformer-based model in renal image segmentation, classification, detection, electronic medical records, and decision-making systems, and compared with CNN-based renal image processing algorithm, analyzing the advantages and disadvantages of this technique in renal image processing. In addition, this paper gives an outlook on the development trend of Transformer in renal image processing, which provides a valuable reference for a lot of renal image analysis.
Collapse
Affiliation(s)
- Yuwei Yin
- The College of Health Sciences and Engineering, University of Shanghai for Science and Technology, 516 Jungong Highway, Yangpu Area, Shanghai, 200093, China
- The College of Medical Technology, Shanghai University of Medicine & Health Sciences, 279 Zhouzhu Highway, Pudong New Area, Shanghai, 201318, China
| | - Zhixian Tang
- The College of Medical Technology, Shanghai University of Medicine & Health Sciences, 279 Zhouzhu Highway, Pudong New Area, Shanghai, 201318, China.
| | - Huachun Weng
- The College of Health Sciences and Engineering, University of Shanghai for Science and Technology, 516 Jungong Highway, Yangpu Area, Shanghai, 200093, China.
- The College of Medical Technology, Shanghai University of Medicine & Health Sciences, 279 Zhouzhu Highway, Pudong New Area, Shanghai, 201318, China.
| |
Collapse
|
2
|
Younes H, Troccaz J, Voros S. Machine learning and registration for automatic seed localization in 3D US images for prostate brachytherapy. Med Phys 2021; 48:1144-1156. [PMID: 33511658 DOI: 10.1002/mp.14628] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2019] [Revised: 09/26/2020] [Accepted: 11/17/2020] [Indexed: 12/24/2022] Open
Abstract
PURPOSE New radiation therapy protocols, in particular adaptive, focal or boost brachytherapy treatments, require determining precisely the position and orientation of the implanted radioactive seeds from real-time ultrasound (US) images. This is necessary to compare them to the planned one and to adjust automatically the dosimetric plan accordingly for next seeds implantations. The image modality, the small size of the seeds, and the artifacts they produce make it a very challenging problem. The objective of the presented work is to setup and to evaluate a robust and automatic method for seed localization in three-dimensional (3D) US images. METHODS The presented method is based on a prelocalization of the needles through which the seeds are injected in the prostate. This prelocalization allows focusing the search on a region of interest (ROI) around the needle tip. Seeds localization starts by binarizing the ROI and removing false positives using, respectively, a Bayesian classifier and a support vector machine (SVM). This is followed by a registration stage using first an iterative closest point (ICP) for localizing the connected set of seeds (named strand) inserted through a needle, and secondly refining each seed position using sum of squared differences (SSD) as a similarity criterion. ICP registers a geometric model of the strand to the candidate voxels while SSD compares an appearance model of a single seed to a subset of the image. The method was evaluated both for 3D images of an Agar-agar phantom and a dataset of clinical 3D images. It was tested on stranded and on loose seeds. RESULTS Results on phantom and clinical images were compared with a manual localization giving mean errors of 1.09 ± 0.61 mm on phantom image and 1.44 ± 0.45 mm on clinical images. On clinical images, the mean errors of individual seeds orientation was 4.33 ± 8 . 51 ∘ . CONCLUSIONS The proposed algorithm for radioactive seed localization is robust, tested on different US images, accurate, giving small mean error values, and returns the five cylindrical seeds degrees of freedom.
Collapse
Affiliation(s)
- Hatem Younes
- University of Grenoble Alpes, CNRS, TIMC-IMAG, F-38000, Grenoble, France
| | - Jocelyne Troccaz
- University of Grenoble Alpes, CNRS, TIMC-IMAG, F-38000, Grenoble, France.,Grenoble INP, INSERM, F-38000, Grenoble, France
| | - Sandrine Voros
- University of Grenoble Alpes, CNRS, TIMC-IMAG, F-38000, Grenoble, France.,Grenoble INP, INSERM, F-38000, Grenoble, France
| |
Collapse
|
3
|
Zaffino P, Pernelle G, Mastmeyer A, Mehrtash A, Zhang H, Kikinis R, Kapur T, Francesca Spadea M. Fully automatic catheter segmentation in MRI with 3D convolutional neural networks: application to MRI-guided gynecologic brachytherapy. Phys Med Biol 2019; 64:165008. [PMID: 31272095 DOI: 10.1088/1361-6560/ab2f47] [Citation(s) in RCA: 36] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
Abstract
External-beam radiotherapy followed by high dose rate (HDR) brachytherapy is the standard-of-care for treating gynecologic cancers. The enhanced soft-tissue contrast provided by magnetic resonance imaging (MRI) makes it a valuable imaging modality for diagnosing and treating these cancers. However, in contrast to computed tomography (CT) imaging, the appearance of the brachytherapy catheters, through which radiation sources are inserted to reach the cancerous tissue later on, is often variable across images. This paper reports, for the first time, a new deep-learning-based method for fully automatic segmentation of multiple closely spaced brachytherapy catheters in intraoperative MRI. Represented in the data are 50 gynecologic cancer patients treated by MRI-guided HDR brachytherapy. For each patient, a single intraoperative MRI was used. 826 catheters in the images were manually segmented by an expert radiation physicist who is also a trained radiation oncologist. The number of catheters in a patient ranged between 10 and 35. A deep 3D convolutional neural network (CNN) model was developed and trained. In order to make the learning process more robust, the network was trained 5 times, each time using a different combination of shown patients. Finally, each test case was processed by the five networks and the final segmentation was generated by voting on the obtained five candidate segmentations. 4-fold validation was executed and all the patients were segmented. An average distance error of 2.0 ± 3.4 mm was achieved. False positive and false negative catheters were 6.7% and 1.5% respectively. Average Dice score was equal to 0.60 ± 0.17. The algorithm is available for use in the open source software platform 3D Slicer allowing for wide scale testing and research discussion. In conclusion, to the best of our knowledge, fully automatic segmentation of multiple closely spaced catheters from intraoperative MR images was achieved for the first time in gynecological brachytherapy.
Collapse
Affiliation(s)
- Paolo Zaffino
- Department of Experimental and Clinical Medicine, Magna Graecia University, 88100, Catanzaro, Italy. Author to whom any correspondence should be addressed
| | | | | | | | | | | | | | | |
Collapse
|
4
|
Mehrtash A, Ghafoorian M, Pernelle G, Ziaei A, Heslinga FG, Tuncali K, Fedorov A, Kikinis R, Tempany CM, Wells WM, Abolmaesumi P, Kapur T. Automatic Needle Segmentation and Localization in MRI With 3-D Convolutional Neural Networks: Application to MRI-Targeted Prostate Biopsy. IEEE TRANSACTIONS ON MEDICAL IMAGING 2019; 38:1026-1036. [PMID: 30334789 PMCID: PMC6450731 DOI: 10.1109/tmi.2018.2876796] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/05/2023]
Abstract
Image guidance improves tissue sampling during biopsy by allowing the physician to visualize the tip and trajectory of the biopsy needle relative to the target in MRI, CT, ultrasound, or other relevant imagery. This paper reports a system for fast automatic needle tip and trajectory localization and visualization in MRI that has been developed and tested in the context of an active clinical research program in prostate biopsy. To the best of our knowledge, this is the first reported system for this clinical application and also the first reported system that leverages deep neural networks for segmentation and localization of needles in MRI across biomedical applications. Needle tip and trajectory were annotated on 583 T2-weighted intra-procedural MRI scans acquired after needle insertion for 71 patients who underwent transperineal MRI-targeted biopsy procedure at our institution. The images were divided into two independent training-validation and test sets at the patient level. A deep 3-D fully convolutional neural network model was developed, trained, and deployed on these samples. The accuracy of the proposed method, as tested on previously unseen data, was 2.80-mm average in needle tip detection and 0.98° in needle trajectory angle. An observer study was designed in which independent annotations by a second observer, blinded to the original observer, were compared with the output of the proposed method. The resultant error was comparable to the measured inter-observer concordance, reinforcing the clinical acceptability of the proposed method. The proposed system has the potential for deployment in clinical routine.
Collapse
Affiliation(s)
- Alireza Mehrtash
- Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, V6T 1Z4, Canada
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | | | | | - Alireza Ziaei
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | - Friso G. Heslinga
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | - Kemal Tuncali
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | - Andriy Fedorov
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | - Ron Kikinis
- Department of Computer Science at the University of Bremen, Bremen, Germany
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | - Clare M. Tempany
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | - William M. Wells
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| | - Purang Abolmaesumi
- Department of Electrical and Computer Engineering, The University of British Columbia Vancouver, BC, V5T 1Z4, Canada
| | - Tina Kapur
- Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, 02115, USA
| |
Collapse
|
5
|
Zhang G, Sun Q, Jiang S, Yang Z, Ma X, Jiang H. Automatic seed picking for brachytherapy postimplant validation with 3D CT images. Int J Comput Assist Radiol Surg 2017. [PMID: 28643024 DOI: 10.1007/s11548-017-1632-3] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/21/2022]
Abstract
PURPOSE Postimplant validation is an indispensable part in the brachytherapy technique. It provides the necessary feedback to ensure the quality of operation. The ability to pick implanted seed relates directly to the accuracy of validation. To address it, an automatic approach is proposed for picking implanted brachytherapy seeds in 3D CT images. METHODS In order to pick seed configuration (location and orientation) efficiently, the approach starts with the segmentation of seed from CT images using a thresholding filter which based on gray-level histogram. Through the process of filtering and denoising, the touching seed and single seed are classified. The true novelty of this approach is found in the application of the canny edge detection and improved concave points matching algorithm to separate touching seeds. Through the computation of image moments, the seed configuration can be determined efficiently. Finally, two different experiments are designed to verify the performance of the proposed approach: (1) physical phantom with 60 model seeds, and (2) patient data with 16 cases. RESULTS Through assessment of validated results by a medical physicist, the proposed method exhibited promising results. Experiment on phantom demonstrates that the error of seed location and orientation is within ([Formula: see text]) mm and ([Formula: see text])[Formula: see text], respectively. In addition, the most seed location and orientation error is controlled within 0.8 mm and 3.5[Formula: see text] in all cases, respectively. The average process time of seed picking is 8.7 s per 100 seeds. CONCLUSIONS In this paper, an automatic, efficient and robust approach, performed on CT images, is proposed to determine the implanted seed location as well as orientation in a 3D workspace. Through the experiments with phantom and patient data, this approach also successfully exhibits good performance.
Collapse
Affiliation(s)
- Guobin Zhang
- Tianjin Key Laboratory of the Design and Intelligent Control of the Advanced Mechatronical System, Tianjin, 300384, China.,School of Mechanical Engineering, Tianjin University of Technology, Tianjin, 300384, China.,School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Qiyuan Sun
- Tianjin Key Laboratory of the Design and Intelligent Control of the Advanced Mechatronical System, Tianjin, 300384, China.,School of Mechanical Engineering, Tianjin University of Technology, Tianjin, 300384, China
| | - Shan Jiang
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China. .,Centre for Advanced Mechanisms and Robotics, Tianjin University, 135 Yaguan Road, Jinnan District, Tianjin, 300350, China.
| | - Zhiyong Yang
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Xiaodong Ma
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Haisong Jiang
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| |
Collapse
|
6
|
Chen A, Qian P, Wang S, Jiang Y. Large-scale fuzzy multiple-medoid clustering method. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2017. [DOI: 10.3233/jifs-152647] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|