1
|
Nie Q, Zhang X, Hu Y, Gong M, Liu J. Medical image registration and its application in retinal images: a review. Vis Comput Ind Biomed Art 2024; 7:21. [PMID: 39167337 PMCID: PMC11339199 DOI: 10.1186/s42492-024-00173-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2024] [Accepted: 07/31/2024] [Indexed: 08/23/2024] Open
Abstract
Medical image registration is vital for disease diagnosis and treatment with its ability to merge diverse information of images, which may be captured under different times, angles, or modalities. Although several surveys have reviewed the development of medical image registration, they have not systematically summarized the existing medical image registration methods. To this end, a comprehensive review of these methods is provided from traditional and deep-learning-based perspectives, aiming to help audiences quickly understand the development of medical image registration. In particular, we review recent advances in retinal image registration, which has not attracted much attention. In addition, current challenges in retinal image registration are discussed and insights and prospects for future research provided.
Collapse
Affiliation(s)
- Qiushi Nie
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, 518055, China
| | - Xiaoqing Zhang
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, 518055, China
- Center for High Performance Computing and Shenzhen Key Laboratory of Intelligent Bioinformatics, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Yan Hu
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, 518055, China
| | - Mingdao Gong
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, 518055, China
| | - Jiang Liu
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, 518055, China.
- Singapore Eye Research Institute, Singapore, 169856, Singapore.
- State Key Laboratory of Ophthalmology, Optometry and Visual Science, Eye Hospital, Wenzhou Medical University, Wenzhou, 325027, China.
| |
Collapse
|
2
|
Chen Z, Zou B, Kui X, Shi Y, Lv D, Chen L. Points of interest linear attention network for real-time non-rigid liver volume to surface registration. Med Phys 2024; 51:5351-5360. [PMID: 38758744 DOI: 10.1002/mp.17108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2023] [Revised: 04/04/2024] [Accepted: 04/14/2024] [Indexed: 05/19/2024] Open
Abstract
BACKGROUND In laparoscopic liver surgery, accurately predicting the displacement of key intrahepatic anatomical structures is crucial for informing the doctor's intraoperative decision-making. However, due to the constrained surgical perspective, only a partial surface of the liver is typically visible. Consequently, the utilization of non-rigid volume to surface registration methods becomes essential. But traditional registration methods lack the necessary accuracy and cannot meet real-time requirements. PURPOSE To achieve high-precision liver registration with only partial surface information and estimate the displacement of internal liver tissues in real-time. METHODS We propose a novel neural network architecture tailored for real-time non-rigid liver volume to surface registration. The network utilizes a voxel-based method, integrating sparse convolution with the newly proposed points of interest (POI) linear attention module. POI linear attention module specifically calculates attention on the previously extracted POI. Additionally, we identified the most suitable normalization method RMSINorm. RESULTS We evaluated our proposed network and other networks on a dataset generated from real liver models and two real datasets. Our method achieves an average error of 4.23 mm and a mean frame rate of 65.4 fps in the generation dataset. It also achieves an average error of 8.29 mm in the human breathing motion dataset. CONCLUSIONS Our network outperforms CNN-based networks and other attention networks in terms of accuracy and inference speed.
Collapse
Affiliation(s)
- Zeming Chen
- School of Computer Science and Engineering, Central South University, Changsha, Hunan, China
- Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, Hunan, China
| | - Beiji Zou
- School of Computer Science and Engineering, Central South University, Changsha, Hunan, China
- Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, Hunan, China
| | - Xiaoyan Kui
- School of Computer Science and Engineering, Central South University, Changsha, Hunan, China
- Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, Hunan, China
| | - Yangyang Shi
- School of Computer Science and Engineering, Central South University, Changsha, Hunan, China
- Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, Hunan, China
| | - Ding Lv
- School of Computer Science and Engineering, Central South University, Changsha, Hunan, China
- Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, Hunan, China
| | - Liming Chen
- Ecole Centrale de Lyon, University of Lyon, Lyon, France
| |
Collapse
|
3
|
Rivas-Villar D, Hervella ÁS, Rouco J, Novo J. ConKeD: multiview contrastive descriptor learning for keypoint-based retinal image registration. Med Biol Eng Comput 2024:10.1007/s11517-024-03160-6. [PMID: 38969811 DOI: 10.1007/s11517-024-03160-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Accepted: 06/25/2024] [Indexed: 07/07/2024]
Abstract
Retinal image registration is of utmost importance due to its wide applications in medical practice. In this context, we propose ConKeD, a novel deep learning approach to learn descriptors for retinal image registration. In contrast to current registration methods, our approach employs a novel multi-positive multi-negative contrastive learning strategy that enables the utilization of additional information from the available training samples. This makes it possible to learn high-quality descriptors from limited training data. To train and evaluate ConKeD, we combine these descriptors with domain-specific keypoints, particularly blood vessel bifurcations and crossovers, that are detected using a deep neural network. Our experimental results demonstrate the benefits of the novel multi-positive multi-negative strategy, as it outperforms the widely used triplet loss technique (single-positive and single-negative) as well as the single-positive multi-negative alternative. Additionally, the combination of ConKeD with the domain-specific keypoints produces comparable results to the state-of-the-art methods for retinal image registration, while offering important advantages such as avoiding pre-processing, utilizing fewer training samples, and requiring fewer detected keypoints, among others. Therefore, ConKeD shows a promising potential towards facilitating the development and application of deep learning-based methods for retinal image registration.
Collapse
Affiliation(s)
- David Rivas-Villar
- Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, 15006, A Coruña, Spain.
- Departamento de Ciencias de la Computación y Tecnologías de la Información, Universidade da Coruña, A Coruña, 15071, A Coruña, Spain.
| | - Álvaro S Hervella
- Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, 15006, A Coruña, Spain
- Departamento de Ciencias de la Computación y Tecnologías de la Información, Universidade da Coruña, A Coruña, 15071, A Coruña, Spain
| | - José Rouco
- Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, 15006, A Coruña, Spain
- Departamento de Ciencias de la Computación y Tecnologías de la Información, Universidade da Coruña, A Coruña, 15071, A Coruña, Spain
| | - Jorge Novo
- Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, 15006, A Coruña, Spain
- Departamento de Ciencias de la Computación y Tecnologías de la Información, Universidade da Coruña, A Coruña, 15071, A Coruña, Spain
| |
Collapse
|
4
|
Wang P, Yan Y, Qian L, Suo S, Xu J, Guo Y, Wang Y. Context-driven pyramid registration network for estimating large topology-preserved deformation. Neurocomputing 2023. [DOI: 10.1016/j.neucom.2022.11.088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
5
|
Xu J, Yang K, Chen Y, Dai L, Zhang D, Shuai P, Shi R, Yang Z. Reliable and stable fundus image registration based on brain-inspired spatially-varying adaptive pyramid context aggregation network. Front Neurosci 2023; 16:1117134. [PMID: 36726854 PMCID: PMC9884961 DOI: 10.3389/fnins.2022.1117134] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 12/28/2022] [Indexed: 01/18/2023] Open
Abstract
The task of fundus image registration aims to find matching keypoints between an image pair. Traditional methods detect the keypoint by hand-designed features, which fail to cope with complex application scenarios. Due to the strong feature learning ability of deep neural network, current image registration methods based on deep learning directly learn to align the geometric transformation between the reference image and test image in an end-to-end manner. Another mainstream of this task aims to learn the displacement vector field between the image pair. In this way, the image registration has achieved significant advances. However, due to the complicated vascular morphology of retinal image, such as texture and shape, current widely used image registration methods based on deep learning fail to achieve reliable and stable keypoint detection and registration results. To this end, in this paper, we aim to bridge this gap. Concretely, since the vessel crossing and branching points can reliably and stably characterize the key components of fundus image, we propose to learn to detect and match all the crossing and branching points of the input images based on a single deep neural network. Moreover, in order to accurately locate the keypoints and learn discriminative feature embedding, a brain-inspired spatially-varying adaptive pyramid context aggregation network is proposed to incorporate the contextual cues under the supervision of structured triplet ranking loss. Experimental results show that the proposed method achieves more accurate registration results with significant speed advantage.
Collapse
Affiliation(s)
- Jie Xu
- Beijing Institute of Ophthalmology, Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University, Beijing Key Laboratory of Ophthalmology and Visual Sciences, Beijing, China
| | - Kang Yang
- Beijing Zhizhen Internet Technology Co. Ltd.,Beijing, China
| | - Youxin Chen
- Department of Ophthalmology, Peking Union Medical College Hospital, Beijing, China
| | - Liming Dai
- Beijing Zhizhen Internet Technology Co. Ltd.,Beijing, China
| | - Dongdong Zhang
- Beijing Zhizhen Internet Technology Co. Ltd.,Beijing, China
| | - Ping Shuai
- Department of Health Management and Physical Examination, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu, China
- School of Medicine, University of Electronic Science and Technology of China, Chengdu, China
| | - Rongjie Shi
- Beijing Zhizhen Internet Technology Co. Ltd.,Beijing, China
| | - Zhanbo Yang
- Beijing Zhizhen Internet Technology Co. Ltd.,Beijing, China
| |
Collapse
|
6
|
Abbasi S, Tavakoli M, Boveiri HR, Mosleh Shirazi MA, Khayami R, Khorasani H, Javidan R, Mehdizadeh A. Medical image registration using unsupervised deep neural network: A scoping literature review. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103444] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
|
7
|
Rivas-Villar D, Hervella ÁS, Rouco J, Novo J. Color fundus image registration using a learning-based domain-specific landmark detection methodology. Comput Biol Med 2022; 140:105101. [PMID: 34875412 DOI: 10.1016/j.compbiomed.2021.105101] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Revised: 11/29/2021] [Accepted: 11/29/2021] [Indexed: 11/17/2022]
Abstract
Medical imaging, and particularly retinal imaging, allows to accurately diagnose many eye pathologies as well as some systemic diseases such as hypertension or diabetes. Registering these images is crucial to correctly compare key structures, not only within patients, but also to contrast data with a model or among a population. Currently, this field is dominated by complex classical methods because the novel deep learning methods cannot compete yet in terms of results and commonly used methods are difficult to adapt to the retinal domain. In this work, we propose a novel method to register color fundus images based on previous works which employed classical approaches to detect domain-specific landmarks. Instead, we propose to use deep learning methods for the detection of these highly-specific domain-related landmarks. Our method uses a neural network to detect the bifurcations and crossovers of the retinal blood vessels, whose arrangement and location are unique to each eye and person. This proposal is the first deep learning feature-based registration method in fundus imaging. These keypoints are matched using a method based on RANSAC (Random Sample Consensus) without the requirement to calculate complex descriptors. Our method was tested using the public FIRE dataset, although the landmark detection network was trained using the DRIVE dataset. Our method provides accurate results, a registration score of 0.657 for the whole FIRE dataset (0.908 for category S, 0.293 for category P and 0.660 for category A). Therefore, our proposal can compete with complex classical methods and beat the deep learning methods in the state of the art.
Collapse
Affiliation(s)
- David Rivas-Villar
- Centro de investigacion CITIC, Universidade da Coruña, 15 071, A Coruña, Spain; Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, 15 006, A Coruña, Spain.
| | - Álvaro S Hervella
- Centro de investigacion CITIC, Universidade da Coruña, 15 071, A Coruña, Spain; Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, 15 006, A Coruña, Spain.
| | - José Rouco
- Centro de investigacion CITIC, Universidade da Coruña, 15 071, A Coruña, Spain; Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, 15 006, A Coruña, Spain.
| | - Jorge Novo
- Centro de investigacion CITIC, Universidade da Coruña, 15 071, A Coruña, Spain; Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña, 15 006, A Coruña, Spain.
| |
Collapse
|
8
|
Dan T, Hu Y, Han C, Fan Z, Huang Z, Zhang B, Tao G, Liu B, Yu H, Cai H. Fusion of multi-source retinal fundus images via automatic registration for clinical diagnosis. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.05.091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
9
|
Li T, Bo W, Hu C, Kang H, Liu H, Wang K, Fu H. Applications of deep learning in fundus images: A review. Med Image Anal 2021; 69:101971. [PMID: 33524824 DOI: 10.1016/j.media.2021.101971] [Citation(s) in RCA: 81] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2020] [Accepted: 01/12/2021] [Indexed: 02/06/2023]
Abstract
The use of fundus images for the early screening of eye diseases is of great clinical importance. Due to its powerful performance, deep learning is becoming more and more popular in related applications, such as lesion segmentation, biomarkers segmentation, disease diagnosis and image synthesis. Therefore, it is very necessary to summarize the recent developments in deep learning for fundus images with a review paper. In this review, we introduce 143 application papers with a carefully designed hierarchy. Moreover, 33 publicly available datasets are presented. Summaries and analyses are provided for each task. Finally, limitations common to all tasks are revealed and possible solutions are given. We will also release and regularly update the state-of-the-art results and newly-released datasets at https://github.com/nkicsl/Fundus_Review to adapt to the rapid development of this field.
Collapse
Affiliation(s)
- Tao Li
- College of Computer Science, Nankai University, Tianjin 300350, China
| | - Wang Bo
- College of Computer Science, Nankai University, Tianjin 300350, China
| | - Chunyu Hu
- College of Computer Science, Nankai University, Tianjin 300350, China
| | - Hong Kang
- College of Computer Science, Nankai University, Tianjin 300350, China
| | - Hanruo Liu
- Beijing Tongren Hospital, Capital Medical University, Address, Beijing 100730 China
| | - Kai Wang
- College of Computer Science, Nankai University, Tianjin 300350, China.
| | - Huazhu Fu
- Inception Institute of Artificial Intelligence (IIAI), Abu Dhabi, UAE
| |
Collapse
|