Fast and Robust Symmetric Image Registration Based on Distances Combining Intensity and Spatial Information

Automated Learning for Deformable Medical Image Registration by Jointly Optimizing Network Architectures and Objective Functions

Deformable image registration plays a critical role in various tasks of medical image analysis. A successful registration algorithm, either derived from conventional energy optimization or deep networks, requires tremendous efforts from computer experts to well design registration energy or to carefully tune network architectures with respect to medical data available for a given registration task/scenario. This paper proposes an automated learning registration algorithm (AutoReg) that cooperatively optimizes both architectures and their corresponding training objectives, enabling non-computer experts to conveniently find off-the-shelf registration algorithms for various registration scenarios. Specifically, we establish a triple-level framework to embrace the searching for both network architectures and objectives with a cooperating optimization. Extensive experiments on multiple volumetric datasets and various registration scenarios demonstrate that AutoReg can automatically learn an optimal deep registration network for given volumes and achieve state-of-the-art performance. The automatically learned network also improves computational efficiency over the mainstream UNet architecture from 0.558 to 0.270 seconds for a volume pair on the same configuration.

Monitoring Muscle Perfusion in Rodents During Short-Term Ischemia Using Power Doppler Ultrasound

OBJECTIVE

The aim of this work was to evaluate the reliability of power Doppler ultrasound (PD-US) measurements made without contrast enhancement to monitor temporal changes in peripheral blood perfusion.

METHODS

On the basis of pre-clinical rodent studies, we found that combinations of spatial registration and clutter filtering techniques applied to PD-US signals reproducibly tracked blood perfusion in skeletal muscle. Perfusion is monitored while modulating hindlimb blood flow. First, in invasive studies, PD-US measurements in deep muscle with laser speckle contrast imaging (LSCI) of superficial tissues made before, during and after short-term arterial clamping were compared. Then, in non-invasive studies, a pressure cuff was employed to generate longer-duration hindlimb ischemia. Here, B-mode imaging was also applied to measure flow-mediated dilation of the femoral artery while, simultaneously, PD-US was used to monitor downstream muscle perfusion to quantify reactive hyperemia. Measurements in adult male and female mice and rats, some with exercise conditioning, were included to explore biological variables.

RESULTS

PD-US methods are validated through comparisons with LSCI measurements. As expected, no significant differences were found between sexes or fitness levels in flow-mediated dilation or reactive hyperemia estimates, although post-ischemic perfusion was enhanced with exercise conditioning, suggesting there could be differences between the hyperemic responses of conduit and resistive vessels.

CONCLUSION

Overall, we found non-contrast PD-US imaging can reliably monitor relative spatiotemporal changes in muscle perfusion. This study supports the development of PD-US methods for monitoring perfusion changes in patients at risk for peripheral artery disease.

Motion estimation based on projective information disentanglement for 3D reconstruction of rotational coronary angiography

The 2D projection space-based motion compensation reconstruction (2D-MCR) is a kind of representative method for 3D reconstruction of rotational coronary angiography owing to its high efficiency. However, due to the lack of accurate motion estimation of the overlapping projection pixels, existing 2D-MCR methods may still have a certain level of under-sampling artifacts or lose accuracy for cases with strong cardiac motion. To overcome this, in this study, we proposed a motion estimation approach based on projective information disentanglement (PID-ME) for 3D reconstruction of rotational coronary angiography. The reconstruction method adopts the framework of 2D-MCR, which is referred to as 2D-PID-MCR. The PID-ME consists of two parts: generation of the reference projection sequence based on the fast simplified distance driven projector (FSDDP) algorithm, motion estimation and correction based on the projective average minimal distance measure (PAMD) model. The FSDDP algorithm generates the reference projection sequence faster and accelerates the whole reconstruction greatly. The PAMD model can disentangle the projection information effectively and estimate the motion of both overlapping and non-overlapping projection pixels accurately. The main contribution of this study is the construction of 2D-PID-MCR to overcome the inherent limitations of the existing 2D-MCR method. Simulated and clinical experiments show that the PID-ME, consisting of FSDDP and PAMD, can estimate the motion of the projection sequence data accurately and efficiently. Our 2D-PID-MCR method outperforms the state-of-the-art approaches in terms of accuracy and real-time performance.

Quantification of the methodological error in kinematic evaluation of the DRUJ using dynamic CT

Distal radio-ulnar joint (DRUJ) motion analysis using dynamic CT is gaining popularity. Following scanning and segmentation, 3D bone models are registered to (4D-)CT target frames. Imaging errors like low signal-to-noise ratio (SNR), limited Z-coverage and motion artefacts influence registration, causing misinterpretation of joint motion. This necessitates quantification of the methodological error. A cadaver arm and dynamic phantom were subjected to multiple 4D-CT scans, while varying tube charge-time product and phantom angular velocity, to evaluate the effects of SNR and motion artefacts on registration accuracy and precision. 4D-CT Z-coverage is limited by the scanner. To quantify the effects of different Z-coverages on registration accuracy and precision, 4D-CT was simulated by acquiring multiple spiral 3D-CT scans of the cadaver arm. Z-coverage was varied by clipping the 3D bone models prior to registration. The radius position relative to the ulna was obtained from the segmentation image. Apparent relative displacement seen in the target images is caused by registration errors. Worst-case translations were 0.45, 0.08 and 1.1 mm for SNR-, Z-coverage- and motion-related errors respectively. Worst-case rotations were 0.41, 0.13 and 6.0 degrees. This study showed that quantification of the methodological error enables composition of accurate and precise DRUJ motion scanning protocols.

INSPIRE: Intensity and spatial information-based deformable image registration

We present INSPIRE, a top-performing general-purpose method for deformable image registration. INSPIRE brings distance measures which combine intensity and spatial information into an elastic B-splines-based transformation model and incorporates an inverse inconsistency penalization supporting symmetric registration performance. We introduce several theoretical and algorithmic solutions which provide high computational efficiency and thereby applicability of the proposed framework in a wide range of real scenarios. We show that INSPIRE delivers highly accurate, as well as stable and robust registration results. We evaluate the method on a 2D dataset created from retinal images, characterized by presence of networks of thin structures. Here INSPIRE exhibits excellent performance, substantially outperforming the widely used reference methods. We also evaluate INSPIRE on the Fundus Image Registration Dataset (FIRE), which consists of 134 pairs of separately acquired retinal images. INSPIRE exhibits excellent performance on the FIRE dataset, substantially outperforming several domain-specific methods. We also evaluate the method on four benchmark datasets of 3D magnetic resonance images of brains, for a total of 2088 pairwise registrations. A comparison with 17 other state-of-the-art methods reveals that INSPIRE provides the best overall performance. Code is available at github.com/MIDA-group/inspire.

Is image-to-image translation the panacea for multimodal image registration? A comparative study

Despite current advancement in the field of biomedical image processing, propelled by the deep learning revolution, multimodal image registration, due to its several challenges, is still often performed manually by specialists. The recent success of image-to-image (I2I) translation in computer vision applications and its growing use in biomedical areas provide a tempting possibility of transforming the multimodal registration problem into a, potentially easier, monomodal one. We conduct an empirical study of the applicability of modern I2I translation methods for the task of rigid registration of multimodal biomedical and medical 2D and 3D images. We compare the performance of four Generative Adversarial Network (GAN)-based I2I translation methods and one contrastive representation learning method, subsequently combined with two representative monomodal registration methods, to judge the effectiveness of modality translation for multimodal image registration. We evaluate these method combinations on four publicly available multimodal (2D and 3D) datasets and compare with the performance of registration achieved by several well-known approaches acting directly on multimodal image data. Our results suggest that, although I2I translation may be helpful when the modalities to register are clearly correlated, registration of modalities which express distinctly different properties of the sample are not well handled by the I2I translation approach. The evaluated representation learning method, which aims to find abstract image-like representations of the information shared between the modalities, manages better, and so does the Mutual Information maximisation approach, acting directly on the original multimodal images. We share our complete experimental setup as open-source (https://github.com/MIDA-group/MultiRegEval), including method implementations, evaluation code, and all datasets, for further reproducing and benchmarking.

Light-Weight Deformable Registration Using Adversarial Learning With Distilling Knowledge

Deformable registration is a crucial step in many medical procedures such as image-guided surgery and radiation therapy. Most recent learning-based methods focus on improving the accuracy by optimizing the non-linear spatial correspondence between the input images. Therefore, these methods are computationally expensive and require modern graphic cards for real-time deployment. In this paper, we introduce a new Light-weight Deformable Registration network that significantly reduces the computational cost while achieving competitive accuracy. In particular, we propose a new adversarial learning with distilling knowledge algorithm that successfully leverages meaningful information from the effective but expensive teacher network to the student network. We design the student network such as it is light-weight and well suitable for deployment on a typical CPU. The extensively experimental results on different public datasets show that our proposed method achieves state-of-the-art accuracy while significantly faster than recent methods. We further show that the use of our adversarial learning algorithm is essential for a time-efficiency deformable registration method. Finally, our source code and trained models are available at https://github.com/aioz-ai/LDR_ALDK.

Infrared and Visible Image Registration Based on Automatic Robust Algorithm

Image registration is the base of subsequent image processing and has been widely utilized in computer vision. Aiming at the differences in the resolution, spectrum, and viewpoint of infrared and visible images, and in order to accurately register infrared and visible images, an automatic robust infrared and visible image registration algorithm, based on a deep convolutional network, was proposed. In order to precisely search and locate the feature points, a deep convolutional network is introduced, which solves the problem that a large number of feature points can still be extracted when the pixels of the infrared image are not clear. Then, in order to achieve accurate feature point matching, a rough-to-fine matching algorithm is designed. The rough matching is obtained by location orientation scale transform Euclidean distance, and then, the fine matching is performed based on the update global optimization, and finally, the image registration is realized. Experimental results show that the proposed algorithm has better robustness and accuracy than several advanced registration algorithms.

Registration of IRT and visible light images in neurosurgery: analysis and comparison of automatic intensity-based registration approaches

PURPOSE

The purpose of this study is to analyze and compare six automatic intensity-based registration methods for intraoperative infrared thermography (IRT) and visible light imaging (VIS/RGB). The practical requirement is to get a good performance of Euclidean distance between manually set landmarks in reference and target images as well as to achieve a high structural similarity index metric (SSIM) and peak signal-to-noise ratio (PSNR) with respect to the reference image.

METHODS

In this study, preprocessing is applied to bring both image types to a similar intensity. Similarity transformation is employed to align roughly IRT and visible light images. Two optimizers and two measures are used in this process. Thereafter, due to locally different displacement of the brain surface through respiration and heartbeat, two non-rigid transformations are applied, and finally, a bicubic interpolation is carried out to compensate for the resulting estimated transformation. Performance was assessed using eleven image datasets. The registration accuracy of the different computational approaches was assessed based on SSIM and PSNR. Additionally, five concise landmarks for each dataset were selected manually in reference and target images and the Euclidean distance between the corresponding landmarks was compared.

RESULTS

The results are showing that the combination of normalized intensity, mutual information measure with one-plus-one evolutionary optimizer in combination with Demon registration results in improved accuracy and performance as compared to all other methods tested here. Furthermore, the obtained results led to [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], and [Formula: see text] registrations for datasets 1, 2, 5, 7, and 8 with respect to the second best result by calculating the mean Euclidean distance of five landmarks.

CONCLUSIONS

We conclude that the mutual information measure with one-plus-one evolutionary optimizer in combination with Demon registration can achieve better accuracy and performance to those other methods mentioned here for automatic registration of IRT and visible light images in neurosurgery.

Comparison of East-Asia and West-Europe cohorts explains disparities in survival outcomes and highlights predictive biomarkers of early gastric cancer aggressiveness

Surgical resection with lymphadenectomy and peri-operative chemotherapy is the universal mainstay for curative treatment of gastric cancer (GC) patients with loco-regional disease. However, GC survival remains asymmetric in West- and East-world regions. We hypothesize this asymmetry derives from differential clinical management. Therefore, we collected chemo-naïve GC patients from Portugal and South-Korea to explore specific immunophenotypic profiles related to disease aggressiveness, and clinicopathological factors potentially explaining associated overall survival (OS) differences. Clinicopathological and survival data were collected from chemo-naïve surgical cohorts from Portugal (West-Europe cohort (WE-C); n=170) and South-Korea (East-Asia cohort (EA-C); n=367), and correlated with immunohistochemical expression profiles of E-cadherin and CD44v6 obtained from consecutive tissue microarrays sections. Survival analysis revealed a subset of 12.4% of WE-C patients, whose tumors concomitantly express E-cadherin_abnormal and CD44v6_very-high, displaying extremely poor OS, even at TNM stages I and II. These WE-C stages I and II patients were particularly aggressive compared to all other, invading deeper into the gastric wall (p=0.032) and more often permeating the vasculature (p=0.018) and nerves (p=0.009). A similar immunophenotypic profile was found in 11.9% of EA-C patients, but unrelated to survival. Stage I and II EA-C patients displaying both biomarkers also permeated more lymphatic vessels (p=0.003), promoting lymph node (LN) metastasis (p=0.019), being diagnosed on average 8-years earlier and submitted to more extensive LN dissection than WE-C. Concomitant E-cadherin_abnormal/CD44v6_very-high expression predicts aggressiveness and poor survival of stage I and II GC submitted to conservative lymphadenectomy.

Satellite-Borne Optical Remote Sensing Image Registration Based on Point Features

Since technologies in image fusion, image splicing, and target recognition have developed rapidly, as the basis of many image applications, the performance of image registration directly affects subsequent work. In this work, for rich features of satellite-borne optical imagery such as panchromatic and multispectral images, the Harris corner algorithm is combined with the scale invariant feature transform (SIFT) operator for feature point extraction. Our rough matching strategy uses the K-D (K-Dimensional) tree combined with the BBF (Best Bin First) method, and the similarity measure is the nearest neighbor/the second-nearest neighbor ratio. Finally, a triangle-area representation (TAR) algorithm is utilized to eliminate false matches in order to ensure registration accuracy. The performance of the proposed algorithm is compared with existing popular algorithms. The experimental results indicate that for visible light and multi-spectral satellite remote sensing images of different sizes and different sources, the proposed algorithm in this work is excellent in accuracy and efficiency.

Towards Automatic Protein Co-Expression Quantification in Immunohistochemical TMA Slides

Immunohistochemical (IHC) analysis of tissue biopsies is currently used for clinical screening of solid cancers to assess protein expression. The large amount of image data produced from these tissue samples requires specialized computational pathology methods to perform integrative analysis. Even though proteins are traditionally studied independently, the study of protein co-expression may offer new insights towards patients' clinical and therapeutic decisions. To explore protein co-expression, we constructed a modular image analysis pipeline to spatially align tissue microarray (TMA) image slides, evaluate alignment quality, define tumor regions, and ultimately quantify protein expression, before and after tumor segmentation. The pipeline was built with open-source tools that can manage gigapixel slides. To evaluate the consensus between pathologist and computer, we characterized a cohort of 142 gastric cancer (GC) cases regarding the extent of E-cadherin and CD44v6 expression. We performed IHC analysis in consecutive TMA slides and compared the automated quantification with the pathologists' manual assessment. Our results show that automated quantification within tumor regions improves agreement with the pathologists' classification. A co-expression map was created to identify the cores co-expressing both proteins. The proposed pipeline provides not only computational tools forwarding current pathology practices to explore co-expression, but also a framework for merging data and transferring information in learning-based approaches to pathology.

An Unordered Image Stitching Method Based on Binary Tree and Estimated Overlapping Area

Aiming at the complex computation and time-consuming problem during unordered image stitching, we present a method based on the binary tree and the estimated overlapping areas to stitch images without order in this paper. For image registration, the overlapping areas between input images are estimated, so that the extraction and matching of feature points are only performed in these areas. For image stitching, we build a model of the binary tree to stitch each two matched images without sorting. Compared to traditional methods, our method significantly reduces the computational time of matching irrelevant image pairs and improves the efficiency of image registration and stitching. Moreover, the stitching model of the binary tree proposed in this paper further reduces the distortion of the panorama. Experimental results show that the number of extracted feature points in the estimated overlapping area is approximately 0.3∼0.6 times of that in the entire image by using the same method, which greatly reduces the computational time of feature extraction and matching. Compared to the exhaustive image matching method, our approach only takes about 1/3 of the time to find all matching images.

Non-Rigid Multi-Modal 3D Medical Image Registration Based on Foveated Modality Independent Neighborhood Descriptor

The non-rigid multi-modal three-dimensional (3D) medical image registration is highly challenging due to the difficulty in the construction of similarity measure and the solution of non-rigid transformation parameters. A novel structural representation based registration method is proposed to address these problems. Firstly, an improved modality independent neighborhood descriptor (MIND) that is based on the foveated nonlocal self-similarity is designed for the effective structural representations of 3D medical images to transform multi-modal image registration into mono-modal one. The sum of absolute differences between structural representations is computed as the similarity measure. Subsequently, the foveated MIND based spatial constraint is introduced into the Markov random field (MRF) optimization to reduce the number of transformation parameters and restrict the calculation of the energy function in the image region involving non-rigid deformation. Finally, the accurate and efficient 3D medical image registration is realized by minimizing the similarity measure based MRF energy function. Extensive experiments on 3D positron emission tomography (PET), computed tomography (CT), T1, T2, and (proton density) PD weighted magnetic resonance (MR) images with synthetic deformation demonstrate that the proposed method has higher computational efficiency and registration accuracy in terms of target registration error (TRE) than the registration methods that are based on the hybrid L-BFGS-B and cat swarm optimization (HLCSO), the sum of squared differences on entropy images, the MIND, and the self-similarity context (SSC) descriptor, except that it provides slightly bigger TRE than the HLCSO for CT-PET image registration. Experiments on real MR and ultrasound images with unknown deformation have also be done to demonstrate the practicality and superiority of the proposed method.