MIND Demons: Symmetric Diffeomorphic Deformable Registration of MR and CT for Image-Guided Spine Surgery

Development of a software system for surgical robots based on multimodal image fusion: study protocol

Background

Surgical robots are gaining increasing popularity because of their capability to improve the precision of pedicle screw placement. However, current surgical robots rely on unimodal computed tomography (CT) images as baseline images, limiting their visualization to vertebral bone structures and excluding soft tissue structures such as intervertebral discs and nerves. This inherent limitation significantly restricts the applicability of surgical robots. To address this issue and further enhance the safety and accuracy of robot-assisted pedicle screw placement, this study will develop a software system for surgical robots based on multimodal image fusion. Such a system can extend the application range of surgical robots, such as surgical channel establishment, nerve decompression, and other related operations.

Methods

Initially, imaging data of the patients included in the study are collected. Professional workstations are employed to establish, train, validate, and optimize algorithms for vertebral bone segmentation in CT and magnetic resonance (MR) images, intervertebral disc segmentation in MR images, nerve segmentation in MR images, and registration fusion of CT and MR images. Subsequently, a spine application model containing independent modules for vertebrae, intervertebral discs, and nerves is constructed, and a software system for surgical robots based on multimodal image fusion is designed. Finally, the software system is clinically validated.

Discussion

We will develop a software system based on multimodal image fusion for surgical robots, which can be applied to surgical access establishment, nerve decompression, and other operations not only for robot-assisted nail placement. The development of this software system is important. First, it can improve the accuracy of pedicle screw placement, percutaneous vertebroplasty, percutaneous kyphoplasty, and other surgeries. Second, it can reduce the number of fluoroscopies, shorten the operation time, and reduce surgical complications. In addition, it would be helpful to expand the application range of surgical robots by providing key imaging data for surgical robots to realize surgical channel establishment, nerve decompression, and other operations.

DiffeoRaptor: diffeomorphic inter-modal image registration using RaPTOR

PURPOSE

Diffeomorphic image registration is essential in many medical imaging applications. Several registration algorithms of such type have been proposed, but primarily for intra-contrast alignment. Currently, efficient inter-modal/contrast diffeomorphic registration, which is vital in numerous applications, remains a challenging task.

METHODS

We proposed a novel inter-modal/contrast registration algorithm that leverages Robust PaTch-based cOrrelation Ratio metric to allow inter-modal/contrast image alignment and bandlimited geodesic shooting demonstrated in Fourier-Approximated Lie Algebras (FLASH) algorithm for fast diffeomorphic registration.

RESULTS

The proposed algorithm, named DiffeoRaptor, was validated with three public databases for the tasks of brain and abdominal image registration while comparing the results against three state-of-the-art techniques, including FLASH, NiftyReg, and Symmetric image Normalization (SyN).

CONCLUSIONS

Our results demonstrated that DiffeoRaptor offered comparable or better registration performance in terms of registration accuracy. Moreover, DiffeoRaptor produces smoother deformations than SyN in inter-modal and contrast registration. The code for DiffeoRaptor is publicly available at https://github.com/nimamasoumi/DiffeoRaptor .

Multimodal image synthesis based on disentanglement representations of anatomical and modality specific features, learned using uncooperative relativistic GAN

Growing number of methods for attenuation-coefficient map estimation from magnetic resonance (MR) images have recently been proposed because of the increasing interest in MR-guided radiotherapy and the introduction of positron emission tomography (PET) MR hybrid systems. We propose a deep-network ensemble incorporating stochastic-binary-anatomical encoders and imaging-modality variational autoencoders, to disentangle image-latent spaces into a space of modality-invariant anatomical features and spaces of modality attributes. The ensemble integrates modality-modulated decoders to normalize features and image intensities based on imaging modality. Besides promoting disentanglement, the architecture fosters uncooperative learning, offering ability to maintain anatomical structure in a cross-modality reconstruction. Introduction of a modality-invariant structural consistency constraint further enforces faithful embedding of anatomy. To improve training stability and fidelity of synthesized modalities, the ensemble is trained in a relativistic generative adversarial framework incorporating multiscale discriminators. Analyses of priors and network architectures as well as performance validation were performed on computed tomography (CT) and MR pelvis datasets. The proposed method demonstrated robustness against intensity inhomogeneity, improved tissue-class differentiation, and offered synthetic CT in Hounsfield units with intensities consistent and smooth across slices compared to the state-of-the-art approaches, offering median normalized mutual information of 1.28, normalized cross correlation of 0.97, and gradient cross correlation of 0.59 over 324 images.

Joint synthesis and registration network for deformable MR-CBCT image registration for neurosurgical guidance

Objective.The accuracy of navigation in minimally invasive neurosurgery is often challenged by deep brain deformations (up to 10 mm due to egress of cerebrospinal fluid during neuroendoscopic approach). We propose a deep learning-based deformable registration method to address such deformations between preoperative MR and intraoperative CBCT.Approach.The registration method uses a joint image synthesis and registration network (denoted JSR) to simultaneously synthesize MR and CBCT images to the CT domain and perform CT domain registration using a multi-resolution pyramid. JSR was first trained using a simulated dataset (simulated CBCT and simulated deformations) and then refined on real clinical images via transfer learning. The performance of the multi-resolution JSR was compared to a single-resolution architecture as well as a series of alternative registration methods (symmetric normalization (SyN), VoxelMorph, and image synthesis-based registration methods).Main results.JSR achieved median Dice coefficient (DSC) of 0.69 in deep brain structures and median target registration error (TRE) of 1.94 mm in the simulation dataset, with improvement from single-resolution architecture (median DSC = 0.68 and median TRE = 2.14 mm). Additionally, JSR achieved superior registration compared to alternative methods-e.g. SyN (median DSC = 0.54, median TRE = 2.77 mm), VoxelMorph (median DSC = 0.52, median TRE = 2.66 mm) and provided registration runtime of less than 3 s. Similarly in the clinical dataset, JSR achieved median DSC = 0.72 and median TRE = 2.05 mm.Significance.The multi-resolution JSR network resolved deep brain deformations between MR and CBCT images with performance superior to other state-of-the-art methods. The accuracy and runtime support translation of the method to further clinical studies in high-precision neurosurgery.

Medical Image Registration Algorithm Based on Bounded Generalized Gaussian Mixture Model

In this paper, a method for medical image registration based on the bounded generalized Gaussian mixture model is proposed. The bounded generalized Gaussian mixture model is used to approach the joint intensity of source medical images. The mixture model is formulated based on a maximum likelihood framework, and is solved by an expectation-maximization algorithm. The registration performance of the proposed approach on different medical images is verified through extensive computer simulations. Empirical findings confirm that the proposed approach is significantly better than other conventional ones.

Light-Weight Deformable Registration Using Adversarial Learning With Distilling Knowledge

Deformable registration is a crucial step in many medical procedures such as image-guided surgery and radiation therapy. Most recent learning-based methods focus on improving the accuracy by optimizing the non-linear spatial correspondence between the input images. Therefore, these methods are computationally expensive and require modern graphic cards for real-time deployment. In this paper, we introduce a new Light-weight Deformable Registration network that significantly reduces the computational cost while achieving competitive accuracy. In particular, we propose a new adversarial learning with distilling knowledge algorithm that successfully leverages meaningful information from the effective but expensive teacher network to the student network. We design the student network such as it is light-weight and well suitable for deployment on a typical CPU. The extensively experimental results on different public datasets show that our proposed method achieves state-of-the-art accuracy while significantly faster than recent methods. We further show that the use of our adversarial learning algorithm is essential for a time-efficiency deformable registration method. Finally, our source code and trained models are available at https://github.com/aioz-ai/LDR_ALDK.

Subclinical Diabetic Peripheral Vascular Disease and Epidemiology Using Logistic Regression Mathematical Model and Medical Image Registration Algorithm

The study aims to explore the effect of subclinical diabetic peripheral vascular disease and an epidemiological investigation of colour Doppler ultrasound images based on a logistic regression mathematical model and a medical image registration algorithm. Subclinical diabetes patients were selected as subjects, and after ultrasound colour Doppler ultrasonography of peripheral blood vessels, ultrasound images were taken. The experimental results show that the area under the curve (AUC) predicted by the model was 0.748, the sensitivity was 94.12%, and the specificity was 67.93%. All Δ were smaller than a single pixel. The detection rate of colour Doppler ultrasonography was 82.6%, which was significantly better than that of clinical examination ( $P<0.01$ ). The age, course of disease, SBP, low-density lipoprotein cholesterol (LDL-C), total cholesterol (TC), and triglyceride (TG) of the peripheral vascular disease group were significantly different from those of the no peripheral vascular disease group ( $P<0.05$ ). The incidence of peripheral vascular diseases and nonperipheral vascular diseases in male patients was remarkably higher than that in female patients ( $P<0.05$ ). Moreover, with the increase of age, the incidence of peripheral vascular disease and nonperipheral vascular disease in diabetic patients showed a trend of gradual increase ( $P<0.05$ ). In summary, the mathematical model and registration method have high accuracy for medical image registration of patients with the diabetes epidemic. In addition, the age, course of disease, SBP, LDL-C, TG, and TC of diabetic patients were significantly different from those of normal people, which can provide a reference for the development of later diabetes epidemiology.

Deformable MR-CT image registration using an unsupervised, dual-channel network for neurosurgical guidance

PURPOSE

The accuracy of minimally invasive, intracranial neurosurgery can be challenged by deformation of brain tissue - e.g., up to 10 mm due to egress of cerebrospinal fluid during neuroendoscopic approach. We report an unsupervised, deep learning-based registration framework to resolve such deformations between preoperative MR and intraoperative CT with fast runtime for neurosurgical guidance.

METHOD

The framework incorporates subnetworks for MR and CT image synthesis with a dual-channel registration subnetwork (with synthesis uncertainty providing spatially varying weights on the dual-channel loss) to estimate a diffeomorphic deformation field from both the MR and CT channels. An end-to-end training is proposed that jointly optimizes both the synthesis and registration subnetworks. The proposed framework was investigated using three datasets: (1) paired MR/CT with simulated deformations; (2) paired MR/CT with real deformations; and (3) a neurosurgery dataset with real deformation. Two state-of-the-art methods (Symmetric Normalization and VoxelMorph) were implemented as a basis of comparison, and variations in the proposed dual-channel network were investigated, including single-channel registration, fusion without uncertainty weighting, and conventional sequential training of the synthesis and registration subnetworks.

RESULTS

The proposed method achieved: (1) Dice coefficient = 0.82±0.07 and TRE = 1.2 ± 0.6 mm on paired MR/CT with simulated deformations; (2) Dice coefficient = 0.83 ± 0.07 and TRE = 1.4 ± 0.7 mm on paired MR/CT with real deformations; and (3) Dice = 0.79 ± 0.13 and TRE = 1.6 ± 1.0 mm on the neurosurgery dataset with real deformations. The dual-channel registration with uncertainty weighting demonstrated superior performance (e.g., TRE = 1.2 ± 0.6 mm) compared to single-channel registration (TRE = 1.6 ± 1.0 mm, p < 0.05 for CT channel and TRE = 1.3 ± 0.7 mm for MR channel) and dual-channel registration without uncertainty weighting (TRE = 1.4 ± 0.8 mm, p < 0.05). End-to-end training of the synthesis and registration subnetworks also improved performance compared to the conventional sequential training strategy (TRE = 1.3 ± 0.6 mm). Registration runtime with the proposed network was ∼3 s.

CONCLUSION

The deformable registration framework based on dual-channel MR/CT registration with spatially varying weights and end-to-end training achieved geometric accuracy and runtime that was superior to state-of-the-art baseline methods and various ablations of the proposed network. The accuracy and runtime of the method may be compatible with the requirements of high-precision neurosurgery.

Brain CT registration using hybrid supervised convolutional neural network

BACKGROUND

Image registration is an essential step in the automated interpretation of the brain computed tomography (CT) images of patients with acute cerebrovascular disease (ACVD). However, performing brain CT registration accurately and rapidly remains greatly challenging due to the large intersubject anatomical variations, low resolution of soft tissues, and heavy computation costs. To this end, the HSCN-Net, a hybrid supervised convolutional neural network, was developed for precise and fast brain CT registration.

METHOD

HSCN-Net generated synthetic deformation fields using a simulator as one supervision for one reference-moving image pair to address the problem of lack of gold standards. Furthermore, the simulator was designed to generate multiscale affine and elastic deformation fields to overcome the registration challenge posed by large intersubject anatomical deformation. Finally, HSCN-Net adopted a hybrid loss function constituted by deformation field and image similarity to improve registration accuracy and generalization capability. In this work, 101 CT images of patients were collected for model construction (57), evaluation (14), and testing (30). HSCN-Net was compared with the classical Demons and VoxelMorph models. Qualitative analysis through the visual evaluation of critical brain tissues and quantitative analysis by determining the endpoint error (EPE) between the predicted sparse deformation vectors and gold-standard sparse deformation vectors, image normalized mutual information (NMI), and the Dice coefficient of the middle cerebral artery (MCA) blood supply area were carried out to assess model performance comprehensively.

RESULTS

HSCN-Net and Demons had a better visual spatial matching performance than VoxelMorph, and HSCN-Net was more competent for smooth and large intersubject deformations than Demons. The mean EPE of HSCN-Net (3.29 mm) was less than that of Demons (3.47 mm) and VoxelMorph (5.12 mm); the mean Dice of HSCN-Net was 0.96, which was higher than that of Demons (0.90) and VoxelMorph (0.87); and the mean NMI of HSCN-Net (0.83) was slightly lower than that of Demons (0.84), but higher than that of VoxelMorph (0.81). Moreover, the mean registration time of HSCN-Net (17.86 s) was shorter than that of VoxelMorph (18.53 s) and Demons (147.21 s).

CONCLUSION

The proposed HSCN-Net could achieve accurate and rapid intersubject brain CT registration.

The lumbar region localization using bone anatomy feature graphs

The automatic localization of the lumbar region is essential for the diagnosis of lumbar diseases, the study of lumbar morphology, and the surgical planning. Although the existing researches have made great progress, it still faces several challenges. First, the various lumbar diseases and pathologies cause different abnormalities in the lumbar shape and appearance. Second, the numbers of lumbar vertebrae are irregular (some people have an additional vertebra L6). To tackle these challenges, we propose a novel lumbar region localization method based on bone anatomy feature graphs. Specifically, a feature graph (called LS) considering the anatomy of the sacrum and the lumbar vertebra is proposed to locate the inferior boundary of L5 or L6. A feature graph (called TL) considering the anatomy of the thoracic vertebra and the lumbar vertebra is proposed to locate the superior boundary of L1. Extensive experimental analysis is performed on a public available dataset xVertSeg and a private dataset which contains 197 CT scans. The localization results show that the proposed method is robust and can be applied to normal scans, scoliosis scans, deformity scans, hyperosteogeny scans, 6 lumbar vertebrae scans and lumbar implant scans. The Dice and Jaccard coefficients are 98.09 ± 0.84% and 96.27 ± 1.62% respectively. Graphical Abstract Lumbar Region Localization Framework.

CT to MR registration of complex deformations in the knee joint through dual quaternion interpolation of rigid transforms

Purpose.To develop a method that enables computed tomography (CT) to magnetic resonance (MR) image registration of complex deformations typically encountered in rotating joints such as the knee joint.Methods.We propose a workflow, denoted quaternion interpolated registration (QIR), consisting of three steps, which makes use of prior knowledge of tissue properties to initialise deformable registration. In the first step, the rigid skeletal components were individually registered. Next, the deformation of soft tissue was estimated using a dual quaternion-based interpolation method. In the final step, the registration was fine-tuned with a rigidity-constrained deformable registration step. The method was applied to paired, unregistered CT and MR images of the knee of 92 patients. It was compared to registration using B-Splines (BS) and B-Splines with a rigidity penalty (BSRP). Registration accuracy was evaluated using mutual information, and by calculating Dice similarity coefficient (DSC), mean absolute surface distance (MASD) and 95th percentile Hausdorff distance (HD95) on bone, and DSC on water and fat dominated tissue. To evaluate the rigidity of bone in the registration, the Jacobian determinant (JD) was calculated.Results.QIR achieved improved results with 0.93, 0.76 mm and 1.88 mm on the DSC, MASD and HD95 metrics on bone, compared to 0.87, 1.40 mm and 4.99 mm for method and 0.87, 1.40 mm and 3.56 mm for the BSRP method. The average DSC of water and fat was 0.77 and 0.86 for the QIR, 0.75 and 0.84 for BS and 0.74 and 0.84 for BSRP. Comparison of the median JD and median interquartile (IQR) ranges of the JD indicated that the QIR (1.00 median, 0.03 IQR) resulted in higher rigidity in the rigid skeletal tissues compared to the BS (0.98 median, 0.19 IQR) and BSRP (1.00 median, 0.05 IQR) methods.Conclusion.This study showed that QIR could improve the outcome of complex registration problems, encountered in joints involving rigid and non-rigid bodies such as occur in the knee, as compared to a conventional registration approach.

A level-wise spine registration framework to account for large pose changes

PURPOSES

Accurate and efficient spine registration is crucial to success of spine image guidance. However, changes in spine pose cause intervertebral motion that can lead to significant registration errors. In this study, we develop a geometrical rectification technique via nonlinear principal component analysis (NLPCA) to achieve level-wise vertebral registration that is robust to large changes in spine pose.

METHODS

We used explanted porcine spines and live pigs to develop and test our technique. Each sample was scanned with preoperative CT (pCT) in an initial pose and rescanned with intraoperative stereovision (iSV) in a different surgical posture. Patient registration rectified arbitrary spinal postures in pCT and iSV into a common, neutral pose through a parameterized moving-frame approach. Topologically encoded depth projection 2D images were then generated to establish invertible point-to-pixel correspondences. Level-wise point correspondences between pCT and iSV vertebral surfaces were generated via 2D image registration. Finally, closed-form vertebral level-wise rigid registration was obtained by directly mapping 3D surface point pairs. Implanted mini-screws were used as fiducial markers to measure registration accuracy.

RESULTS

In seven explanted porcine spines and two live animal surgeries (maximum in-spine pose change of 87.5 mm and 32.7 degrees averaged from all spines), average target registration errors (TRE) of 1.70  ±  0.15 mm and 1.85 ± 0.16 mm were achieved, respectively. The automated spine rectification took 3-5 min, followed by an additional 30 secs for depth image projection and level-wise registration.

CONCLUSIONS

Accuracy and efficiency of the proposed level-wise spine registration support its application in human open spine surgeries. The registration framework, itself, may also be applicable to other intraoperative imaging modalities such as ultrasound and MRI, which may expand utility of the approach in spine registration in general.

A method of 3D-3D multi-stage non-rigid registration of the spine based on binocular structured light

BACKGROUND

Intraoperative deformation and radiation are common problems in spinal surgery. A three-dimensional multi-stage dynamic iterative non-rigid registration method of the spine based on binocular structured light is proposed in this paper to overcome these problems.

METHOD

The problem of intraoperative radiation in traditional X-ray and CT is overcome by using binocular structured light. A three-dimensional spinal mask based on binary code is designed to reduce the influence of non-interested regions on the operation. Principal component analysis (PCA) algorithm is used to complete the rough registration between the preoperative CT model of the spine and the reconstructed surface of the intraoperative structured light. A new framework of multi-stage dynamic iterative non-rigid registration of the spine is proposed. The Iterative Closest Point (ICP) algorithm based on bidirectional selection is proposed to complete the single-stage registration of the spine. Then the multi-stage dynamic iterative registration of the spine is completed to solve the problem of large registration error caused by the deformation of the spine.

RESULTS

The method proposed in this paper is compared with traditional registration methods, and its application is verified experimentally. The results show that the registration accuracy and time of the proposed method are 0 . 51 ± 0 . 31  mm and 5 . 21 ± 0 . 23  s, respectively. The accuracy of the method is 81.5% and 78.2% higher than that of the contour method and the method of marker points, respectively.

CONCLUSIONS

The method can effectively avoid intraoperative radiation, reduce the registration error caused by the deformation of the spine, and has a high practicability.

Balanced multi-image demons for non-rigid registration of magnetic resonance images

A new approach is introduced for non-rigid registration of a pair of magnetic resonance images (MRI). It is a generalization of the demons algorithm with low computational cost, based on local information augmentation (by integrating multiple images) and balanced implementation. Specifically, a single deformation that best registers more pairs of images is estimated. All these images are extracted by applying different operators to the two original ones, processing local neighbors of each pixel. The following five images were found to be appropriate for MRI registration: the raw image and those obtained by contrast-limited adaptive histogram equalization, local median, local entropy and phase symmetry. Thus, each local point in the images is supplemented by augmented information coming by processing its neighbor. Moreover, image pairs are processed in alternation for each iteration of the algorithm (in a balanced way), computing both a forward and a backward registration. The new method (called balanced multi-image demons) is tested on sagittal MRIs from 10 patients, both in simulated and experimental conditions, improving the performances over the classical demons approach with minimal increase of the computational cost (processing time around twice that of standard demons). Specifically, a simulated deformation was applied to the MRIs (either original or corrupted by additive Gaussian or speckle noises). In all tested cases, the new algorithm improved the estimation of the simulated deformation (squared estimation error decreased by about 65% in the average). Moreover, statistically significant improvements were obtained in experimental tests, in which different brain regions (i.e., brain, posterior fossa and cerebellum) were identified by the atlas approach and compared to those manually delineated (in the average, Dice coefficient increased of about 6%). The conclusion is that a balanced method applied to multiple information extracted from neighboring pixels is a low cost approach to improve registration of MRIs.

An Indirect Multimodal Image Registration and Completion Method Guided by Image Synthesis

Multimodal registration is a challenging task due to the significant variations exhibited from images of different modalities. CT and MRI are two of the most commonly used medical images in clinical diagnosis, since MRI with multicontrast images, together with CT, can provide complementary auxiliary information. The deformable image registration between MRI and CT is essential to analyze the relationships among different modality images. Here, we proposed an indirect multimodal image registration method, i.e., sCT-guided multimodal image registration and problematic image completion method. In addition, we also designed a deep learning-based generative network, Conditional Auto-Encoder Generative Adversarial Network, called CAE-GAN, combining the idea of VAE and GAN under a conditional process to tackle the problem of synthetic CT (sCT) synthesis. Our main contributions in this work can be summarized into three aspects: (1) We designed a new generative network called CAE-GAN, which incorporates the advantages of two popular image synthesis methods, i.e., VAE and GAN, and produced high-quality synthetic images with limited training data. (2) We utilized the sCT generated from multicontrast MRI as an intermediary to transform multimodal MRI-CT registration into monomodal sCT-CT registration, which greatly reduces the registration difficulty. (3) Using normal CT as guidance and reference, we repaired the abnormal MRI while registering the MRI to the normal CT.

To Align Multimodal Lumbar Spine Images via Bending Energy Constrained Normalized Mutual Information

To align multimodal images is important for information fusion, clinical diagnosis, treatment planning, and delivery, while few methods have been dedicated to matching computerized tomography (CT) and magnetic resonance (MR) images of lumbar spine. This study proposes a coarse-to-fine registration framework to address this issue. Firstly, a pair of CT-MR images are rigidly aligned for global positioning. Then, a bending energy term is penalized into the normalized mutual information for the local deformation of soft tissues. In the end, the framework is validated on 40 pairs of CT-MR images from our in-house collection and 15 image pairs from the SpineWeb database. Experimental results show high overlapping ratio (in-house collection, vertebrae 0.97 ± 0.02, blood vessel 0.88 ± 0.07; SpineWeb, vertebrae 0.95 ± 0.03, blood vessel 0.93 ± 0.10) and low target registration error (in-house collection, ≤2.00 ± 0.62 mm; SpineWeb, ≤2.37 ± 0.76 mm) are achieved. The proposed framework concerns both the incompressibility of bone structures and the nonrigid deformation of soft tissues. It enables accurate CT-MR registration of lumbar spine images and facilitates image fusion, spine disease diagnosis, and interventional treatment delivery.

SPST-CNN: Spatial pyramid based searching and tagging of liver's intraoperative live views via CNN for minimal invasive surgery

Laparoscopic liver surgery is challenging to perform because of compromised ability of the surgeon to localize subsurface anatomy due to minimal invasive visibility. While image guidance has the potential to address this barrier, intraoperative factors, such as insufflations and variable degrees of organ mobilization from supporting ligaments, may generate substantial deformation. The navigation ability in terms of searching and tagging within liver views has not been characterized, and current object detection methods do not account for the mechanics of how these features could be applied to the liver images. In this research, we have proposed spatial pyramid based searching and tagging of liver's intraoperative views using convolution neural network (SPST-CNN). By exploiting a hybrid combination of an image pyramid at input and spatial pyramid pooling layer at deeper stages of SPST-CNN, we reveal the gains of full-image representations for searching and tagging variable scaled liver live views. SPST-CNN provides pinpoint searching and tagging of intraoperative liver views to obtain up-to-date information about the location and shape of the area of interest. Downsampling input using image pyramid enables SPST-CNN framework to deploy input images with a diversity of resolutions for achieving scale-invariance feature. We have compared the proposed approach to the four recent state-of-the-art approaches and our method achieved better mAP up to 85.9%.

Lung 4D CT Image Registration Based on High-Order Markov Random Field

To solve the problem that traditional image registration methods based on continuous optimization for large motion lung 4D CT image sequences are easy to fall into local optimal solutions and lead to serious misregistration, a novel image registration method based on high-order Markov Random Field (MRF) is proposed. By analyzing the effect of the deformation field constraint of the potential functions with different order cliques in MRF model, energy functions with high-order cliques form are designed separately for 2D and 3D images to preserve topology of the deformation field. In order to preserve the topology of the deformation field more effectively, it is necessary to apply a smooth term and a topology preservation term simultaneously in the energy function and use logarithmic function to impose a penalty on the Jacobian matrix with high-order cliques in the topology preservation term. For the complexity of the designed energy function with high-order cliques form, Markov Chain Monte Carlo (MCMC) algorithm is used to solve the optimization problem of the designed energy function. To address the high computational requirements in lung 4D CT image registration, a multi-level processing strategy is adopted to reduce the space complexity of the proposed registration method and promotes the computational efficiency. In the DIR-lab dataset with 4D CT images and the COPD (Chronic Obstructive Pulmonary Disease) dataset with 3D CT images, the average target registration error (TRE) of our proposed method can reach 0.95 mm respectively.

Space Independent Image Registration Using Curve-Based Method with Combination of Multiple Deformable Vector Fields

This paper proposes a novel curve-based or edge-based image registration technique that utilizes the curve transformation function and Gaussian function. It enables deformable image registration between images in different spaces, e.g., different color spaces or different medical image modalities. In particular, piecewise polynomial fitting is used to fit a curve and convert it to the global cubic B-spline control points. The transformation between the curves in the reference and source images are performed by using these control points. The image area is segmented with respect to the reference curve for the moving pixels. The Gaussian function, which is symmetric about the coordinates of the points of the reference curve, was used to improve the continuity in the intra- and inter-segmented areas. The overall result on curve transformation by means of the Hausdroff distance was 5.820 ± 1.127 pixels on average on several 512 × 512 synthetic images. The proposed method was compared with an ImageJ plugin, namely bUnwarpJ, and a software suite for deformable image registration and adaptive radiotherapy research, namely DIRART, to evaluate the image registration performance. The experimental result shows that the proposed method yielded better image registration performance than its counterparts. On average, the proposed method could reduce the root mean square error from 2970.66 before registration to 1677.94 after registration and can increase the normalized cross-correlation coefficient from 91.87% before registration to 97.40% after registration.

Motion Correction in Optical Resolution Photoacoustic Microscopy

In this paper, we are proposing a novel motion correction algorithm for high-resolution OR-PAM imaging. Our algorithm combines a modified demons-based tracking approach with a newly developed multi-scale vascular feature matching method to track motion between adjacent B-scan images without needing any reference object. We first applied this algorithm to correct motion artifacts within one three-dimensional (3D) data segment of rat iris obtained with OR-PAM imaging. We then extended the application of this algorithm to correct motions to obtain vasculature imaging in the whole mouse back. In here, we stitched five adjacent 3D data segments (large field-of-view) obtained while changing the focus of OR-PAM differently for each subarea. The results showed that the motion artifacts of both large blood vessels and microvessels could be accurately corrected in both cases. Compared to the manually stitching method and the traditional SIFT algorithm, the algorithm proposed in this paper has better performance in stitching adjacent data segments. The high accuracy of the motion correction algorithm makes it valuable in OR-PAM for high-resolution imaging of large animals and for quantitative functional imaging.

A momentum-based diffeomorphic demons framework for deformable MR-CT image registration

Neuro-navigated procedures require a high degree of geometric accuracy but are subject to geometric error from complex deformation in the deep brain-e.g. regions about the ventricles due to egress of cerebrospinal fluid (CSF) upon neuroendoscopic approach or placement of a ventricular shunt. We report a multi-modality, diffeomorphic, deformable registration method using momentum-based acceleration of the Demons algorithm to solve the transformation relating preoperative MRI and intraoperative CT as a basis for high-precision guidance. The registration method (pMI-Demons) extends the mono-modality, diffeomorphic form of the Demons algorithm to multi-modality registration using pointwise mutual information (pMI) as a similarity metric. The method incorporates a preprocessing step to nonlinearly stretch CT image values and incorporates a momentum-based approach to accelerate convergence. Registration performance was evaluated in phantom and patient images: first, the sensitivity of performance to algorithm parameter selection (including update and displacement field smoothing, histogram stretch, and the momentum term) was analyzed in a phantom study over a range of simulated deformations; and second, the algorithm was applied to registration of MR and CT images for four patients undergoing minimally invasive neurosurgery. Performance was compared to two previously reported methods (free-form deformation using mutual information (MI-FFD) and symmetric normalization using mutual information (MI-SyN)) in terms of target registration error (TRE), Jacobian determinant (J), and runtime. The phantom study identified optimal or nominal settings of algorithm parameters for translation to clinical studies. In the phantom study, the pMI-Demons method achieved comparable registration accuracy to the reference methods and strongly reduced outliers in TRE (p [Formula: see text] 0.001 in Kolmogorov-Smirnov test). Similarly, in the clinical study: median TRE  =  1.54 mm (0.83-1.66 mm interquartile range, IQR) for pMI-Demons compared to 1.40 mm (1.02-1.67 mm IQR) for MI-FFD and 1.64 mm (0.90-1.92 mm IQR) for MI-SyN. The pMI-Demons and MI-SyN methods yielded diffeomorphic transformations (J  >  0) that preserved topology, whereas MI-FFD yielded unrealistic (J  <  0) deformations subject to tissue folding and tearing. Momentum-based acceleration gave a ~35% speedup of the pMI-Demons method, providing registration runtime of 10.5 min (reduced to 2.2 min on GPU), compared to 15.5 min for MI-FFD and 34.7 min for MI-SyN. The pMI-Demons method achieved registration accuracy comparable to MI-FFD and MI-SyN, maintained diffeomorphic transformation similar to MI-SyN, and accelerated runtime in a manner that facilitates translation to image-guided neurosurgery.

Nonrigid Image Registration Using Spatially Region-Weighted Correlation Ratio and GPU-Acceleration

OBJECTIVE

Nonrigid image registration with high accuracy and efficiency remains a challenging task for medical image analysis. In this paper, we present the spatially region-weighted correlation ratio (SRWCR) as a novel similarity measure to improve the registration performance.

METHODS

SRWCR is rigorously deduced from a three-dimension joint probability density function combining the intensity channels with an extra spatial information channel. SRWCR estimates the optimal functional dependence between the intensities for each spatial bin, in which the spatial distribution modeled by a cubic B-spline function is used to differentiate the contribution of voxels. We also analytically derive the gradient of SRWCR with respect to the transformation parameters and optimize it using a quasi-Newton approach. Furthermore, we propose a GPU-based parallel mechanism to accelerate the computation of SRWCR and its derivatives.

RESULTS

The experiments on synthetic images, public four-dimensional thoracic computed tomography (CT) dataset, retinal optical coherence tomography data, and clinical CT and positron emission tomography images confirm that SRWCR significantly outperforms some state-of-the-art techniques such as spatially encoded mutual information and Robust PaTch-based cOrrelation Ration.

CONCLUSION

This study demonstrates the advantages of SRWCR in tackling the practical difficulties due to distinct intensity changes, serious speckle noise, or different imaging modalities.

SIGNIFICANCE

The proposed registration framework might be more reliable to correct the nonrigid deformations and more potential for clinical applications.

Inter-scanner Variation Independent Descriptors for Constrained Diffeomorphic Demons Registration of Retina OCT

PURPOSE

OCT offers high in-plane micrometer resolution, enabling studies of neurodegenerative and ocular-disease mechanisms via imaging of the retina at low cost. An important component to such studies is inter-scanner deformable image registration. Image quality of OCT, however, is suboptimal with poor signal-to-noise ratio and through-plane resolution. Geometry of OCT is additionally improperly defined. We developed a diffeomorphic deformable registration method incorporating constraints accommodating the improper geometry and a decentralized-modality-insensitive-neighborhood-descriptors (D-MIND) robust against degradation of OCT image quality and inter-scanner variability.

METHOD

The method, called D-MIND Demons, estimates diffeomorphisms using D-MINDs under constraints on the direction of velocity fields in a MIND-Demons framework. Descriptiveness of D-MINDs with/without denoising was ranked against four other shape/texture-based descriptors. Performance of D-MIND Demons and its variants incorporating other descriptors was compared for cross-scanner, intra- and inter-subject deformable registration using clinical retina OCT data.

RESULT

D-MINDs outperformed other descriptors with the difference in mutual descriptiveness between high-contrast and homogenous regions > 0.2. Among Demons variants, D-MIND-Demons was computationally efficient, demonstrating robustness against OCT image degradation (noise, speckle, intensity-non-uniformity, and poor through-plane resolution) and consistent registration accuracy [(4±4 μm) and (4±6 μm) in cross-scanner intra- and inter-subject registration] regardless of denoising.

CONCLUSIONS

A promising method for cross-scanner, intra- and inter-subject OCT image registration has been developed for ophthalmological and neurological studies of retinal structures. The approach could assist image segmentation, evaluation of longitudinal disease progression, and patient population analysis, which in turn, facilitate diagnosis and patient-specific treatment.

Performance evaluation of MIND demons deformable registration of MR and CT images in spinal interventions

Accurate intraoperative localization of target anatomy and adjacent nervous and vascular tissue is essential to safe, effective surgery, and multimodality deformable registration can be used to identify such anatomy by fusing preoperative CT or MR images with intraoperative images. A deformable image registration method has been developed to estimate viscoelastic diffeomorphisms between preoperative MR and intraoperative CT using modality-independent neighborhood descriptors (MIND) and a Huber metric for robust registration. The method, called MIND Demons, optimizes a constrained symmetric energy functional incorporating priors on smoothness, geodesics, and invertibility by alternating between Gauss-Newton optimization and Tikhonov regularization in a multiresolution scheme. Registration performance was evaluated for the MIND Demons method with a symmetric energy formulation in comparison to an asymmetric form, and sensitivity to anisotropic MR voxel-size was analyzed in phantom experiments emulating image-guided spine-surgery in comparison to a free-form deformation (FFD) method using local mutual information (LMI). Performance was validated in a clinical study involving 15 patients undergoing intervention of the cervical, thoracic, and lumbar spine. The target registration error (TRE) for the symmetric MIND Demons formulation (1.3  ±  0.8 mm (median  ±  interquartile)) outperformed the asymmetric form (3.6  ±  4.4 mm). The method demonstrated fairly minor sensitivity to anisotropic MR voxel size, with median TRE ranging 1.3-2.9 mm for MR slice thickness ranging 0.9-9.9 mm, compared to TRE  =  3.2-4.1 mm for LMI FFD over the same range. Evaluation in clinical data demonstrated sub-voxel TRE (<2 mm) in all fifteen cases with realistic deformations that preserved topology with sub-voxel invertibility (0.001 mm) and positive-determinant spatial Jacobians. The approach therefore appears robust against realistic anisotropic resolution characteristics in MR and yields registration accuracy suitable to application in image-guided spine-surgery.