Unsupervised registration for liver CT-MR images based on the multiscale integrated spatial-weight module and dual similarity guidance

PURPOSE

Multimodal registration is a key task in medical image analysis. Due to the large differences of multimodal images in intensity scale and texture pattern, it is a great challenge to design distinctive similarity metrics to guide deep learning-based multimodal image registration. Besides, since the limitation of the small receptive field, existing deep learning-based methods are mainly suitable for small deformation, but helpless for large deformation. To address the above issues, we present an unsupervised multimodal image registration method based on the multiscale integrated spatial-weight module and dual similarity guidance.

METHODS

In this method, a U-shape network with our multiscale integrated spatial-weight module is embedded into a multi-resolution image registration architecture to achieve end-to-end large deformation registration, where the spatial-weight module can effectively highlight the regions with large deformation and aggregate discriminative features, and the multi-resolution architecture further helps to solve the optimization problem of the network in a coarse-to-fine pattern. Furthermore, we introduce a special loss function based on dual similarity, which represents both global gray-scale similarity and local feature similarity, to optimize the unsupervised multimodal registration network.

RESULTS

We verified the effectiveness of the proposed method on liver CT-MR images. Experimental results indicate that the proposed method achieves the optimal DSC value and TRE value of 92.70 ± 1.75(%) and 6.52 ± 2.94(mm), compared with other state-of-the-art registration algorithms.

CONCLUSION

The proposed method can accurately estimate the large deformation field by aggregating multiscale features, and achieve higher registration accuracy and fast registration speed. Comparative experiments also demonstrate the effectiveness and generalization ability of the algorithm.

Cell specificity of Manganese-enhanced MRI signal in the cerebellum

Magnetic Resonance Imaging (MRI) resolution continues to improve, making it important to understand the cellular basis for different MRI contrast mechanisms. Manganese-enhanced MRI (MEMRI) produces layer-specific contrast throughout the brain enabling in vivo visualization of cellular cytoarchitecture, particularly in the cerebellum. Due to the unique geometry of the cerebellum, especially near the midline, 2D MEMRI images can be acquired from a relatively thick slice by averaging through areas of uniform morphology and cytoarchitecture to produce very high-resolution visualization of sagittal planes. In such images, MEMRI hyperintensity is uniform in thickness throughout the anterior-posterior axis of sagittal sections and is centrally located in the cerebellar cortex. These signal features suggested that the Purkinje cell layer, which houses the cell bodies of the Purkinje cells and the Bergmann glia, is the source of hyperintensity. Despite this circumstantial evidence, the cellular source of MRI contrast has been difficult to define. In this study, we quantified the effects of selective ablation of Purkinje cells or Bergmann glia on cerebellar MEMRI signal to determine whether signal could be assigned to one cell type. We found that the Purkinje cells, not the Bergmann glia, are the primary of source of the enhancement in the Purkinje cell layer. This cell-ablation strategy should be useful for determining the cell specificity of other MRI contrast mechanisms.

Optical biopsy mapping on endoscopic image mosaics with a marker-free probe

Gastric cancer is the second leading cause of cancer-related deaths worldwide. Early diagnosis significantly increases the chances of survival; therefore, improved assisted exploration and screening techniques are necessary. Previously, we made use of an augmented multi-spectral endoscope by inserting an optical probe into the instrumentation channel. However, the limited field of view and the lack of markings left by optical biopsies on the tissue complicate the navigation and revisit of the suspect areas probed in-vivo. In this contribution two innovative tools are introduced to significantly increase the traceability and monitoring of patients in clinical practice: (i) video mosaicing to build a more comprehensive and panoramic view of large gastric areas; (ii) optical biopsy targeting and registration with the endoscopic images. The proposed optical flow-based mosaicing technique selects images that minimize texture discontinuities and is robust despite the lack of texture and illumination variations. The optical biopsy targeting is based on automatic tracking of a free-marker probe in the endoscopic view using deep learning to dynamically estimate its pose during exploration. The accuracy of pose estimation is sufficient to ensure a precise overlapping of the standard white-light color image and the hyperspectral probe image, assuming that the small target area of the organ is almost flat. This allows the mapping of all spatio-temporally tracked biopsy sites onto the panoramic mosaic. Experimental validations are carried out from videos acquired on patients in hospital. The proposed technique is purely software-based and therefore easily integrable into clinical practice. It is also generic and compatible to any imaging modality that connects to a fiberscope.

Modality-agnostic self-supervised deep feature learning and fast instance optimisation for multimodal fusion in ultrasound-guided interventions

BACKGROUND AND OBJECTIVE

Fast and robust alignment of pre-operative MRI planning scans to intra-operative ultrasound is an important aspect for automatically supporting image-guided interventions. Thus far, learning-based approaches have failed to tackle the intertwined objectives of fast inference computation time and robustness to unexpectedly large motion and misalignment. In this work, we propose a novel method that decouples deep feature learning and the computation of long ranging local displacement probability maps from fast and robust global transformation prediction.

METHODS

In our approach, we firstly train a convolutional neural network (CNN) to extract modality-agnostic features with sub-second computation times for both 3D volumes during inference. Using sparsity-based network weight pruning, the model complexity and computation times can be substantially reduced. Based on these features, a large discretized search range of 3D motion vectors is explored to compute a probabilistic displacement map for each control point. These 3D probability maps are employed in our newly proposed, computationally efficient, instance optimisation that robustly estimates the most likely globally linear transformation that best reflects the local displacement beliefs subject to outlier rejection.

RESULTS

Our experimental validation demonstrates state-of-the-art accuracy on the challenging CuRIOUS dataset with average target registration errors of 2.50 mm, model size of only 1.2 MByte and run times of approx. 3 seconds for a full 3D multimodal registration.

CONCLUSION

We show that a significant improvement in accuracy and robustness can be gained with instance optimisation and our fast self-supervised deep learning model can achieve state-of-the-art accuracy on challenging registration task in only 3 seconds.

Anatomy-guided multimodal registration by learning segmentation without ground truth: Application to intraprocedural CBCT/MR liver segmentation and registration

Multimodal image registration has many applications in diagnostic medical imaging and image-guided interventions, such as Transcatheter Arterial Chemoembolization (TACE) of liver cancer guided by intraprocedural CBCT and pre-operative MR. The ability to register peri-procedurally acquired diagnostic images into the intraprocedural environment can potentially improve the intra-procedural tumor targeting, which will significantly improve therapeutic outcomes. However, the intra-procedural CBCT often suffers from suboptimal image quality due to lack of signal calibration for Hounsfield unit, limited FOV, and motion/metal artifacts. These non-ideal conditions make standard intensity-based multimodal registration methods infeasible to generate correct transformation across modalities. While registration based on anatomic structures, such as segmentation or landmarks, provides an efficient alternative, such anatomic structure information is not always available. One can train a deep learning-based anatomy extractor, but it requires large-scale manual annotations on specific modalities, which are often extremely time-consuming to obtain and require expert radiological readers. To tackle these issues, we leverage annotated datasets already existing in a source modality and propose an anatomy-preserving domain adaptation to segmentation network (APA2Seg-Net) for learning segmentation without target modality ground truth. The segmenters are then integrated into our anatomy-guided multimodal registration based on the robust point matching machine. Our experimental results on in-house TACE patient data demonstrated that our APA2Seg-Net can generate robust CBCT and MR liver segmentation, and the anatomy-guided registration framework with these segmenters can provide high-quality multimodal registrations.

3D digital breast cancer models with multimodal fusion algorithms

Breast cancer image fusion consists of registering and visualizing different sets of a patient synchronized torso and radiological images into a 3D model. Breast spatial interpretation and visualization by the treating physician can be augmented with a patient-specific digital breast model that integrates radiological images. But the absence of a ground truth for a good correlation between surface and radiological information has impaired the development of potential clinical applications.

A new image acquisition protocol was designed to acquire breast Magnetic Resonance Imaging (MRI) and 3D surface scan data with surface markers on the patient’s breasts and torso. A patient-specific digital breast model integrating the real breast torso and the tumor location was created and validated with a MRI/3D surface scan fusion algorithm in 16 breast cancer patients.

This protocol was used to quantify breast shape differences between different modalities, and to measure the target registration error of several variants of the MRI/3D scan fusion algorithm. The fusion of single breasts without the biomechanical model of pose transformation had acceptable registration errors and accurate tumor locations. The performance of the fusion algorithm was not affected by breast volume. Further research and virtual clinical interfaces could lead to fast integration of this fusion technology into clinical practice.

•

MRI/3D surface scan fusion algorithm to create 3D breast cancer models.

•

A replicable clinical validation protocol for MRI/3D surface scan fusion algorithms.

•

Anthropometric study that quantifies breast deformations by area in MRI and 3D scans.

Automated postoperative muscle assessment of hip arthroplasty patients using multimodal imaging joint segmentation

BACKGROUND AND OBJECTIVE

In patients treated with hip arthroplasty, the muscular condition and presence of inflammatory reactions are assessed using magnetic resonance imaging (MRI). As MRI lacks contrast for bony structures, computed tomography (CT) is preferred for clinical evaluation of bone tissue and orthopaedic surgical planning. Combining the complementary information of MRI and CT could improve current clinical practice for diagnosis, monitoring and treatment planning. In particular, the different contrast of these modalities could help better quantify the presence of fatty infiltration to characterise muscular condition and assess implant failure. In this work, we combine CT and MRI for joint bone and muscle segmentation and we propose a novel Intramuscular Fat Fraction estimation method for the quantification of muscle atrophy.

METHODS

Our multimodal framework is able to segment healthy and pathological musculoskeletal structures as well as implants, and develops into three steps. First, input images are pre-processed to improve the low quality of clinically acquired images and to reduce the noise associated with metal artefact. Subsequently, CT and MRI are non-linearly aligned using a novel approach which imposes rigidity constraints on bony structures to ensure realistic deformation. Finally, taking advantage of a multimodal atlas we created for this task, a multi-atlas based segmentation delineates pelvic bones, abductor muscles and implants on both modalities jointly. From the obtained segmentation, a multimodal estimation of the Intramuscular Fat Fraction can be automatically derived.

RESULTS

Evaluation of the segmentation in a leave-one-out cross-validation study on 22 hip sides resulted in an average Dice score of 0.90 for skeletal and 0.84 for muscular structures. Our multimodal Intramuscular Fat Fraction was benchmarked on 27 different cases against a standard radiological score, showing stronger association than a single modality approach in a one-way ANOVA F-test analysis.

CONCLUSIONS

The proposed framework represents a promising tool to support image analysis in hip arthroplasty, being robust to the presence of implants and associated image artefacts. By allowing for the automated extraction of a muscle atrophy imaging biomarker, it could quantitatively inform the decision-making process about patient's management.

An ultrasound based platform for image-guided radiotherapy in canine bladder cancer patients

Background and purpose

Ultrasound (US) is a non-invasive, non-radiographic imaging technique with high spatial and temporal resolution that can be used for localizing soft-tissue structures and tumors in real-time during radiotherapy (RT) (inter- and intra-fraction). A comprehensive approach incorporating an in-house 3D-US system within RT is presented. This system is easier to adopt into existing treatment protocols than current US based systems, with the aim of providing millimeter intra-fraction alignment errors and sensitivity to track intra-fraction bladder movement.

Materials and methods

An in-house integrated US manipulator and platform was designed to relate the computed tomographic (CT) scanner, 3D-US and linear accelerator coordinate systems. An agar-based phantom with measured speed of sound and densities consistent with tissues surrounding the bladder was rotated (0–45°) and translated (up to 55 mm) relative to the US and CT coordinate systems to validate this device. After acquiring and integrating CT and US images into the treatment planning system, US-to-US and US-to-CT images were co-registered to re-align the phantom relative to the linear accelerator.

Results

Statistical errors from US-to-US registrations for various patient orientations ranged from 0.1 to 1.7 mm for x, y, and z translation components, and 0.0–1.1° for rotational components. Statistical errors from US-to-CT registrations were 0.3–1.2 mm for the x, y and z translational components and 0.1–2.5° for the rotational components.

Conclusions

An ultrasound-based platform was designed, constructed and tested on a CT/US tissue-equivalent phantom to track bladder displacement with a statistical uncertainty to correct and track inter- and intra-fractional displacements of the bladder during radiation treatments.

Three-dimensional reconstruction of the distribution of elemental tags in single cells using laser ablation ICP-mass spectrometry via registration approaches

This paper describes a workflow towards the reconstruction of the three-dimensional elemental distribution profile within human cervical carcinoma cells (HeLa), at a spatial resolution down to 1 μm, employing state-of-the-art laser ablation-inductively coupled plasma-mass spectrometry (LA-ICP-MS) instrumentation. The suspended cells underwent a series of fixation/embedding protocols and were stained with uranyl acetate and an Ir-based DNA intercalator. A priori, laboratory-based absorption micro-computed tomography (μ-CT) was applied to acquire a reference frame of the morphology of the cells and their spatial distribution before sectioning. After CT analysis, a trimmed 300 × 300 × 300 μm³ block was sectioned into a sequential series of 132 sections with a thickness of 2 μm, which were subjected to LA-ICP-MS imaging. A pixel acquisition rate of 250 pixels s^-1 was achieved, through a bidirectional scanning strategy. After acquisition, the two-dimensional elemental images were reconstructed using the timestamps in the laser log file. The synchronization of the data required an improved optimization algorithm, which forces the pixels of scans in different ablation directions to be spatially coherent in the direction orthogonal to the scan direction. The volume was reconstructed using multiple registration approaches. Registration using the section outline itself as a fiducial marker resulted into a volume which was in good agreement with the morphology visualized in the μ-CT volume. The 3D μ-CT volume could be registered to the LA-ICP-MS volume, consisting of 2.9 × 10⁷ voxels, and the nucleus dimensions in 3D space could be derived.

An atlas-based multimodal registration method for 2D images with discrepancy structures

An atlas-based multimodal registration method for 2-dimension images with discrepancy structures was proposed in this paper. Atlas was utilized for complementing the discrepancy structure information in multimodal medical images. The scheme includes three steps: floating image to atlas registration, atlas to reference image registration, and field-based deformation. To evaluate the performance, a frame model, a brain model, and clinical images were employed in registration experiments. We measured the registration performance by the squared sum of intensity differences. Results indicate that this method is robust and performs better than the direct registration for multimodal images with discrepancy structures. We conclude that the proposed method is suitable for multimodal images with discrepancy structures. Graphical Abstract An Atlas-based multimodal registration method schematic diagram.

Automated, reference-free local error assessment of multimodal deformable image registration for radiotherapy in the head and neck

BACKGROUND

Head and neck MR-CT deformable image registration (DIR) for radiotherapy planning is hindered by the lack of both ground-truth and per-patient accuracy assessment methods. This study assesses novel post-registration reference-free error assessment algorithms, based on local rigid re-registration of native and pseudomodality images.

METHODS

Head and neck MR obtained in and out of the treatment position underwent DIR to planning CT. Block-wise mutual information (b-MI) and pseudomodality mutual information (b-pmMI) algorithms were validated against applied rotations and translations. Inherent registration error detection was compared across 14 patient datasets.

RESULTS

Using radiotherapy position MR-CT DIR, quantitative comparison of applied rotations and translations revealed that errors between 1 and 4 mm were accurately determined by both algorithms. Using diagnostic position MR-CT DIR, translations of up to 5 mm were accurately detected within the gross tumour volume by both methods. In 14 patient datasets, b-MI and b-pmMI detected similar errors with improved stability in regions of low contrast or CT artefact and a 10-fold speedup for b-pmMI.

CONCLUSIONS

b-MI and b-pmMI algorithms have been validated as providing accurate reference-free quantitative assessment of DIR accuracy on a per-patient basis. b-pmMI is faster and more robust in the presence of modality-specific information.

Guiding multimodal registration with learned optimization updates

In this paper, we address the multimodal registration problem from a novel perspective, aiming to predict the transformation aligning images directly from their visual appearance. We formulate the prediction as a supervised regression task, with joint image descriptors as input and the output are the parameters of the transformation that guide the moving image towards alignment. We model the joint local appearance with context aware descriptors that capture both local and global cues simultaneously in the two modalities, while the regression function is based on the gradient boosted trees method capable of handling the very large contextual feature space. The good properties of our predictions allow us to couple them with a simple gradient-based optimization for the final registration. Our approach can be applied to any transformation parametrization as well as a broad range of modality pairs. Our method learns the relationship between the intensity distributions of a pair of modalities by using prior knowledge in the form of a small training set of aligned image pairs (in the order of 1-5 in our experiments). We demonstrate the flexibility and generality of our method by evaluating its performance on a variety of multimodal imaging pairs obtained from two publicly available datasets, RIRE (brain MR, CT and PET) and IXI (brain MR). We also show results for the very challenging deformable registration of Intravascular Ultrasound and Histology images. In these experiments, our approach has a larger capture range when compared to other state-of-the-art methods, while improving registration accuracy in complex cases.

Automated Registration of Freehand B-Mode Ultrasound and Magnetic Resonance Imaging of the Carotid Arteries Based on Geometric Features

An automated method for registering B-mode ultrasound (US) and magnetic resonance imaging (MRI) of the carotid arteries is proposed. The registration uses geometric features, namely, lumen centerlines and lumen segmentations, which are extracted fully automatically from the images after manual annotation of three seed points in US and MRI. The registration procedure starts with alignment of the lumen centerlines using a point-based registration algorithm. The resulting rigid transformation is used to initialize a rigid and subsequent non-rigid registration procedure that jointly aligns centerlines and segmentations by minimizing a weighted sum of the Euclidean distance between centerlines and the dissimilarity between segmentations. The method was evaluated in 28 carotid arteries from eight patients and six healthy volunteers. First, the automated US lumen segmentation method was validated and optimized in a cross-validation experiment. Next, the effect of the weighting parameter of the proposed registration dissimilarity metric and the control point spacing in the non-rigid registration was evaluated. Finally, the proposed registration method was evaluated in comparison to an existing intensity-and-point-based method, a registration using only the centerlines and a registration using manual US lumen segmentations. Registration accuracy was measured in terms of the mean surface distance between manual US segmentations and the registered MRI segmentations. The average mean surface distance was 0.78 ± 0.34 mm for all subjects, 0.65 ± 0.09 mm for healthy volunteers and 0.87 ± 0.42 mm for patients. The results for the complete set were significantly better (Wilcoxon test, p < 0.01) than the results for the intensity-and-point-based method and the centerline-based registration method. We conclude that the proposed method can robustly and accurately register US and MR images of the carotid artery, allowing multimodal analysis of the carotid plaque to improve plaque assessment.

Robust inverse-consistent affine CT-MR registration in MRI-assisted and MRI-alone prostate radiation therapy

BACKGROUND

CT-MR registration is a critical component of many radiation oncology protocols. In prostate external beam radiation therapy, it allows the propagation of MR-derived contours to reference CT images at the planning stage, and it enables dose mapping during dosimetry studies. The use of carefully registered CT-MR atlases allows the estimation of patient specific electron density maps from MRI scans, enabling MRI-alone radiation therapy planning and treatment adaptation. In all cases, the precision and accuracy achieved by registration influences the quality of the entire process.

PROBLEM

Most current registration algorithms do not robustly generalize and lack inverse-consistency, increasing the risk of human error and acting as a source of bias in studies where information is propagated in a particular direction, e.g. CT to MR or vice versa. In MRI-based treatment planning where both CT and MR scans serve as spatial references, inverse-consistency is critical, if under-acknowledged.

PURPOSE

A robust, inverse-consistent, rigid/affine registration algorithm that is well suited to CT-MR alignment in prostate radiation therapy is presented.

METHOD

The presented method is based on a robust block-matching optimization process that utilises a half-way space definition to maintain inverse-consistency. Inverse-consistency substantially reduces the influence of the order of input images, simplifying analysis, and increasing robustness. An open source implementation is available online at http://aehrc.github.io/Mirorr/.

RESULTS

Experimental results on a challenging 35 CT-MR pelvis dataset demonstrate that the proposed method is more accurate than other popular registration packages and is at least as accurate as the state of the art, while being more robust and having an order of magnitude higher inverse-consistency than competing approaches.

CONCLUSION

The presented results demonstrate that the proposed registration algorithm is readily applicable to prostate radiation therapy planning.