Slice-to-volume medical image registration: A survey

Advanced Image Stitching Method for Dual-Sensor Inspection

Efficient image stitching plays a vital role in the Non-Destructive Evaluation (NDE) of infrastructures. An essential challenge in the NDE of infrastructures is precisely visualizing defects within large structures. The existing literature predominantly relies on high-resolution close-distance images to detect surface or subsurface defects. While the automatic detection of all defect types represents a significant advancement, understanding the location and continuity of defects is imperative. It is worth noting that some defects may be too small to capture from a considerable distance. Consequently, multiple image sequences are captured and processed using image stitching techniques. Additionally, visible and infrared data fusion strategies prove essential for acquiring comprehensive information to detect defects across vast structures. Hence, there is a need for an effective image stitching method appropriate for infrared and visible images of structures and industrial assets, facilitating enhanced visualization and automated inspection for structural maintenance. This paper proposes an advanced image stitching method appropriate for dual-sensor inspections. The proposed image stitching technique employs self-supervised feature detection to enhance the quality and quantity of feature detection. Subsequently, a graph neural network is employed for robust feature matching. Ultimately, the proposed method results in image stitching that effectively eliminates perspective distortion in both infrared and visible images, a prerequisite for subsequent multi-modal fusion strategies. Our results substantially enhance the visualization capabilities for infrastructure inspection. Comparative analysis with popular state-of-the-art methods confirms the effectiveness of the proposed approach.

Recursive Deformable Pyramid Network for Unsupervised Medical Image Registration

Complicated deformation problems are frequently encountered in medical image registration tasks. Although various advanced registration models have been proposed, accurate and efficient deformable registration remains challenging, especially for handling the large volumetric deformations. To this end, we propose a novel recursive deformable pyramid (RDP) network for unsupervised non-rigid registration. Our network is a pure convolutional pyramid, which fully utilizes the advantages of the pyramid structure itself, but does not rely on any high-weight attentions or transformers. In particular, our network leverages a step-by-step recursion strategy with the integration of high-level semantics to predict the deformation field from coarse to fine, while ensuring the rationality of the deformation field. Meanwhile, due to the recursive pyramid strategy, our network can effectively attain deformable registration without separate affine pre-alignment. We compare the RDP network with several existing registration methods on three public brain magnetic resonance imaging (MRI) datasets, including LPBA, Mindboggle and IXI. Experimental results demonstrate our network consistently outcompetes state of the art with respect to the metrics of Dice score, average symmetric surface distance, Hausdorff distance, and Jacobian. Even for the data without the affine pre-alignment, our network maintains satisfactory performance on compensating for the large deformation. The code is publicly available at https://github.com/ZAX130/RDP.

Three-dimensional histology reveals dissociable human hippocampal long-axis gradients of Alzheimer's pathology

INTRODUCTION

Three-dimensional (3D) histology analyses are essential to overcome sampling variability and understand pathological differences beyond the dissection axis. We present Path2MR, the first pipeline allowing 3D reconstruction of sparse human histology without a magnetic resonance imaging (MRI) reference. We implemented Path2MR with post-mortem hippocampal sections to explore pathology gradients in Alzheimer's disease.

METHODS

Blockface photographs of brain hemisphere slices are used for 3D reconstruction, from which an MRI-like image is generated using machine learning. Histology sections are aligned to the reconstructed hemisphere and subsequently to an atlas in standard space.

RESULTS

Path2MR successfully registered histological sections to their anatomic position along the hippocampal longitudinal axis. Combined with histopathology quantification, we found an expected peak of tau pathology at the anterior end of the hippocampus, whereas amyloid-beta (Aβ) displayed a quadratic anterior-posterior distribution.

CONCLUSION

Path2MR, which enables 3D histology using any brain bank data set, revealed significant differences along the hippocampus between tau and Aβ.

HIGHLIGHTS

Path2MR enables three-dimensional (3D) brain reconstruction from blockface dissection photographs. This pipeline does not require dense specimen sampling or a subject-specific magnetic resonance (MR) image. Anatomically consistent mapping of hippocampal sections was obtained with Path2MR. Our analyses revealed an anterior-posterior gradient of hippocampal tau pathology. In contrast, the peak of amyloid-beta (Aβ) deposition was closer to the hippocampal body.

Computer-aided accurate calculation of interacted volumes for 3D isosurface point clouds of molecular electrostatic potential

The quality of chiral environment (i.e. catalytic pocket) is directly related to the performance of chiral catalysts. The existing methods need super computing power and time, i.e., it is difficult to quickly judge the interaction between chiral catalysts and substrates for accurately evaluating the effects of chiral catalytic pockets. In this paper, for the 3D isosurface point clouds of molecular electrostatic potential, by using computer simulations, we propose a robust method to detect interacted points, and then accurately have the corresponding interacted volumes. First, by using the existing marching cubes algorithm, we construct the 3D models with triangular surface for isosurface point clouds of molecular electrostatic potentials. Second, by using our improved hierarchical bounding boxes algorithm, we significantly filter out most redundant non-collision points. Third, by using the normal vectors of the remaining points and related triangles, we robustly determine the interacted points to construct interacted sets. And finally, by combining the classical slicing with our multi-contour segmenting, we accurately calculate the interacted volumes. Over three groups of the point clouds of the chemical molecules, experimental results show that our method effectively removes the non-interacted points at average rates of 71.65%, 77.76%, and 71.82%, and calculates the interacted volumes with the average relative errors of 1.7%, 1.6%, and 1.9%, respectively.

Three-dimensional histology reveals dissociable human hippocampal long axis gradients of Alzheimer's pathology

INTRODUCTION

Three-dimensional (3D) histology analyses are essential to overcome sampling variability and understand pathological differences beyond the dissection axis. We present Path2MR, the first pipeline allowing 3D reconstruction of sparse human histology without an MRI reference. We implemented Path2MR with post-mortem hippocampal sections to explore pathology gradients in Alzheimer's Disease.

METHODS

Blockface photographs of brain hemisphere slices are used for 3D reconstruction, from which an MRI-like image is generated using machine learning. Histology sections are aligned to the reconstructed hemisphere and subsequently to an atlas in standard space.

RESULTS

Path2MR successfully registered histological sections to their anatomical position along the hippocampal longitudinal axis. Combined with histopathology quantification, we found an expected peak of tau pathology at the anterior end of the hippocampus, while amyloid-β displayed a quadratic anterior-posterior distribution.

CONCLUSION

Path2MR, which enables 3D histology using any brain bank dataset, revealed significant differences along the hippocampus between tau and amyloid-β.

Multi-class point cloud completion networks for 3D cardiac anatomy reconstruction from cine magnetic resonance images

Cine magnetic resonance imaging (MRI) is the current gold standard for the assessment of cardiac anatomy and function. However, it typically only acquires a set of two-dimensional (2D) slices of the underlying three-dimensional (3D) anatomy of the heart, thus limiting the understanding and analysis of both healthy and pathological cardiac morphology and physiology. In this paper, we propose a novel fully automatic surface reconstruction pipeline capable of reconstructing multi-class 3D cardiac anatomy meshes from raw cine MRI acquisitions. Its key component is a multi-class point cloud completion network (PCCN) capable of correcting both the sparsity and misalignment issues of the 3D reconstruction task in a unified model. We first evaluate the PCCN on a large synthetic dataset of biventricular anatomies and observe Chamfer distances between reconstructed and gold standard anatomies below or similar to the underlying image resolution for multiple levels of slice misalignment. Furthermore, we find a reduction in reconstruction error compared to a benchmark 3D U-Net by 32% and 24% in terms of Hausdorff distance and mean surface distance, respectively. We then apply the PCCN as part of our automated reconstruction pipeline to 1000 subjects from the UK Biobank study in a cross-domain transfer setting and demonstrate its ability to reconstruct accurate and topologically plausible biventricular heart meshes with clinical metrics comparable to the previous literature. Finally, we investigate the robustness of our proposed approach and observe its capacity to successfully handle multiple common outlier conditions.

Bone marrow sparing oriented multi-model image registration in cervical cancer radiotherapy

Cervical cancer poses a serious threat to the health of women and radiotherapy is one of the primary treatment methods for this condition. However, this treatment is associated with a high risk of causing acute hematologic toxicity. Delineating the bone marrow (BM) for sparing based on computer tomography (CT) images before radiotherapy can effectively avoid this risk. Unfortunately, compared to magnetic resonance (MR) images, CT images lack the ability to express the activity of BM. Therefore, medical practitioners currently manually delineate the BM on CT images by corresponding to MR images. However, the manual delineation of BM is time-consuming and cannot guarantee accuracy due to the inconsistency of the CT-MR multimodal images. This study proposes a multimodal image-oriented automatic registration method for pelvic BM sparing. The proposed method includes three-dimensional (3D) bone point clouds reconstruction and an iterative closest point registration based on a local spherical system for marking BM on CT images. By introducing a joint coordinate system that combines the global Cartesian coordinate system with the local point clouds' spherical coordinate system, the increasement of point descriptive dimension avoids the local optimal registration and improves the registration accuracy. Experiments on the dataset of patients demonstrate that our proposed method can enhance the multimodal image registration accuracy and efficiency for medical practitioners in BM-sparing of cervical cancer radiotherapy. The method proposed in this contribution might also provide a solution to multimodal registration, especially in multimodal sequential images in other clinical applications, such as the diagnosis of cervical cancer and the preservation of normal organs during radiotherapy.

3D X-ray Histology for the Investigation of Temporal Lobe Epilepsy in a Mouse Model

The most common form of epilepsy among adults is mesial temporal lobe epilepsy (mTLE), with seizures often originating in the hippocampus due to abnormal electrical activity. The gold standard for the histopathological analysis of mTLE is histology, which is a two-dimensional technique. To fill this gap, we propose complementary three-dimensional (3D) X-ray histology. Herein, we used synchrotron radiation-based phase-contrast microtomography with 1.6 μm-wide voxels for the post mortem visualization of tissue microstructure in an intrahippocampal-kainate mouse model for mTLE. We demonstrated that the 3D X-ray histology of unstained, unsectioned, paraffin-embedded brain hemispheres can identify hippocampal sclerosis through the loss of pyramidal neurons in the first and third regions of the Cornu ammonis as well as granule cell dispersion within the dentate gyrus. Morphology and density changes during epileptogenesis were quantified by segmentations from a deep convolutional neural network. Compared to control mice, the total dentate gyrus volume doubled and the granular layer volume quadrupled 21 days after injecting kainate. Subsequent sectioning of the same mouse brains allowed for benchmarking 3D X-ray histology against well-established histochemical and immunofluorescence stainings. Thus, 3D X-ray histology is a complementary neuroimaging tool to unlock the third dimension for the cellular-resolution histopathological analysis of mTLE.

Review on deep learning fetal brain segmentation from Magnetic Resonance images

Brain segmentation is often the first and most critical step in quantitative analysis of the brain for many clinical applications, including fetal imaging. Different aspects challenge the segmentation of the fetal brain in magnetic resonance imaging (MRI), such as the non-standard position of the fetus owing to his/her movements during the examination, rapid brain development, and the limited availability of imaging data. In recent years, several segmentation methods have been proposed for automatically partitioning the fetal brain from MR images. These algorithms aim to define regions of interest with different shapes and intensities, encompassing the entire brain, or isolating specific structures. Deep learning techniques, particularly convolutional neural networks (CNNs), have become a state-of-the-art approach in the field because they can provide reliable segmentation results over heterogeneous datasets. Here, we review the deep learning algorithms developed in the field of fetal brain segmentation and categorize them according to their target structures. Finally, we discuss the perceived research gaps in the literature of the fetal domain, suggesting possible future research directions that could impact the management of fetal MR images.

Ultrasound Frame-to-Volume Registration via Deep Learning for Interventional Guidance

Fusing intraoperative 2-D ultrasound (US) frames with preoperative 3-D magnetic resonance (MR) images for guiding interventions has become the clinical gold standard in image-guided prostate cancer biopsy. However, developing an automatic image registration system for this application is challenging because of the modality gap between US/MR and the dimensionality gap between 2-D/3-D data. To overcome these challenges, we propose a novel US frame-to-volume registration (FVReg) pipeline to bridge the dimensionality gap between 2-D US frames and 3-D US volume. The developed pipeline is implemented using deep neural networks, which are fully automatic without requiring external tracking devices. The framework consists of three major components, including one) a frame-to-frame registration network (Frame2Frame) that estimates the current frame's 3-D spatial position based on previous video context, two) a frame-to-slice correction network (Frame2Slice) adjusting the estimated frame position using the 3-D US volumetric information, and three) a similarity filtering (SF) mechanism selecting the frame with the highest image similarity with the query frame. We validated our method on a clinical dataset with 618 subjects and tested its potential on real-time 2-D-US to 3-D-MR fusion navigation tasks. The proposed FVReg achieved an average target navigation error of 1.93 mm at 5-14 fps. Our source code is publicly available at https://github.com/DIAL-RPI/Frame-to-Volume-Registration.

Fetal MRI by Robust Deep Generative Prior Reconstruction and Diffeomorphic Registration

Magnetic resonance imaging of whole fetal body and placenta is limited by different sources of motion affecting the womb. Usual scanning techniques employ single-shot multi-slice sequences where anatomical information in different slices may be subject to different deformations, contrast variations or artifacts. Volumetric reconstruction formulations have been proposed to correct for these factors, but they must accommodate a non-homogeneous and non-isotropic sampling, so regularization becomes necessary. Thus, in this paper we propose a deep generative prior for robust volumetric reconstructions integrated with a diffeomorphic volume to slice registration method. Experiments are performed to validate our contributions and compare with ifdefined tmiformat R2.5a state of the art method methods in the literature in a cohort of 72 fetal datasets in the range of 20-36 weeks gestational age. Results suggest improved image resolution Quantitative as well as radiological assessment suggest improved image quality and more accurate prediction of gestational age at scan is obtained when comparing to a state of the art reconstruction method methods. In addition, gestational age prediction results from our volumetric reconstructions compare favourably are competitive with existing brain-based approaches, with boosted accuracy when integrating information of organs other than the brain. Namely, a mean absolute error of 0.618 weeks ( R²=0.958 ) is achieved when combining fetal brain and trunk information.

AFFIRM: Affinity Fusion-Based Framework for Iteratively Random Motion Correction of Multi-Slice Fetal Brain MRI

Multi-slice magnetic resonance images of the fetal brain are usually contaminated by severe and arbitrary fetal and maternal motion. Hence, stable and robust motion correction is necessary to reconstruct high-resolution 3D fetal brain volume for clinical diagnosis and quantitative analysis. However, the conventional registration-based correction has a limited capture range and is insufficient for detecting relatively large motions. Here, we present a novel Affinity Fusion-based Framework for Iteratively Random Motion (AFFIRM) correction of the multi-slice fetal brain MRI. It learns the sequential motion from multiple stacks of slices and integrates the features between 2D slices and reconstructed 3D volume using affinity fusion, which resembles the iterations between slice-to-volume registration and volumetric reconstruction in the regular pipeline. The method accurately estimates the motion regardless of brain orientations and outperforms other state-of-the-art learning-based methods on the simulated motion-corrupted data, with a 48.4% reduction of mean absolute error for rotation and 61.3% for displacement. We then incorporated AFFIRM into the multi-resolution slice-to-volume registration and tested it on the real-world fetal MRI scans at different gestation stages. The results indicated that adding AFFIRM to the conventional pipeline improved the success rate of fetal brain super-resolution reconstruction from 77.2% to 91.9%.

Tensor image registration library: Deformable registration of stand-alone histology images to whole-brain post-mortem MRI data

BACKGROUND

Accurate registration between microscopy and MRI data is necessary for validating imaging biomarkers against neuropathology, and to disentangle complex signal dependencies in microstructural MRI. Existing registration methods often rely on serial histological sampling or significant manual input, providing limited scope to work with a large number of stand-alone histology sections. Here we present a customisable pipeline to assist the registration of stand-alone histology sections to whole-brain MRI data.

METHODS

Our pipeline registers stained histology sections to whole-brain post-mortem MRI in 4 stages, with the help of two photographic intermediaries: a block face image (to undistort histology sections) and coronal brain slab photographs (to insert them into MRI space). Each registration stage is implemented as a configurable stand-alone Python script using our novel platform, Tensor Image Registration Library (TIRL), which provides flexibility for wider adaptation. We report our experience of registering 87 PLP-stained histology sections from 14 subjects and perform various experiments to assess the accuracy and robustness of each stage of the pipeline.

RESULTS

All 87 histology sections were successfully registered to MRI. Histology-to-block registration (Stage 1) achieved 0.2-0.4 mm accuracy, better than commonly used existing methods. Block-to-slice matching (Stage 2) showed great robustness in automatically identifying and inserting small tissue blocks into whole brain slices with 0.2 mm accuracy. Simulations demonstrated sub-voxel level accuracy (0.13 mm) of the slice-to-volume registration (Stage 3) algorithm, which was observed in over 200 actual brain slice registrations, compensating 3D slice deformations up to 6.5 mm. Stage 4 combined the previous stages and generated refined pixelwise aligned multi-modal histology-MRI stacks.

CONCLUSIONS

Our open-source pipeline provides robust automation tools for registering stand-alone histology sections to MRI data with sub-voxel level precision, and the underlying framework makes it readily adaptable to a diverse range of microscopy-MRI studies.

Combining High-Resolution Hard X-ray Tomography and Histology for Stem Cell-Mediated Distraction Osteogenesis

Distraction osteogenesis is a clinically established technique for lengthening, molding and shaping bone by new bone formation. The experimental evaluation of this expensive and time-consuming treatment is of high impact for better understanding of tissue engineering but mainly relies on a limited number of histological slices. These tissue slices contain two-dimensional information comprising only about one percent of the volume of interest. In order to analyze the soft and hard tissues of the entire jaw of a single rat in a multimodal assessment, we combined micro computed tomography (µCT) with histology. The µCT data acquired before and after decalcification were registered to determine the impact of decalcification on local tissue shrinkage. Identification of the location of the H&E-stained specimen within the synchrotron radiation-based µCT data collected after decalcification was achieved via non-rigid slice-to-volume registration. The resulting bi- and tri-variate histograms were divided into clusters related to anatomical features from bone and soft tissues, which allowed for a comparison of the approaches and resulted in the hypothesis that the combination of laboratory-based µCT before decalcification, synchrotron radiation-based µCT after decalcification and histology with hematoxylin-and-eosin staining could be used to discriminate between different types of collagen, key components of new bone formation.

Dynamic tracking of scaphoid, lunate, and capitate carpal bones using four-dimensional MRI

A preliminary exploration of technical methodology for dynamic analysis of scaphoid, capitate, and lunate during unconstrained movements is performed in this study. A heavily accelerated and fat-saturated 3D Cartesian MRI acquisition was used to capture temporal frames of the unconstrained moving wrist of 5 healthy subjects. A slab-to-volume point-cloud based registration was then utilized to register the moving volumes to a high-resolution image volume collected at a neutral resting position. Comprehensive in-silico error analyses for different acquisition parameter settings were performed to evaluate the performance limits of several dynamic metrics derived from the registration parameters. Computational analysis suggested that sufficient volume coverage for the dynamic acquisitions was reached when collecting 12 slice-encodes at 2.5mm resolution, which yielded a temporal resolution of and 2.6 seconds per volumetric frame. These acquisition parameters resulted in total in-silico errors of 1.9°±1.8° and 3°±4.6° in derived principal rotation angles within ulnar-radial deviation and flexion-extension motion, respectively. Rotation components of the carpal bones in the radius coordinate system were calculated and found to be consistent with earlier 4D-CT studies. Temporal metric profiles derived from ulnar-radial deviation motion demonstrated better performance than those derived from flexion/extension movements. Future work will continue to explore the use of these methods in deriving more complex dynamic metrics and their application to subjects with symptomatic carpal dysfunction.

Reattachable fiducial skin marker for automatic multimodality registration

Purpose

Fusing image information has become increasingly important for optimal diagnosis and treatment of the patient. Despite intensive research towards markerless registration approaches, fiducial marker-based methods remain the default choice for a wide range of applications in clinical practice. However, as especially non-invasive markers cannot be positioned reproducibly in the same pose on the patient, pre-interventional imaging has to be performed immediately before the intervention for fiducial marker-based registrations.

Methods

We propose a new non-invasive, reattachable fiducial skin marker concept for multi-modal registration approaches including the use of electromagnetic or optical tracking technologies. We furthermore describe a robust, automatic fiducial marker localization algorithm for computed tomography (CT) and magnetic resonance imaging (MRI) images. Localization of the new fiducial marker has been assessed for different marker configurations using both CT and MRI. Furthermore, we applied the marker in an abdominal phantom study. For this, we attached the marker at three poses to the phantom, registered ten segmented targets of the phantom’s CT image to live ultrasound images and determined the target registration error (TRE) for each target and each marker pose.

Results

Reattachment of the marker was possible with a mean precision of 0.02 mm ± 0.01 mm. Our algorithm successfully localized the marker automatically in all ($$n=201$$ n = 201 ) evaluated CT/MRI images. Depending on the marker pose, the mean ($$n=10$$ n = 10 ) TRE of the abdominal phantom study ranged from 1.51 ± 0.75 mm to 4.65 ± 1.22 mm.

Conclusions

The non-invasive, reattachable skin marker concept allows reproducible positioning of the marker and automatic localization in different imaging modalities. The low TREs indicate the potential applicability of the marker concept for clinical interventions, such as the puncture of abdominal lesions, where current image-based registration approaches still lack robustness and existing marker-based methods are often impractical.

Registration of computed tomography images of a lung infected with COVID-19 based in the new meta-heuristic algorithm HPSGWO

Computed tomography (CT) helps the radiologist in the rapid and correct detection of a person infected with the coronavirus disease 2019 (COVID-19), and this by showing the presence of the ground-glass opacity in the lung of with the virus. Tracking the evolution of the spread of the ground-glass opacity (GGO) in the lung of the person infected with the virus needs to study more than one image in different times. The various CT images must be registration to identify the evolution of the ground glass in the lung and to facilitate the study and identification of the virus. Due to the process of registration images is essentially an improvement problem, we present in this paper a new HPSGWO algorithm for registration CT images of a lung infected with the COVID-19. This algorithm is a hybridization of the two algorithms Particle swarm optimization (PSO) and Grey wolf optimizer (GWO). The simulation results obtained after applying the algorithm to the test images show that the proposed approach achieved high-precision and robust registration compared to other methods such as GWO, PSO, Firefly Algorithm (FA), and Crow Searcha Algorithms (CSA).

Virtual histology of an entire mouse brain from formalin fixation to paraffin embedding. Part 1: Data acquisition, anatomical feature segmentation, tracking global volume and density changes

BACKGROUND

Micrometer-resolution neuroimaging with gold-standard conventional histology requires tissue fixation and embedding. The exchange of solvents for the creation of sectionable paraffin blocks modifies tissue density and generates non-uniform brain shrinkage.

NEW METHOD

We employed synchrotron radiation-based X-ray microtomography for slicing- and label-free virtual histology of the mouse brain at different stages of the standard preparation protocol from formalin fixation via ascending ethanol solutions and xylene to paraffin embedding. Segmentation of anatomical regions allowed us to quantify non-uniform tissue shrinkage. Global and local changes in X-ray absorption gave insight into contrast enhancement for virtual histology.

RESULTS

The volume of the entire mouse brain was 60%, 56%, and 40% of that in formalin for, respectively, 100% ethanol, xylene, and paraffin. The volume changes of anatomical regions such as the hippocampus, anterior commissure, and ventricles differ from the global volume change. X-ray absorption of the full brain decreased, while local absorption differences increased, resulting in enhanced contrast for virtual histology. These trends were also observed with laboratory microtomography measurements.

COMPARISON WITH EXISTING METHODS

Microtomography provided sub-10 μm spatial resolution with sufficient density resolution to resolve anatomical structures at each step of the embedding protocol. The spatial resolution of conventional computed tomography and magnetic resonance microscopy is an order of magnitude lower and both do not match the contrast of microtomography over the entire embedding protocol. Unlike feature-to-feature or total volume measurements, our approach allows for calculation of volume change based on segmentation.

CONCLUSION

We present isotropic micrometer-resolution imaging to quantify morphology and composition changes in a mouse brain during the standard histological preparation. The proposed method can be employed to identify the most appropriate embedding medium for anatomical feature visualization, to reveal the basis for the dramatic X-ray contrast enhancement observed in numerous embedded tissues, and to quantify morphological changes during tissue fixation and embedding.

Immunofluorescence-guided segmentation of three-dimensional features in micro-computed tomography datasets of human lung tissue

Micro-computed tomography (µCT) provides non-destructive three-dimensional (3D) imaging of soft tissue microstructures. Specific features in µCT images can be identified using correlated two-dimensional (2D) histology images allowing manual segmentation. However, this is very time-consuming and requires specialist knowledge of the tissue and imaging modalities involved. Using a custom-designed µCT system optimized for imaging unstained formalin-fixed paraffin-embedded soft tissues, we imaged human lung tissue at isotropic voxel sizes less than 10 µm. Tissue sections were stained with haematoxylin and eosin or cytokeratin 18 in columnar airway epithelial cells using immunofluorescence (IF), as an exemplar of this workflow. Novel utilization of tissue autofluorescence allowed automatic alignment of 2D microscopy images to the 3D µCT data using scripted co-registration and automated image warping algorithms. Warped IF images, which were accurately aligned with the µCT datasets, allowed 3D segmentation of immunoreactive tissue microstructures in the human lung. Blood vessels were segmented semi-automatically using the co-registered µCT datasets. Correlating 2D IF and 3D µCT data enables accurate identification, localization and segmentation of features in fixed soft lung tissue. Our novel correlative imaging workflow provides faster and more automated 3D segmentation of µCT datasets. This is applicable to the huge range of formalin-fixed paraffin-embedded tissues held in biobanks and archives.

From Picoscale Pathology to Decascale Disease: Image Registration with a Scattering Transform and Varifolds for Manipulating Multiscale Data

Advances in neuroimaging have yielded extensive variety in the scale and type of data available. Effective integration of such data promises deeper understanding of anatomy and disease-with consequences for both diagnosis and treatment. Often catered to particular datatypes or scales, current computational tools and mathematical frameworks remain inadequate for simultaneously registering these multiple modes of "images" and statistically analyzing the ensuing menagerie of data. Here, we present (1) a registration algorithm using a "scattering transform" to align high and low resolution images and (2) a varifold-based modeling framework to compute 3D spatial statistics of multiscale data. We use our methods to quantify microscopic tau pathology across macroscopic 3D regions of the medial temporal lobe to address a major challenge in the diagnosis of Alzheimer's Disease-the reliance on invasive methods to detect microscopic pathology.

An Unsupervised Learning-Based Multi-Organ Registration Method for 3D Abdominal CT Images

Medical image registration is an essential technique to achieve spatial consistency geometric positions of different medical images obtained from single- or multi-sensor, such as computed tomography (CT), magnetic resonance (MR), and ultrasound (US) images. In this paper, an improved unsupervised learning-based framework is proposed for multi-organ registration on 3D abdominal CT images. First, the explored coarse-to-fine recursive cascaded network (RCN) modules are embedded into a basic U-net framework to achieve more accurate multi-organ registration results from 3D abdominal CT images. Then, a topology-preserving loss is added in the total loss function to avoid a distortion of the predicted transformation field. Four public databases are selected to validate the registration performances of the proposed method. The experimental results show that the proposed method is superior to some existing traditional and deep learning-based methods and is promising to meet the real-time and high-precision clinical registration requirements of 3D abdominal CT images.

An Abdominal Registration Technology for Integration of Nanomaterial Imaging-Aided Diagnosis and Treatment

Image registration technology is a key technology used in the process of nanomaterial imaging-aided diagnosis and targeted therapy effect monitoring for abdominal diseases. Recently, the deep-learning based methods have been increasingly used for large-scale medical image registration, because their iteration is much less than those of traditional ones. In this paper, a coarse-to-fine unsupervised learning-based three-dimensional (3D) abdominal CT image registration method is presented. Firstly, an affine transformation was used as an initial step to deal with large deformation between two images. Secondly, an unsupervised total loss function containing similarity, smoothness, and topology preservation measures was proposed to achieve better registration performances during convolutional neural network (CNN) training and testing. The experimental results demonstrated that the proposed method severally obtains the average MSE, PSNR, and SSIM values of 0.0055, 22.7950, and 0.8241, which outperformed some existing traditional and unsupervised learning-based methods. Moreover, our method can register 3D abdominal CT images with shortest time and is expected to become a real-time method for clinical application.

Segmented simultaneous multi-slice diffusion-weighted imaging with navigated 3D rigid motion correction

PURPOSE

To improve the robustness of diffusion-weighted imaging (DWI) data acquired with segmented simultaneous multi-slice (SMS) echo-planar imaging (EPI) against in-plane and through-plane rigid motion.

THEORY AND METHODS

The proposed algorithm incorporates a 3D rigid motion correction and wavelet denoising into the image reconstruction of segmented SMS-EPI diffusion data. Low-resolution navigators are used to estimate shot-specific diffusion phase corruptions and 3D rigid motion parameters through SMS-to-volume registration. The shot-wise rigid motion and phase parameters are integrated into a SENSE-based full-volume reconstruction for each diffusion direction. The algorithm is compared to a navigated SMS reconstruction without gross motion correction in simulations and in vivo studies with four-fold interleaved 3-SMS diffusion tensor acquisitions.

RESULTS

Simulations demonstrate high fidelity was achieved in the SMS-to-volume registration, with submillimeter registration errors and improved image reconstruction quality. In vivo experiments validate successful artifact reduction in 3D motion-compromised in vivo scans with a temporal motion resolution of approximately 0.3 s.

CONCLUSION

This work demonstrates the feasibility of retrospective 3D rigid motion correction from shot navigators for segmented SMS DWI.

A novel approach for computerized quantitative image analysis of proximal femur bone shape deformities based on the hip joint symmetry

As a result of most of the bone disorders seen in hip joints, shape deformities occur in the structural form of the hip joint components. Image-based quantitative analysis and assessment of these deformities in bone shapes are very important for the evaluation, treatment, and prognosis of the various hip joint bone disorders. In this article, a novel approach for the image-based computerized quantitative analysis of proximal femur shape deformities is presented. In the proposed approach, shape deformities of the pathological proximal femurs were quantified over the contralateral healthy proximal femur shape structure of the same patient in 2D by taking the hip joint symmetry property of human anatomy into consideration. It is based on the idea that if the right and left proximal femurs in bilateral hip joints are highly symmetrical and also if one of the proximal femurs is healthy and the contralateral one is pathological, the non-overlapping bone shape regions can represent the deformities in pathological proximal femurs when both proximal femurs are registered to overlap each other. In the methodological process of the proposed study, a set of image preprocessing operations was primarily performed on the raw magnetic resonance imaging (MRI) data. Then, the segmented proximal femurs in bilateral hip joint images were automatically aligned with the Iterative Closest Point (ICP) rigid registration method. Following the registration, a set of image postprocessing operations was performed on the images of proximal femurs aligned. In the quantification phase, the bone shape deformities in pathological proximal femurs were quantified simply in terms of the mismatching area in 2D by measuring a shape variation index representing the total bone shape deformity ratio. To evaluate the proposed quantitative shape analysis approach, bilateral hip joints in a total of 13 coronal MRI sections of 13 patients with Legg-Calve-Perthes disease (LCPD) were used. Experimental studies have shown that the proposed approach has quite promising results in the quantitative representation of the pathological proximal femur shape deformities. Furthermore, consistent results have been observed for the Waldenström classification stages of the disease. The shape deformity ratios in pathological proximal femurs were quantified as 9.44% (±1.40), 18.38% (±6.30), 24.73% (±12.42), and 27.66% (±10.41), respectively for the Initial, Fragmentation, Reossification, and Remodelling stages of LCPD with the quantification error rates of 0.29% (±0.16), 0.58% (±0.71), 1.12% (±0.82), and 0.80% (±0.98). Additionally, a mean error rate of 0.65% (±0.68) was observed for the quantified shape deformity ratios of all samples.

Learning to map 2D ultrasound images into 3D space with minimal human annotation

In fetal neurosonography, aligning two-dimensional (2D) ultrasound scans to their corresponding plane in the three-dimensional (3D) space remains a challenging task. In this paper, we propose a convolutional neural network that predicts the position of 2D ultrasound fetal brain scans in 3D atlas space. Instead of purely supervised learning that requires heavy annotations for each 2D scan, we train the model by sampling 2D slices from 3D fetal brain volumes, and target the model to predict the inverse of the sampling process, resembling the idea of self-supervised learning. We propose a model that takes a set of images as input, and learns to compare them in pairs. The pairwise comparison is weighted by the attention module based on its contribution to the prediction, which is learnt implicitly during training. The feature representation for each image is thus computed by incorporating the relative position information to all the other images in the set, and is later used for the final prediction. We benchmark our model on 2D slices sampled from 3D fetal brain volumes at 18-22 weeks' gestational age. Using three evaluation metrics, namely, Euclidean distance, plane angles and normalized cross correlation, which account for both the geometric and appearance discrepancy between the ground-truth and prediction, in all these metrics, our model outperforms a baseline model by as much as 23%, when the number of input images increases. We further demonstrate that our model generalizes to (i) real 2D standard transthalamic plane images, achieving comparable performance as human annotations, as well as (ii) video sequences of 2D freehand fetal brain scans.

Bi-channel image registration and deep-learning segmentation (BIRDS) for efficient, versatile 3D mapping of mouse brain

We have developed an open-source software called bi-channel image registration and deep-learning segmentation (BIRDS) for the mapping and analysis of 3D microscopy data and applied this to the mouse brain. The BIRDS pipeline includes image preprocessing, bi-channel registration, automatic annotation, creation of a 3D digital frame, high-resolution visualization, and expandable quantitative analysis. This new bi-channel registration algorithm is adaptive to various types of whole-brain data from different microscopy platforms and shows dramatically improved registration accuracy. Additionally, as this platform combines registration with neural networks, its improved function relative to the other platforms lies in the fact that the registration procedure can readily provide training data for network construction, while the trained neural network can efficiently segment-incomplete/defective brain data that is otherwise difficult to register. Our software is thus optimized to enable either minute-timescale registration-based segmentation of cross-modality, whole-brain datasets or real-time inference-based image segmentation of various brain regions of interest. Jobs can be easily submitted and implemented via a Fiji plugin that can be adapted to most computing environments.

Deep Predictive Motion Tracking in Magnetic Resonance Imaging: Application to Fetal Imaging

Fetal magnetic resonance imaging (MRI) is challenged by uncontrollable, large, and irregular fetal movements. It is, therefore, performed through visual monitoring of fetal motion and repeated acquisitions to ensure diagnostic-quality images are acquired. Nevertheless, visual monitoring of fetal motion based on displayed slices, and navigation at the level of stacks-of-slices is inefficient. The current process is highly operator-dependent, increases scanner usage and cost, and significantly increases the length of fetal MRI scans which makes them hard to tolerate for pregnant women. To help build automatic MRI motion tracking and navigation systems to overcome the limitations of the current process and improve fetal imaging, we have developed a new real-time image-based motion tracking method based on deep learning that learns to predict fetal motion directly from acquired images. Our method is based on a recurrent neural network, composed of spatial and temporal encoder-decoders, that infers motion parameters from anatomical features extracted from sequences of acquired slices. We compared our trained network on held-out test sets (including data with different characteristics, e.g. different fetuses scanned at different ages, and motion trajectories recorded from volunteer subjects) with networks designed for estimation as well as methods adopted to make predictions. The results show that our method outperformed alternative techniques, and achieved real-time performance with average errors of 3.5 and 8 degrees for the estimation and prediction tasks, respectively. Our real-time deep predictive motion tracking technique can be used to assess fetal movements, to guide slice acquisitions, and to build navigation systems for fetal MRI.

Fusion of Multiple-angles Intraoperative US Images and Pretreatment MR Images for USgHIFU Treatment of Uterine Fibroid: Retrospective Evaluation Based on Clinical Dataset

High-intensity focused ultrasound (HIFU) has been widely used for treatment of uterine fibroids. However, due to the limited resolution of ultrasound image in deep organs, the guidance of ultrasound-guided HIFU (USgHIFU) treatment greatly depends on clinicians' experience in US image. To address this issue, fusion of intraoperative US images and pretreatment MR images has been proposed. Contour segmentation and multiple-angles 2D US images combination are performed to obtain 3D points along the contour of uterus. Iterative closest point (ICP) algorithm based on prior knowledge is used to register these point sets. MR and US images of six treated patients are used for evaluation. The mean distance error (MDE) of our algorithm is 1.71±0.59 mm, and the average running time is 0.18 s. The results have verified the feasibility of fusion of MR images and US images for USgHIFU guidance. In addition, this method may be also potential for post-ablation evaluation with follow-up MR images.

SLIMM: Slice localization integrated MRI monitoring

Functional MRI (fMRI) is extremely challenging to perform in subjects who move because subject motion disrupts blood oxygenation level dependent (BOLD) signal measurement. It has become common to use retrospective framewise motion detection and censoring in fMRI studies to eliminate artifacts arising from motion. Data censoring results in significant loss of data and statistical power unless the data acquisition is extended to acquire more data not corrupted by motion. Acquiring more data than is necessary leads to longer than necessary scan duration, which is more expensive and may lead to additional subject non-compliance. Therefore, it is well established that real-time prospective motion monitoring is crucial to ensure data quality and reduce imaging costs. In addition, real-time monitoring of motion allows for feedback to the operator and the subject during the acquisition, to enable intervention to reduce the subject motion. The most widely used form of motion monitoring for fMRI is based on volume-to-volume registration (VVR), which quantifies motion as the misalignment between subsequent volumes. However, motion is not constrained to occur only at the boundaries of volume acquisition, but instead may occur at any time. Consequently, each slice of an fMRI acquisition may be displaced by motion, and assessment of whole volume to volume motion may be insensitive to both intra-volume and inter-volume motion that is revealed by displacement of the slices. We developed the first slice-by-slice self-navigated motion monitoring system for fMRI by developing a real-time slice-to-volume registration (SVR) algorithm. Our real-time SVR algorithm, which is the core of the system, uses a local image patch-based matching criterion along with a Levenberg-Marquardt optimizer, all accelerated via symmetric multi-processing, with interleaved and simultaneous multi-slice acquisition schemes. Extensive experimental results on real motion data demonstrated that our fast motion monitoring system, named Slice Localization Integrated MRI Monitoring (SLIMM), provides more accurate motion measurements than a VVR based approach. Therefore, SLIMM offers improved online motion monitoring which is particularly important in fMRI for challenging patient populations. Real-time motion monitoring is crucial for online data quality control and assurance, for enabling feedback to the subject and the operator to act to mitigate motion, and in adaptive acquisition strategies that aim to ensure enough data of sufficient quality is acquired without acquiring excess data.

A multimodal computational pipeline for 3D histology of the human brain

Ex vivo imaging enables analysis of the human brain at a level of detail that is not possible in vivo with MRI. In particular, histology can be used to study brain tissue at the microscopic level, using a wide array of different stains that highlight different microanatomical features. Complementing MRI with histology has important applications in ex vivo atlas building and in modeling the link between microstructure and macroscopic MR signal. However, histology requires sectioning tissue, hence distorting its 3D structure, particularly in larger human samples. Here, we present an open-source computational pipeline to produce 3D consistent histology reconstructions of the human brain. The pipeline relies on a volumetric MRI scan that serves as undistorted reference, and on an intermediate imaging modality (blockface photography) that bridges the gap between MRI and histology. We present results on 3D histology reconstruction of whole human hemispheres from two donors.

Image Matching from Handcrafted to Deep Features: A Survey

As a fundamental and critical task in various visual applications, image matching can identify then correspond the same or similar structure/content from two or more images. Over the past decades, growing amount and diversity of methods have been proposed for image matching, particularly with the development of deep learning techniques over the recent years. However, it may leave several open questions about which method would be a suitable choice for specific applications with respect to different scenarios and task requirements and how to design better image matching methods with superior performance in accuracy, robustness and efficiency. This encourages us to conduct a comprehensive and systematic review and analysis for those classical and latest techniques. Following the feature-based image matching pipeline, we first introduce feature detection, description, and matching techniques from handcrafted methods to trainable ones and provide an analysis of the development of these methods in theory and practice. Secondly, we briefly introduce several typical image matching-based applications for a comprehensive understanding of the significance of image matching. In addition, we also provide a comprehensive and objective comparison of these classical and latest techniques through extensive experiments on representative datasets. Finally, we conclude with the current status of image matching technologies and deliver insightful discussions and prospects for future works. This survey can serve as a reference for (but not limited to) researchers and engineers in image matching and related fields.

MRI restoration using edge-guided adversarial learning

Magnetic resonance imaging (MRI) images acquired as multislice two-dimensional (2D) images present challenges when reformatted in orthogonal planes due to sparser sampling in the through-plane direction. Restoring the "missing" through-plane slices, or regions of an MRI image damaged by acquisition artifacts can be modeled as an image imputation task. In this work, we consider the damaged image data or missing through-plane slices as image masks and proposed an edge-guided generative adversarial network to restore brain MRI images. Inspired by the procedure of image inpainting, our proposed method decouples image repair into two stages: edge connection and contrast completion, both of which used general adversarial networks (GAN). We trained and tested on a dataset from the Human Connectome Project to test the application of our method for thick slice imputation, while we tested the artifact correction on clinical data and simulated datasets. Our Edge-Guided GAN had superior PSNR, SSIM, conspicuity and signal texture compared to traditional imputation tools, the Context Encoder and the Densely Connected Super Resolution Network with GAN (DCSRN-GAN). The proposed network may improve utilization of clinical 2D scans for 3D atlas generation and big-data comparative studies of brain morphometry.

Automatic Histogram Specification for Glioma Grading Using Multicenter Data

Multicenter sharing is an effective method to increase the data size for glioma research, but the data inconsistency among different institutions hindered the efficiency. This paper proposes a histogram specification with automatic selection of reference frames for magnetic resonance images to alleviate this problem (HSASR). The selection of reference frames is automatically performed by an optimized grid search strategy with coarse and fine search. The search range is firstly narrowed by coarse search of intraglioma samples, and then the suitable reference frame in histogram is selected by fine search within the sample selected by coarse search. Validation experiments are conducted on two datasets GliomaHPPH2018 and BraTS2017 to perform glioma grading. The results demonstrate the high performance of the proposed method. On the mixed dataset, the average AUC, accuracy, sensitivity, and specificity are 0.9786, 94.13%, 94.64%, and 93.00%, respectively. It is about 15% higher on all indicators compared with those without HSASR and has a slight advantage over the result of a manually selected reference frame by radiologists. Results show that our methods can effectively alleviate multicenter data inconsistencies and lift the performance of the prediction model.

M3VR-A multi-stage, multi-resolution, and multi-volumes-of-interest volume registration method applied to 3D endovaginal ultrasound

Heterogeneity of echo-texture and lack of sharply delineated tissue boundaries in diagnostic ultrasound images make three-dimensional (3D) registration challenging, especially when the volumes to be registered are considerably different due to local changes. We implemented a novel computational method that optimally registers volumetric ultrasound image data containing significant and local anatomical differences. It is A Multi-stage, Multi-resolution, and Multi-volumes-of-interest Volume Registration Method. A single region registration is optimized first for a close initial alignment to avoid convergence to a locally optimal solution. Multiple sub-volumes of interest can then be selected as target alignment regions to achieve confident consistency across the volume. Finally, a multi-resolution rigid registration is performed on these sub-volumes associated with different weights in the cost function. We applied the method on 3D endovaginal ultrasound image data acquired from patients during biopsy procedure of the pelvic floor muscle. Systematic assessment of our proposed method through cross validation demonstrated its accuracy and robustness. The algorithm can also be applied on medical imaging data of other modalities for which the traditional rigid registration methods would fail.

Resting State fMRI: Going Through the Motions

Resting state functional magnetic resonance imaging (rs-fMRI) has become an indispensable tool in neuroscience research. Despite this, rs-fMRI signals are easily contaminated by artifacts arising from movement of the head during data collection. The artifacts can be problematic even for motions on the millimeter scale, with complex spatiotemporal properties that can lead to substantial errors in functional connectivity estimates. Effective correction methods must be employed, therefore, to distinguish true functional networks from motion-related noise. Research over the last three decades has produced numerous correction methods, many of which must be applied in combination to achieve satisfactory data quality. Subject instruction, training, and mild restraints are helpful at the outset, but usually insufficient. Improvements come from applying multiple motion correction algorithms retrospectively after rs-fMRI data are collected, although residual artifacts can still remain in cases of elevated motion, which are especially prevalent in patient populations. Although not commonly adopted at present, “real-time” correction methods are emerging that can be combined with retrospective methods and that promise better correction and increased rs-fMRI signal sensitivity. While the search for the ideal motion correction protocol continues, rs-fMRI research will benefit from good disclosure practices, such as: (1) reporting motion-related quality control metrics to provide better comparison between studies; and (2) including motion covariates in group-level analyses to limit the extent of motion-related confounds when studying group differences.

The deformable most-likely-point paradigm

In this paper, we present three deformable registration algorithms designed within a paradigm that uses 3D statistical shape models to accomplish two tasks simultaneously: 1) register point features from previously unseen data to a statistically derived shape (e.g., mean shape), and 2) deform the statistically derived shape to estimate the shape represented by the point features. This paradigm, called the deformable most-likely-point paradigm, is motivated by the idea that generative shape models built from available data can be used to estimate previously unseen data. We developed three deformable registration algorithms within this paradigm using statistical shape models built from reliably segmented objects with correspondences. Results from several experiments show that our algorithms produce accurate registrations and reconstructions in a variety of applications with errors up to CT resolution on medical datasets. Our code is available at https://github.com/AyushiSinha/cisstICP.

Two-dimensional ultrasound-computed tomography image registration for monitoring percutaneous hepatic intervention

PURPOSE

Deformable registration of ultrasound (US) and contrast enhanced computed tomography (CECT) images are essential for quantitative comparison of ablation boundaries and dimensions determined using these modalities. This comparison is essential as stiffness-based imaging using US has become popular and offers a nonionizing and cost-effective imaging modality for monitoring minimally invasive microwave ablation procedures. A sensible manual registration method is presented that performs the required CT-US image registration.

METHODS

The two-dimensional (2D) virtual CT image plane that corresponds to the clinical US B-mode was obtained by "virtually slicing" the 3D CT volume along the plane containing non-anatomical landmarks, namely points along the microwave ablation antenna. The initial slice plane was generated using the vector acquired by rotating the normal vector of the transverse (i.e., xz) plane along the angle subtended by the antenna. This plane was then further rotated along the ablation antenna and shifted along with the direction of normal vector to obtain similar anatomical structures, such as the liver surface and vasculature that is visualized on both the CT virtual slice and US B-mode images on 20 patients. Finally, an affine transformation was estimated using anatomic and non-anatomic landmarks to account for distortion between the colocated CT virtual slice and US B-mode image resulting in a final registered CT virtual slice. Registration accuracy was measured by estimating the Euclidean distance between corresponding registered points on CT and US B-mode images.

RESULTS

Mean and SD of the affine transformed registration error was 1.85 ± 2.14 (mm), computed from 20 coregistered data sets.

CONCLUSIONS

Our results demonstrate the ability to obtain 2D virtual CT slices that are registered to clinical US B-mode images. The use of both anatomical and non-anatomical landmarks result in accurate registration useful for validating ablative margins and comparison to electrode displacement elastography based images.

Automatic 3D Nonlinear Registration of Mass Spectrometry Imaging and Magnetic Resonance Imaging Data

Multimodal integration between mass spectrometry imaging (MSI) and radiology-established modalities such as magnetic resonance imaging (MRI) would allow the investigations of key questions in complex biological systems such as the central nervous system. Such integration would provide complementary multiscale data to bridge the gap between molecular and anatomical phenotypes, potentially revealing new insights into molecular mechanisms underlying anatomical pathologies presented on MRI. Automatic coregistration between 3D MSI/MRI is a computationally challenging process due to dimensional complexity, MSI data sparsity, lack of direct spatial-correspondences, and nonlinear tissue deformation. Here, we present a new computational approach based on stochastic neighbor embedding to nonlinearly align 3D MSI to MRI data, identify and reconstruct biologically relevant molecular patterns in 3D, and fuse the MSI datacube to the MRI space. We demonstrate our method using multimodal high-spectral resolution matrix-assisted laser desorption ionization (MALDI) 9.4 T MSI and 7 T in vivo MRI data, acquired from a patient-derived, xenograft mouse brain model of glioblastoma following administration of the EGFR inhibitor drug of Erlotinib. Results show the distribution of some identified molecular ions of the EGFR inhibitor erlotinib, a phosphatidylcholine lipid, and cholesterol, which were reconstructed in 3D and mapped to the MRI space. The registration quality was evaluated on two normal mouse brains using the Dice coefficient for the regions of brainstem, hippocampus, and cortex. The method is generic and can therefore be applied to hyperspectral images from different mass spectrometers and integrated with other established in vivo imaging modalities such as computed tomography (CT) and positron emission tomography (PET).