Automatic ultrasound-MRI registration for neurosurgery using the 2D and 3D LC(2) Metric

Automatic rigid image Fusion of preoperative MR and intraoperative US acquired after craniotomy

BACKGROUND

Neuronavigation of preoperative MRI is limited by several errors. Intraoperative ultrasound (iUS) with navigated probes that provide automatic superposition of pre-operative MRI and iUS and three-dimensional iUS reconstruction may overcome some of these limitations. Aim of the present study is to verify the accuracy of an automatic MRI - iUS fusion algorithm to improve MR-based neuronavigation accuracy.

METHODS

An algorithm using Linear Correlation of Linear Combination (LC2)-based similarity metric has been retrospectively evaluated for twelve datasets acquired in patients with brain tumor. A series of landmarks were defined both in MRI and iUS scans. The Target Registration Error (TRE) was determined for each pair of landmarks before and after the automatic Rigid Image Fusion (RIF). The algorithm has been tested on two conditions of the initial image alignment: registration-based fusion (RBF), as given by the navigated ultrasound probe, and different simulated course alignments during convergence test.

RESULTS

Except for one case RIF was successfully applied in all patients considering the RBF as initial alignment. Here, mean TRE after RBF was significantly reduced from 4.03 (± 1.40) mm to (2.08 ± 0.96 mm) (p = 0.002), after RIF. For convergence test, the mean TRE value after initial perturbations was 8.82 (± 0.23) mm which has been reduced to a mean TRE of 2.64 (± 1.20) mm after RIF (p < 0.001).

CONCLUSIONS

The integration of an automatic image fusion method for co-registration of pre-operative MRI and iUS data may improve the accuracy in MR-based neuronavigation.

Multimodal registration of ultrasound and MR images using weighted self-similarity structure vector

PROPOSE

Multimodal registration of 2D Ultrasound (US) and 3D Magnetic Resonance (MR) for fusion navigation can improve the intraoperative detection accuracy of lesion. However, multimodal registration remains a challenge because of the poor US image quality. In the study, a weighted self-similarity structure vector (WSSV) is proposed to registrate multimodal images.

METHOD

The self-similarity structure vector utilizes the normalized distance of symmetrically located patches in the neighborhood to describe the local structure information. The texture weights are extracted using the local standard deviation to reduce the speckle interference in the US images. The multimodal similarity metric is constructed by combining a self-similarity structure vector with a texture weight map.

RESULTS

Experiments were performed on US and MR images of the liver from 88 groups of data including 8 patients and 80 simulated samples. The average target registration error was reduced from 14.91 ± 3.86 mm to 4.95 ± 2.23 mm using the WSSV-based method.

CONCLUSIONS

The experimental results show that the WSSV-based registration method could robustly align the US and MR images of the liver. With further acceleration, the registration framework can be potentially applied in time-sensitive clinical settings, such as US-MR image registration in image-guided surgery.

Insights into Infusion-Based Targeted Drug Delivery in the Brain: Perspectives, Challenges and Opportunities

Targeted drug delivery in the brain is instrumental in the treatment of lethal brain diseases, such as glioblastoma multiforme, the most aggressive primary central nervous system tumour in adults. Infusion-based drug delivery techniques, which directly administer to the tissue for local treatment, as in convection-enhanced delivery (CED), provide an important opportunity; however, poor understanding of the pressure-driven drug transport mechanisms in the brain has hindered its ultimate success in clinical applications. In this review, we focus on the biomechanical and biochemical aspects of infusion-based targeted drug delivery in the brain and look into the underlying molecular level mechanisms. We discuss recent advances and challenges in the complementary field of medical robotics and its use in targeted drug delivery in the brain. A critical overview of current research in these areas and their clinical implications is provided. This review delivers new ideas and perspectives for further studies of targeted drug delivery in the brain.

Symmetry-based representation for registration of multimodal images

We propose a new two-dimensional structural representation method for registration of multimodal images by using the local structural symmetry of images, which is similar at different modalities. The symmetry is measured in various orientations and the best is mapped and used for the representation image. The optimum performance is obtained when using only two different orientations, which is called binary dominant symmetry representation (BDSR). This representation is highly robust to noise and intensity non-uniformity. We also propose a new objective function based on L2 distance with low sensitivity to the overlapping region. Then, five different meta-heuristic algorithms are comparatively applied. Two of them have been used for the first time on image registration. BDSR remarkably outperforms the previous successful representations, such as entropy images, self-similarity context, and modality-independent local binary pattern, as well as mutual information-based registration, in terms of success rate, runtime, convergence error, and representation construction.

Modality-agnostic self-supervised deep feature learning and fast instance optimisation for multimodal fusion in ultrasound-guided interventions

BACKGROUND AND OBJECTIVE

Fast and robust alignment of pre-operative MRI planning scans to intra-operative ultrasound is an important aspect for automatically supporting image-guided interventions. Thus far, learning-based approaches have failed to tackle the intertwined objectives of fast inference computation time and robustness to unexpectedly large motion and misalignment. In this work, we propose a novel method that decouples deep feature learning and the computation of long ranging local displacement probability maps from fast and robust global transformation prediction.

METHODS

In our approach, we firstly train a convolutional neural network (CNN) to extract modality-agnostic features with sub-second computation times for both 3D volumes during inference. Using sparsity-based network weight pruning, the model complexity and computation times can be substantially reduced. Based on these features, a large discretized search range of 3D motion vectors is explored to compute a probabilistic displacement map for each control point. These 3D probability maps are employed in our newly proposed, computationally efficient, instance optimisation that robustly estimates the most likely globally linear transformation that best reflects the local displacement beliefs subject to outlier rejection.

RESULTS

Our experimental validation demonstrates state-of-the-art accuracy on the challenging CuRIOUS dataset with average target registration errors of 2.50 mm, model size of only 1.2 MByte and run times of approx. 3 seconds for a full 3D multimodal registration.

CONCLUSION

We show that a significant improvement in accuracy and robustness can be gained with instance optimisation and our fast self-supervised deep learning model can achieve state-of-the-art accuracy on challenging registration task in only 3 seconds.

Intraoperative Computed Tomography and Finite Element Modelling for Multimodal Image Fusion in Brain Surgery

BACKGROUND

intraoperative computer tomography (iCT) and advanced image fusion algorithms could improve the management of brainshift and the navigation accuracy.

OBJECTIVE

To evaluate the performance of an iCT-based fusion algorithm using clinical data.

METHODS

Ten patients with brain tumors were enrolled; preoperative MRI was acquired. The iCT was applied at the end of microsurgical resection. Elastic image fusion of the preoperative MRI to iCT data was performed by deformable fusion employing a biomechanical simulation based on a finite element model. Fusion accuracy was evaluated: the target registration error (TRE, mm) was measured for rigid and elastic fusion (Rf and Ef) and anatomical landmark pairs were divided into test and control structures according to distinct involvement by the brainshift. Intraoperative points describing the stereotactic position of the brain were also acquired and a qualitative evaluation of the adaptive morphing of the preoperative MRI was performed by 5 observers.

RESULTS

The mean TRE for control and test structures with Rf was 1.81 ± 1.52 and 5.53 ± 2.46 mm, respectively. No significant change was observed applying Ef to control structures; the test structures showed reduced TRE values of 3.34 ± 2.10 mm after Ef (P < .001). A 32% average gain (range 9%-54%) in accuracy of image registration was recorded. The morphed MRI showed robust matching with iCT scans and intraoperative stereotactic points.

CONCLUSIONS

The evaluated method increased the registration accuracy of preoperative MRI and iCT data. The iCT-based non-linear morphing of the preoperative MRI can potentially enhance the consistency of neuronavigation intraoperatively.

CBCT-based navigation system for open liver surgery: Accurate guidance toward mobile and deformable targets with a semi-rigid organ approximation and electromagnetic tracking of the liver

Purpose

The surgical navigation system that provides guidance throughout the surgery can facilitate safer and more radical liver resections, but such a system should also be able to handle organ motion. This work investigates the accuracy of intraoperative surgical guidance during open liver resection, with a semi‐rigid organ approximation and electromagnetic tracking of the target area.

Methods

The suggested navigation technique incorporates a preoperative 3D liver model based on diagnostic 4D MRI scan, intraoperative contrast‐enhanced CBCT imaging and electromagnetic (EM) tracking of the liver surface, as well as surgical instruments, by means of six degrees‐of‐freedom micro‐EM sensors.

Results

The system was evaluated during surgeries with 35 patients and resulted in an accurate and intuitive real‐time visualization of liver anatomy and tumor's location, confirmed by intraoperative checks on visible anatomical landmarks. Based on accuracy measurements verified by intraoperative CBCT, the system’s average accuracy was 4.0 ± 3.0 mm, while the total surgical delay due to navigation stayed below 20 min.

Conclusions

The electromagnetic navigation system for open liver surgery developed in this work allows for accurate localization of liver lesions and critical anatomical structures surrounding the resection area, even when the liver was manipulated. However, further clinically integrating the method requires shortening the guidance‐related surgical delay, which can be achieved by shifting to faster intraoperative imaging like ultrasound. Our approach is adaptable to navigation on other mobile and deformable organs, and therefore may benefit various clinical applications.

The Essential Role of Open Data and Software for the Future of Ultrasound-Based Neuronavigation

With the recent developments in machine learning and modern graphics processing units (GPUs), there is a marked shift in the way intra-operative ultrasound (iUS) images can be processed and presented during surgery. Real-time processing of images to highlight important anatomical structures combined with in-situ display, has the potential to greatly facilitate the acquisition and interpretation of iUS images when guiding an operation. In order to take full advantage of the recent advances in machine learning, large amounts of high-quality annotated training data are necessary to develop and validate the algorithms. To ensure efficient collection of a sufficient number of patient images and external validity of the models, training data should be collected at several centers by different neurosurgeons, and stored in a standard format directly compatible with the most commonly used machine learning toolkits and libraries. In this paper, we argue that such effort to collect and organize large-scale multi-center datasets should be based on common open source software and databases. We first describe the development of existing open-source ultrasound based neuronavigation systems and how these systems have contributed to enhanced neurosurgical guidance over the last 15 years. We review the impact of the large number of projects worldwide that have benefited from the publicly available datasets “Brain Images of Tumors for Evaluation” (BITE) and “Retrospective evaluation of Cerebral Tumors” (RESECT) that include MR and US data from brain tumor cases. We also describe the need for continuous data collection and how this effort can be organized through the use of a well-adapted and user-friendly open-source software platform that integrates both continually improved guidance and automated data collection functionalities.

HFUS Imaging of the Cochlea: A Feasibility Study for Anatomical Identification by Registration with MicroCT

Cochlear implantation consists in electrically stimulating the auditory nerve by inserting an electrode array inside the cochlea, a bony structure of the inner ear. In the absence of any visual feedback, the insertion results in many cases of damages of the internal structures. This paper presents a feasibility study on intraoperative imaging and identification of cochlear structures with high-frequency ultrasound (HFUS). 6 ex-vivo guinea pig cochleae were subjected to both US and microcomputed tomography (µCT) we respectively referred as intraoperative and preoperative modalities. For each sample, registration based on simulating US from the scanner was performed to allow a precise matching between the visible structures. According to two otologists, the procedure led to a target registration error of 0.32 mm ± 0.05. Thanks to referring to a better preoperative anatomical representation, we were able to intraoperatively identify the modiolus, both scalae vestibuli and tympani and deduce the location of the basilar membrane, all of which is of great interest for cochlear implantation. Our main objective is to extend this procedure to the human case and thus provide a new tool for inner ear surgery.

Analyzing the co-localization of substantia nigra hyper-echogenicities and iron accumulation in Parkinson's disease: A multi-modal atlas study with transcranial ultrasound and MRI

•

Volumetric 3D analysis of hyper-echogenicities from transcranial ultrasound (TCS).

•

First multi-modal analysis of TCS and QSM-MRI in Parkinson's disease.

•

Computations of TCS-MRI registration and a novel multi-modal anatomical template.

•

TCS hyper-echogenicities are co-localized with QSM iron accumulations.

•

Co-localizations occur in the SNc and VTA, but nowhere else in the midbrain.

Background

Transcranial B-mode sonography (TCS) can detect hyperechogenic speckles in the area of the substantia nigra (SN) in Parkinson's disease (PD). These speckles correlate with iron accumulation in the SN tissue, but an exact volumetric localization in and around the SN is still unknown. Areas of increased iron content in brain tissue can be detected in vivo with magnetic resonance imaging, using quantitative susceptibility mapping (QSM).

Methods

In this work, we i) acquire, co-register and transform TCS and QSM imaging from a cohort of 23 PD patients and 27 healthy control subjects into a normalized atlas template space and ii) analyze and compare the 3D spatial distributions of iron accumulation in the midbrain, as detected by a signal increase (TCS+ and QSM+) in both modalities.

Results

We achieved sufficiently accurate intra-modal target registration errors (TRE<1 mm) for all MRI volumes and multi-modal TCS-MRI co-localization (TRE<4 mm) for 66.7% of TCS scans. In the caudal part of the midbrain, enlarged TCS+ and QSM+ areas were located within the SN pars compacta in PD patients in comparison to healthy controls. More cranially, overlapping TCS+ and QSM+ areas in PD subjects were found in the area of the ventral tegmental area (VTA).

Conclusion

Our findings are concordant with several QSM-based studies on iron-related alterations in the area SN pars compacta. They substantiate that TCS+ is an indicator of iron accumulation in Parkinson's disease within and in the vicinity of the SN. Furthermore, they are in favor of an involvement of the VTA and thereby the mesolimbic system in Parkinson's disease.

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.

Deformable MRI-Ultrasound registration using correlation-based attribute matching for brain shift correction: Accuracy and generality in multi-site data

Intraoperative tissue deformation, known as brain shift, decreases the benefit of using preoperative images to guide neurosurgery. Non-rigid registration of preoperative magnetic resonance (MR) to intraoperative ultrasound (iUS) has been proposed as a means to compensate for brain shift. We focus on the initial registration from MR to predurotomy iUS. We present a method that builds on previous work to address the need for accuracy and generality of MR-iUS registration algorithms in multi-site clinical data. High-dimensional texture attributes were used instead of image intensities for image registration and the standard difference-based attribute matching was replaced with correlation-based attribute matching. A strategy that deals explicitly with the large field-of-view mismatch between MR and iUS images was proposed. Key parameters were optimized across independent MR-iUS brain tumor datasets acquired at 3 institutions, with a total of 43 tumor patients and 758 reference landmarks for evaluating the accuracy of the proposed algorithm. Despite differences in imaging protocols, patient demographics and landmark distributions, the algorithm is able to reduce landmark errors prior to registration in three data sets (5.37±4.27, 4.18±1.97 and 6.18±3.38 mm, respectively) to a consistently low level (2.28±0.71, 2.08±0.37 and 2.24±0.78 mm, respectively). This algorithm was tested against 15 other algorithms and it is competitive with the state-of-the-art on multiple datasets. We show that the algorithm has one of the lowest errors in all datasets (accuracy), and this is achieved while sticking to a fixed set of parameters for multi-site data (generality). In contrast, other algorithms/tools of similar performance need per-dataset parameter tuning (high accuracy but lower generality), and those that stick to fixed parameters have larger errors or inconsistent performance (generality but not the top accuracy). Landmark errors were further characterized according to brain regions and tumor types, a topic so far missing in the literature.

A comparison of thin-plate spline deformation and finite element modeling to compensate for brain shift during tumor resection

PURPOSE

Brain shift during tumor resection can progressively invalidate the accuracy of neuronavigation systems and affect neurosurgeons' ability to achieve optimal resections. This paper compares two methods that have been presented in the literature to compensate for brain shift: a thin-plate spline deformation model and a finite element method (FEM). For this comparison, both methods are driven by identical sparse data. Specifically, both methods are driven by displacements between automatically detected and matched feature points from intraoperative 3D ultrasound (iUS). Both methods have been shown to be fast enough for intraoperative brain shift correction (Machado et al. in Int J Comput Assist Radiol Surg 13(10):1525-1538, 2018; Luo et al. in J Med Imaging (Bellingham) 4(3):035003, 2017). However, the spline method requires no preprocessing and ignores physical properties of the brain while the FEM method requires significant preprocessing and incorporates patient-specific physical and geometric constraints. The goal of this work was to explore the relative merits of these methods on recent clinical data.

METHODS

Data acquired during 19 sequential tumor resections in Brigham and Women's Hospital's Advanced Multi-modal Image-Guided Operating Suite between December 2017 and October 2018 were considered for this retrospective study. Of these, 15 cases and a total of 24 iUS to iUS image pairs met inclusion requirements. Automatic feature detection (Machado et al. in Int J Comput Assist Radiol Surg 13(10):1525-1538, 2018) was used to detect and match features in each pair of iUS images. Displacements between matched features were then used to drive both the spline model and the FEM method to compensate for brain shift between image acquisitions. The accuracies of the resultant deformation models were measured by comparing the displacements of manually identified landmarks before and after deformation.

RESULTS

The mean initial subcortical registration error between preoperative MRI and the first iUS image averaged 5.3 ± 0.75 mm. The mean subcortical brain shift, measured using displacements between manually identified landmarks in pairs of iUS images, was 2.5 ± 1.3 mm. Our results showed that FEM was able to reduce subcortical registration error by a small but statistically significant amount (from 2.46 to 2.02 mm). A large variability in the results of the spline method prevented us from demonstrating either a statistically significant reduction in subcortical registration error after applying the spline method or a statistically significant difference between the results of the two methods.

CONCLUSIONS

In this study, we observed less subcortical brain shift than has previously been reported in the literature (Frisken et al., in: Miller (ed) Biomechanics of the brain, Springer, Cham, 2019). This may be due to the fact that we separated out the initial misregistration between preoperative MRI and the first iUS image from our brain shift measurements or it may be due to modern neurosurgical practices designed to reduce brain shift, including reduced craniotomy sizes and better control of intracranial pressure with the use of mannitol and other medications. It appears that the FEM method and its use of geometric and biomechanical constraints provided more consistent brain shift correction and better correction farther from the driving feature displacements than the simple spline model. The spline-based method was simpler and tended to give better results for small deformations. However, large variability in the spline results and relatively small brain shift prevented this study from demonstrating a statistically significant difference between the results of the two methods.

Multiple-correlation similarity for block-matching based fast CT to ultrasound registration in liver interventions

In this work we present a fast approach to perform registration of computed tomography to ultrasound volumes for image guided intervention applications. The method is based on a combination of block-matching and outlier rejection. The block-matching uses a correlation based multimodal similarity metric, where the intensity and the gradient of the computed tomography images along with the ultrasound volumes are the input images to find correspondences between blocks in the computed tomography and the ultrasound volumes. A variance and octree based feature point-set selection method is used for selecting distinct and evenly spread point locations for block-matching. Geometric consistency and smoothness criteria are imposed in an outlier rejection step to refine the block-matching results. The block-matching results after outlier rejection are used to determine the affine transformation between the computed tomography and the ultrasound volumes. Various experiments are carried out to assess the optimal performance and the influence of parameters on accuracy and computational time of the registration. A leave-one-patient-out cross-validation registration error of 3.6 mm is achieved over 29 datasets, acquired from 17 patients.

Preliminary Results Comparing Thin Plate Splines with Finite Element Methods for Modeling Brain Deformation during Neurosurgery using Intraoperative Ultrasound

Brain shift compensation attempts to model the deformation of the brain which occurs during the surgical removal of brain tumors to enable mapping of presurgical image data into patient coordinates during surgery and thus improve the accuracy and utility of neuro-navigation. We present preliminary results from clinical tumor resections that compare two methods for modeling brain deformation, a simple thin plate spline method that interpolates displacements and a more complex finite element method (FEM) that models physical and geometric constraints of the brain and its material properties. Both methods are driven by the same set of displacements at locations surrounding the tumor. These displacements were derived from sets of corresponding matched features that were automatically detected using the SIFT-Rank algorithm. The deformation accuracy was tested using a set of manually identified landmarks. The FEM method requires significantly more preprocessing than the spline method but both methods can be used to model deformations in the operating room in reasonable time frames. Our preliminary results indicate that the FEM deformation model significantly out-performs the spline-based approach for predicting the deformation of manual landmarks. While both methods compensate for brain shift, this work suggests that models that incorporate biophysics and geometric constraints may be more accurate.

Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

Learning deep similarity metric for 3D MR-TRUS image registration

PURPOSE

The fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images for guiding targeted prostate biopsy has significantly improved the biopsy yield of aggressive cancers. A key component of MR-TRUS fusion is image registration. However, it is very challenging to obtain a robust automatic MR-TRUS registration due to the large appearance difference between the two imaging modalities. The work presented in this paper aims to tackle this problem by addressing two challenges: (i) the definition of a suitable similarity metric and (ii) the determination of a suitable optimization strategy.

METHODS

This work proposes the use of a deep convolutional neural network to learn a similarity metric for MR-TRUS registration. We also use a composite optimization strategy that explores the solution space in order to search for a suitable initialization for the second-order optimization of the learned metric. Further, a multi-pass approach is used in order to smooth the metric for optimization.

RESULTS

The learned similarity metric outperforms the classical mutual information and also the state-of-the-art MIND feature-based methods. The results indicate that the overall registration framework has a large capture range. The proposed deep similarity metric-based approach obtained a mean TRE of 3.86 mm (with an initial TRE of 16 mm) for this challenging problem.

CONCLUSION

A similarity metric that is learned using a deep neural network can be used to assess the quality of any given image registration and can be used in conjunction with the aforementioned optimization framework to perform automatic registration that is robust to poor initialization.

Real-time, image-based slice-to-volume registration for ultrasound-guided spinal intervention

Real-time fusion of magnetic resonance (MR) and ultrasound (US) images could facilitate safe and accurate needle placement in spinal interventions. We develop an entirely image-based registration method (independent of or complementary to surgical trackers) that includes an efficient US probe pose initialization algorithm. The registration enables the simultaneous display of 2D ultrasound image slices relative to 3D pre-procedure MR images for navigation. A dictionary-based 3D-2D pose initialization algorithm was developed in which likely probe positions are predefined in a dictionary with feature encoding by Haar wavelet filters. Feature vectors representing the 2D US image are computed by scaling and translating multiple Haar basis filters to capture scale, location, and relative intensity patterns of distinct anatomical features. Following pose initialization, fast 3D-2D registration was performed by optimizing normalized cross-correlation between intra- and pre-procedure images using Powell's method. Experiments were performed using a lumbar puncture phantom and a fresh cadaver specimen presenting realistic image quality in spinal US imaging. Accuracy was quantified by comparing registration transforms to ground truth motion imparted by a computer-controlled motion system and calculating target registration error (TRE) in anatomical landmarks. Initialization using a 315-length feature vector yielded median translation accuracy of 2.7 mm (3.4 mm interquartile range, IQR) in the phantom and 2.1 mm (2.5 mm IQR) in the cadaver. By comparison, storing the entire image set in the dictionary and optimizing correlation yielded a comparable median accuracy of 2.1 mm (2.8 mm IQR) in the phantom and 2.9 mm (3.5 mm IQR) in the cadaver. However, the dictionary-based method reduced memory requirements by 47×  compared to storing the entire image set. The overall 3D error after registration measured using 3D landmarks was 3.2 mm (1.8 mm IQR) mm in the phantom and 3.0 mm (2.3 mm IQR) mm in the cadaver. The system was implemented in a 3D Slicer interface to facilitate translation to clinical studies. Haar feature based initialization provided accuracy and robustness at a level that was sufficient for real-time registration using an entirely image-based method for ultrasound navigation. Such an approach could improve the accuracy and safety of spinal interventions in broad utilization, since it is entirely software-based and can operate free from the cost and workflow requirements of surgical trackers.

Development of a robust MRI fiducial system for automated fusion of MR-US abdominal images

We present the development of a two‐component magnetic resonance (MR) fiducial system, that is, a fiducial marker device combined with an auto‐segmentation algorithm, designed to be paired with existing ultrasound probe tracking and image fusion technology to automatically fuse MR and ultrasound (US) images. The fiducial device consisted of four ~6.4 mL cylindrical wells filled with 1 g/L copper sulfate solution. The algorithm was designed to automatically segment the device in clinical abdominal MR images. The algorithm's detection rate and repeatability were investigated through a phantom study and in human volunteers. The detection rate was 100% in all phantom and human images. The center‐of‐mass of the fiducial device was robustly identified with maximum variations of 2.9 mm in position and 0.9° in angular orientation. In volunteer images, average differences between algorithm‐measured inter‐marker spacings and actual separation distances were 0.53 ± 0.36 mm. “Proof‐of‐concept” automatic MR‐US fusions were conducted with sets of images from both a phantom and volunteer using a commercial prototype system, which was built based on the above findings. Image fusion accuracy was measured to be within 5 mm for breath‐hold scanning. These results demonstrate the capability of this approach to automatically fuse US and MR images acquired across a wide range of clinical abdominal pulse sequences.

Optimized 3D co-registration of ultra-low-field and high-field magnetic resonance images

The prototypes of ultra-low-field (ULF) MRI scanners developed in recent years represent new, innovative, cost-effective and safer systems, which are suitable to be integrated in multi-modal (Magnetoencephalography and MRI) devices. Integrated ULF-MRI and MEG scanners could represent an ideal solution to obtain functional (MEG) and anatomical (ULF MRI) information in the same environment, without errors that may limit source reconstruction accuracy. However, the low resolution and signal-to-noise ratio (SNR) of ULF images, as well as their limited coverage, do not generally allow for the construction of an accurate individual volume conductor model suitable for MEG localization. Thus, for practical usage, a high-field (HF) MRI image is also acquired, and the HF-MRI images are co-registered to the ULF-MRI ones. We address here this issue through an optimized pipeline (SWIM—Sliding WIndow grouping supporting Mutual information). The co-registration is performed by an affine transformation, the parameters of which are estimated using Normalized Mutual Information as the cost function, and Adaptive Simulated Annealing as the minimization algorithm. The sub-voxel resolution of the ULF images is handled by a sliding-window approach applying multiple grouping strategies to down-sample HF MRI to the ULF-MRI resolution. The pipeline has been tested on phantom and real data from different ULF-MRI devices, and comparison with well-known toolboxes for fMRI analysis has been performed. Our pipeline always outperformed the fMRI toolboxes (FSL and SPM). The HF–ULF MRI co-registration obtained by means of our pipeline could lead to an effective integration of ULF MRI with MEG, with the aim of improving localization accuracy, but also to help exploit ULF MRI in tumor imaging.

Inter-scanner Variation Independent Descriptors for Constrained Diffeomorphic Demons Registration of Retina OCT

PURPOSE

OCT offers high in-plane micrometer resolution, enabling studies of neurodegenerative and ocular-disease mechanisms via imaging of the retina at low cost. An important component to such studies is inter-scanner deformable image registration. Image quality of OCT, however, is suboptimal with poor signal-to-noise ratio and through-plane resolution. Geometry of OCT is additionally improperly defined. We developed a diffeomorphic deformable registration method incorporating constraints accommodating the improper geometry and a decentralized-modality-insensitive-neighborhood-descriptors (D-MIND) robust against degradation of OCT image quality and inter-scanner variability.

METHOD

The method, called D-MIND Demons, estimates diffeomorphisms using D-MINDs under constraints on the direction of velocity fields in a MIND-Demons framework. Descriptiveness of D-MINDs with/without denoising was ranked against four other shape/texture-based descriptors. Performance of D-MIND Demons and its variants incorporating other descriptors was compared for cross-scanner, intra- and inter-subject deformable registration using clinical retina OCT data.

RESULT

D-MINDs outperformed other descriptors with the difference in mutual descriptiveness between high-contrast and homogenous regions > 0.2. Among Demons variants, D-MIND-Demons was computationally efficient, demonstrating robustness against OCT image degradation (noise, speckle, intensity-non-uniformity, and poor through-plane resolution) and consistent registration accuracy [(4±4 μm) and (4±6 μm) in cross-scanner intra- and inter-subject registration] regardless of denoising.

CONCLUSIONS

A promising method for cross-scanner, intra- and inter-subject OCT image registration has been developed for ophthalmological and neurological studies of retinal structures. The approach could assist image segmentation, evaluation of longitudinal disease progression, and patient population analysis, which in turn, facilitate diagnosis and patient-specific treatment.

Patient-specific model-based segmentation of brain tumors in 3D intraoperative ultrasound images

PURPOSE

Intraoperative ultrasound (iUS) imaging is commonly used to support brain tumor operation. The tumor segmentation in the iUS images is a difficult task and still under improvement because of the low signal-to-noise ratio. The success of automatic methods is also limited due to the high noise sensibility. Therefore, an alternative brain tumor segmentation method in 3D-iUS data using a tumor model obtained from magnetic resonance (MR) data for local MR-iUS registration is presented in this paper. The aim is to enhance the visualization of the brain tumor contours in iUS.

METHODS

A multistep approach is proposed. First, a region of interest (ROI) based on the specific patient tumor model is defined. Second, hyperechogenic structures, mainly tumor tissues, are extracted from the ROI of both modalities by using automatic thresholding techniques. Third, the registration is performed over the extracted binary sub-volumes using a similarity measure based on gradient values, and rigid and affine transformations. Finally, the tumor model is aligned with the 3D-iUS data, and its contours are represented.

RESULTS

Experiments were successfully conducted on a dataset of 33 patients. The method was evaluated by comparing the tumor segmentation with expert manual delineations using two binary metrics: contour mean distance and Dice index. The proposed segmentation method using local and binary registration was compared with two grayscale-based approaches. The outcomes showed that our approach reached better results in terms of computational time and accuracy than the comparative methods.

CONCLUSION

The proposed approach requires limited interaction and reduced computation time, making it relevant for intraoperative use. Experimental results and evaluations were performed offline. The developed tool could be useful for brain tumor resection supporting neurosurgeons to improve tumor border visualization in the iUS volumes.

Multimodal image registration based on binary gradient angle descriptor

PURPOSE

Multimodal image registration plays an important role in image-guided interventions/therapy and atlas building, and it is still a challenging task due to the complex intensity variations in different modalities.

METHODS

The paper addresses the problem and proposes a simple, compact, fast and generally applicable modality-independent binary gradient angle descriptor (BGA) based on the rationale of gradient orientation alignment. The BGA can be easily calculated at each voxel by coding the quadrant in which a local gradient vector falls, and it has an extremely low computational complexity, requiring only three convolutions, two multiplication operations and two comparison operations. Meanwhile, the binarized encoding of the gradient orientation makes the BGA more resistant to image degradations compared with conventional gradient orientation methods. The BGA can extract similar feature descriptors for different modalities and enable the use of simple similarity measures, which makes it applicable within a wide range of optimization frameworks.

RESULTS

The results for pairwise multimodal and monomodal registrations between various images (T1, T2, PD, T1c, Flair) consistently show that the BGA significantly outperforms localized mutual information. The experimental results also confirm that the BGA can be a reliable alternative to the sum of absolute difference in monomodal image registration. The BGA can also achieve an accuracy of [Formula: see text], similar to that of the SSC, for the deformable registration of inhale and exhale CT scans. Specifically, for the highly challenging deformable registration of preoperative MRI and 3D intraoperative ultrasound images, the BGA achieves a similar registration accuracy of [Formula: see text] compared with state-of-the-art approaches, with a computation time of 18.3 s per case.

CONCLUSIONS

The BGA improves the registration performance in terms of both accuracy and time efficiency. With further acceleration, the framework has the potential for application in time-sensitive clinical environments, such as for preoperative MRI and intraoperative US image registration for image-guided intervention.

REtroSpective Evaluation of Cerebral Tumors (RESECT): A clinical database of pre-operative MRI and intra-operative ultrasound in low-grade glioma surgeries

PURPOSE

The advancement of medical image processing techniques, such as image registration, can effectively help improve the accuracy and efficiency of brain tumor surgeries. However, it is often challenging to validate these techniques with real clinical data due to the rarity of such publicly available repositories.

ACQUISITION AND VALIDATION METHODS

Pre-operative magnetic resonance images (MRI), and intra-operative ultrasound (US) scans were acquired from 23 patients with low-grade gliomas who underwent surgeries at St. Olavs University Hospital between 2011 and 2016. Each patient was scanned by Gadolinium-enhanced T1w and T2-FLAIR MRI protocols to reveal the anatomy and pathology, and series of B-mode ultrasound images were obtained before, during, and after tumor resection to track the surgical progress and tissue deformation. Retrospectively, corresponding anatomical landmarks were identified across US images of different surgical stages, and between MRI and US, and can be used to validate image registration algorithms. Quality of landmark identification was assessed with intra- and inter-rater variability.

DATA FORMAT AND ACCESS

In addition to co-registered MRIs, each series of US scans are provided as a reconstructed 3D volume. All images are accessible in MINC2 and NIFTI formats, and the anatomical landmarks were annotated in MNI tag files. Both the imaging data and the corresponding landmarks are available online as the RESECT database at https://archive.norstore.no (search for "RESECT").

POTENTIAL IMPACT

The proposed database provides real high-quality multi-modal clinical data to validate and compare image registration algorithms that can potentially benefit the accuracy and efficiency of brain tumor resection. Furthermore, the database can also be used to test other image processing methods and neuro-navigation software platforms.

Fast and robust multimodal image registration using a local derivative pattern

PURPOSE

Deformable multimodal image registration, which can benefit radiotherapy and image guided surgery by providing complementary information, remains a challenging task in the medical image analysis field due to the difficulty of defining a proper similarity measure. This article presents a novel, robust and fast binary descriptor, the discriminative local derivative pattern (dLDP), which is able to encode images of different modalities into similar image representations.

METHODS

dLDP calculates a binary string for each voxel according to the pattern of intensity derivatives in its neighborhood. The descriptor similarity is evaluated using the Hamming distance, which can be efficiently computed, instead of conventional L1 or L2 norms. For the first time, we validated the effectiveness and feasibility of the local derivative pattern for multimodal deformable image registration with several multi-modal registration applications.

RESULTS

dLDP was compared with three state-of-the-art methods in artificial image and clinical settings. In the experiments of deformable registration between different magnetic resonance imaging (MRI) modalities from BrainWeb, between computed tomography and MRI images from patient data, and between MRI and ultrasound images from BITE database, we show our method outperforms localized mutual information and entropy images in terms of both accuracy and time efficiency. We have further validated dLDP for the deformable registration of preoperative MRI and three-dimensional intraoperative ultrasound images. Our results indicate that dLDP reduces the average mean target registration error from 4.12 mm to 2.30 mm. This accuracy is statistically equivalent to the accuracy of the state-of-the-art methods in the study; however, in terms of computational complexity, our method significantly outperforms other methods and is even comparable to the sum of the absolute difference.

CONCLUSIONS

The results reveal that dLDP can achieve superior performance regarding both accuracy and time efficiency in general multimodal image registration. In addition, dLDP also indicates the potential for clinical ultrasound guided intervention.

Geometric modeling of hepatic arteries in 3D ultrasound with unsupervised MRA fusion during liver interventions

PURPOSE

Modulating the chemotherapy injection rate with regard to blood flow velocities in the tumor-feeding arteries during intra-arterial therapies may help improve liver tumor targeting while decreasing systemic exposure. These velocities can be obtained noninvasively using Doppler ultrasound (US). However, small vessels situated in the liver are difficult to identify and follow in US. We propose a multimodal fusion approach that non-rigidly registers a 3D geometric mesh model of the hepatic arteries obtained from preoperative MR angiography (MRA) acquisitions with intra-operative 3D US imaging.

METHODS

The proposed fusion tool integrates 3 imaging modalities: an arterial MRA, a portal phase MRA and an intra-operative 3D US. Preoperatively, the arterial phase MRA is used to generate a 3D model of the hepatic arteries, which is then non-rigidly co-registered with the portal phase MRA. Once the intra-operative 3D US is acquired, we register it with the portal MRA using a vessel-based rigid initialization followed by a non-rigid registration using an image-based metric based on linear correlation of linear combination. Using the combined non-rigid transformation matrices, the 3D mesh model is fused with the 3D US.

RESULTS

3D US and multi-phase MRA images acquired from 10 porcine models were used to test the performance of the proposed fusion tool. Unimodal registration of the MRA phases yielded a target registration error (TRE) of [Formula: see text] mm. Initial rigid alignment of the portal MRA and 3D US yielded a mean TRE of [Formula: see text] mm, which was significantly reduced to [Formula: see text] mm ([Formula: see text]) after affine image-based registration. The following deformable registration step allowed for further decrease of the mean TRE to [Formula: see text] mm.

CONCLUSION

The proposed tool could facilitate visualization and localization of these vessels when using 3D US intra-operatively for either intravascular or percutaneous interventions to avoid vessel perforation.

Towards MRI-Based Autonomous Robotic US Acquisitions: A First Feasibility Study

Robotic ultrasound has the potential to assist and guide physicians during interventions. In this work, we present a set of methods and a workflow to enable autonomous MRI-guided ultrasound acquisitions. Our approach uses a structured-light 3D scanner for patient-to-robot and image-to-patient calibration, which in turn is used to plan 3D ultrasound trajectories. These MRI-based trajectories are followed autonomously by the robot and are further refined online using automatic MRI/US registration. Despite the low spatial resolution of structured light scanners, the initial planned acquisition path can be followed with an accuracy of 2.46 ± 0.96 mm. This leads to a good initialization of the MRI/US registration: the 3D-scan-based alignment for planning and acquisition shows an accuracy (distance between planned ultrasound and MRI) of 4.47 mm, and 0.97 mm after an online-update of the calibration based on a closed loop registration.

miLBP: a robust and fast modality-independent 3D LBP for multimodal deformable registration

Purpose

Computer-assisted intervention often depends on multimodal deformable registration to provide complementary information. However, multimodal deformable registration remains a challenging task.

Methods

This paper introduces a novel robust and fast modality-independent 3D binary descriptor, called miLBP, which integrates the principle of local self-similarity with a form of local binary pattern and can robustly extract the similar geometry features from 3D volumes across different modalities. miLBP is a bit string that can be computed by simply thresholding the voxel distance. Furthermore, the descriptor similarity can be evaluated efficiently using the Hamming distance.

Results

miLBP was compared to vector-valued self-similarity context (SSC) in artificial image and clinical settings. The results show that miLBP is more robust than SSC in extracting local geometry features across modalities and achieved higher registration accuracy in different registration scenarios. Furthermore, in the most challenging registration between preoperative magnetic resonance imaging and intra-operative ultrasound images, our approach significantly outperforms the state-of-the-art methods in terms of both accuracy (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2.15\pm 1.1 \hbox { mm}$$\end{document}2.15±1.1mm) and speed (29.2 s for one case).

Conclusions

Registration performance and speed indicate that miLBP has the potential of being applied to the time-sensitive intra-operative computer-assisted intervention.

Vascular Structure Identification in Intraoperative 3D Contrast-Enhanced Ultrasound Data

In this paper, a method of vascular structure identification in intraoperative 3D Contrast-Enhanced Ultrasound (CEUS) data is presented. Ultrasound imaging is commonly used in brain tumor surgery to investigate in real time the current status of cerebral structures. The use of an ultrasound contrast agent enables to highlight tumor tissue, but also surrounding blood vessels. However, these structures can be used as landmarks to estimate and correct the brain shift. This work proposes an alternative method for extracting small vascular segments close to the tumor as landmark. The patient image dataset involved in brain tumor operations includes preoperative contrast T1MR (cT1MR) data and 3D intraoperative contrast enhanced ultrasound data acquired before (3D-iCEUS(start) and after (3D-iCEUS(end) tumor resection. Based on rigid registration techniques, a preselected vascular segment in cT1MR is searched in 3D-iCEUS(start) and 3D-iCEUS(end) data. The method was validated by using three similarity measures (Normalized Gradient Field, Normalized Mutual Information and Normalized Cross Correlation). Tests were performed on data obtained from ten patients overcoming a brain tumor operation and it succeeded in nine cases. Despite the small size of the vascular structures, the artifacts in the ultrasound images and the brain tissue deformations, blood vessels were successfully identified.

Automated 3D-2D registration of X-ray microcomputed tomography with histological sections for dental implants in bone using chamfer matching and simulated annealing

We propose a novel 3D-2D registration approach for micro-computed tomography (μCT) and histology (HI), constructed for dental implant biopsies, that finds the position and normal vector of the oblique slice from μCT that corresponds to HI. During image pre-processing, the implants and the bone tissue are segmented using a combination of thresholding, morphological filters and component labeling. After this, chamfer matching is employed to register the implant edges and fine registration of the bone tissues is achieved using simulated annealing. The method was tested on n=10 biopsies, obtained at 20 weeks after non-submerged healing in the canine mandible. The specimens were scanned with μCT 100 and processed for hard tissue sectioning. After registration, we assessed the agreement of bone to implant contact (BIC) using automated and manual measurements. Statistical analysis was conducted to test the agreement of the BIC measurements in the registered samples. Registration was successful for all specimens and agreement of the respective binary images was high (median: 0.90, 1.-3. Qu.: 0.89-0.91). Direct comparison of BIC yielded that automated (median 0.82, 1.-3. Qu.: 0.75-0.85) and manual (median 0.61, 1.-3. Qu.: 0.52-0.67) measures from μCT were significant positively correlated with HI (median 0.65, 1.-3. Qu.: 0.59-0.72) between μCT and HI groups (manual: R(2)=0.87, automated: R(2)=0.75, p<0.001). The results show that this method yields promising results and that μCT may become a valid alternative to assess osseointegration in three dimensions.