Automated 2D-3D registration of a radiograph and a cone beam CT using line-segment enhancement

Image preprocessing to improve the accuracy and robustness of mutual-information-based automatic image registration in proton therapy

PURPOSE

We propose a method that potentially improves the outcome of mutual-information-based automatic image registration by using the contrast enhancement filter (CEF).

METHODS

Seventy-six pairs of two-dimensional X-ray images and digitally reconstructed radiographs for 20 head and neck and nine lung cancer patients were analyzed retrospectively. Automatic image registration was performed using the mutual-information-based algorithm in VeriSuite®. Images were preprocessed using the CEF in VeriSuite®. The correction vector for translation and rotation error was calculated and manual image registration was compared with automatic image registration, with and without CEF. In addition, the normalized mutual information (NMI) distribution between two-dimensional images was compared, with and without CEF.

RESULTS

In the correction vector comparison between manual and automatic image registration, the average differences in translation error were < 1 mm in most cases in the head and neck region. The average differences in rotation error were 0.71 and 0.16 degrees without and with CEF, respectively, in the head and neck region; they were 2.67 and 1.64 degrees, respectively, in the chest region. When used with oblique projection, the average rotation error was 0.39 degrees with CEF. CEF improved the NMI by 17.9 % in head and neck images and 18.2 % in chest images.

CONCLUSIONS

CEF preprocessing improved the NMI and registration accuracy of mutual-information-based automatic image registration on the medical images. The proposed method achieved accuracy equivalent to that achieved by experienced therapists and it will significantly contribute to the standardization of image registration quality.

A novel U-Net approach to segment the cardiac chamber in magnetic resonance images with ghost artifacts

OBJECTIVE

We propose a robust technique for segmenting magnetic resonance images of post-atrial septal occlusion intervention in the cardiac chamber.

METHODS

A variant of the U-Net architecture is used to perform atrial segmentation via a deep convolutional neural network, and we compare performance with the Kass snake model. It can be used to determine the surgical success of atrial septal occlusion (ASO) pre- and post- the implantation of the septal occluder, which is based on the volume restoration of the right atria (RA) and left atria (LA).

RESULTS

The method was evaluated on a test dataset containing 550 two-dimensional image slices, outperforming conventional active contouring regarding the Dice similarity coefficient, Jaccard index, and Hausdorff distance, and achieving segmentation in the presence of ghost artifacts that occlude the atrium outline. This problem has been unsolvable using traditional machine learning algorithm pertaining to active contouring via the Kass snake algorithm. Moreover, the proposed technique is closer to manual segmentation than the snakes active contour model in mean of atrial area (M-AA), mean of atrial maximum diameter (M-AMXD), mean atrial minimum diameter (M-AMID), and mean angle of the atrial long axis (M-AALA).

CONCLUSION

After segmentation, we compute the volume ratio of right to left atria, obtaining a smaller ratio that indicates better restoration. Hence, the proposed technique allows to evaluate the surgical success of atrial septal occlusion and may support diagnosis regarding the accurate evaluation of atrial septal defects before and after occlusion procedures.

Cardiac Chamber Segmentation Using Deep Learning on Magnetic Resonance Images from Patients Before and After Atrial Septal Occlusion Surgery

We propose a robust technique for segmenting magnetic resonance images of post-atrial septal occlusion intervention in the cardiac chamber. The technique can be used to determine the surgical outcomes of atrial septal defects before and after implantation of a septal occluder, which intends to provide volume restoration of the right and left atria. A variant of the U-Net architecture is used to perform atrial segmentation via a deep convolutional neural network. The method was evaluated on a dataset containing 550 two-dimensional image slices, outperforming conventional active contouring regarding the Dice similarity coefficient, Jaccard index, and Hausdorff distance, and achieving segmentation in the presence of ghost artifacts that occlude the atrium outline. Moreover, the proposed technique is closer to manual segmentation than the snakes active contour model. After segmentation, we computed the volume ratio of right to left atria, obtaining a smaller ratio that indicates better restoration. Hence, the proposed technique allows to evaluate the surgical success of atrial septal occlusion and may support diagnosis regarding the accurate evaluation of atrial septal defects before and after occlusion procedures.

Multilevel 2D-3D Intensity-Based Image Registration

2D-3D image registration is an important task for computer-aided minimally invasive vascular therapies. A crucial component for practical image registration is the use of multilevel strategies to avoid local optima and to speed-up runtime. However, due to the different dimensionalities of the 2D fixed and 3D moving image, the setup of multilevel strategies is not straightforward.

In this work, we propose an intensity-driven 2D-3D multiresolution registration approach using the normalized gradient fields (NGF) distance measure. We discuss and empirically analyze the impact on the choice of 2D and 3D image resolutions. Furthermore, we show that our approach produces results that are comparable or superior to other state-of-the-art methods.

The need for, and implementation of, image guidance in radiation therapy

The introduction of image guidance in radiation therapy and its subsequent innovations have revolutionised the delivery of cancer treatment. Modern imaging systems can supplement and often replace the historical practice of relying on external landmarks and laser alignment systems. Rather than depending on markings on the patient's skin, image-guided radiation therapy (IGRT), using techniques such as computed tomography (CT), cone beam CT, MV on-board imaging (OBI), and kV OBI, allows the patient to be positioned based on the internal anatomy. These advances in technology have enabled more accurate delivery of radiation doses to anatomically complex and temporally changing tumour volumes, while simultaneously sparing surrounding healthy tissues. While these imaging modalities provide excellent bony anatomy image quality, magnetic resonance imaging (MRI) surpasses them in soft tissue image contrast for better visualisation and tracking of soft tissue tumours with no additional radiation dose to the patient. However, the introduction of MRI into a radiotherapy facility has a number of complications, including the influence of the magnetic field on the dose deposition, as well as the effects it can have on dosimetry systems. The development and introduction of these new IGRT techniques will be reviewed, and the benefits and disadvantages of each will be described.

A frequency-based approach to locate common structure for 2D-3D intensity-based registration of setup images in prostate radiotherapy

In many radiotherapy clinics, geometric uncertainties in the delivery of 3D conformal radiation therapy and intensity modulated radiation therapy of the prostate are reduced by aligning the patient's bony anatomy in the planning 3D CT to corresponding bony anatomy in 2D portal images acquired before every treatment fraction. In this paper, we seek to determine if there is a frequency band within the portal images and the digitally reconstructed radiographs (DRRs) of the planning CT in which bony anatomy predominates over non-bony anatomy such that portal images and DRRs can be suitably filtered to achieve high registration accuracy in an automated 2D-3D single portal intensity-based registration framework. Two similarity measures, mutual information and the Pearson correlation coefficient were tested on carefully collected gold-standard data consisting of a kilovoltage cone-beam CT (CBCT) and megavoltage portal images in the anterior-posterior (AP) view of an anthropomorphic phantom acquired under clinical conditions at known poses, and on patient data. It was found that filtering the portal images and DRRs during the registration considerably improved registration performance. Without filtering, the registration did not always converge while with filtering it always converged to an accurate solution. For the pose-determination experiments conducted on the anthropomorphic phantom with the correlation coefficient, the mean (and standard deviation) of the absolute errors in recovering each of the six transformation parameters were Theta(x):0.18(0.19) degrees, Theta(y):0.04(0.04) degrees, Theta(z):0.04(0.02) degrees, t(x):0.14(0.15) mm, t(y):0.09(0.05) mm, and t(z):0.49(0.40) mm. The mutual information-based registration with filtered images also resulted in similarly small errors. For the patient data, visual inspection of the superimposed registered images showed that they were correctly aligned in all instances. The results presented in this paper suggest that robust and accurate registration can be achieved with intensity-based methods by focusing on rigid bony structures in the images while diminishing the influence of artifacts with similar frequencies as soft tissue.

Local metric learning in 2D/3D deformable registration with application in the abdomen

In image-guided radiotherapy (IGRT) of disease sites subject to respiratory motion, soft tissue deformations can affect localization accuracy. We describe the application of a method of 2D/3D deformable registration to soft tissue localization in abdomen. The method, called registration efficiency and accuracy through learning a metric on shape (REALMS), is designed to support real-time IGRT. In a previously developed version of REALMS, the method interpolated 3D deformation parameters for any credible deformation in a deformation space using a single globally-trained Riemannian metric for each parameter. We propose a refinement of the method in which the metric is trained over a particular region of the deformation space, such that interpolation accuracy within that region is improved. We report on the application of the proposed algorithm to IGRT in abdominal disease sites, which is more challenging than in lung because of low intensity contrast and nonrespiratory deformation. We introduce a rigid translation vector to compensate for nonrespiratory deformation, and design a special region-of-interest around fiducial markers implanted near the tumor to produce a more reliable registration. Both synthetic data and actual data tests on abdominal datasets show that the localized approach achieves more accurate 2D/3D deformable registration than the global approach.

2D/3D Image Registration using Regression Learning

In computer vision and image analysis, image registration between 2D projections and a 3D image that achieves high accuracy and near real-time computation is challenging. In this paper, we propose a novel method that can rapidly detect an object's 3D rigid motion or deformation from a 2D projection image or a small set thereof. The method is called CLARET (Correction via Limited-Angle Residues in External Beam Therapy) and consists of two stages: registration preceded by shape space and regression learning. In the registration stage, linear operators are used to iteratively estimate the motion/deformation parameters based on the current intensity residue between the target projec-tion(s) and the digitally reconstructed radiograph(s) (DRRs) of the estimated 3D image. The method determines the linear operators via a two-step learning process. First, it builds a low-order parametric model of the image region's motion/deformation shape space from its prior 3D images. Second, using learning-time samples produced from the 3D images, it formulates the relationships between the model parameters and the co-varying 2D projection intensity residues by multi-scale linear regressions. The calculated multi-scale regression matrices yield the coarse-to-fine linear operators used in estimating the model parameters from the 2D projection intensity residues in the registration. The method's application to Image-guided Radiation Therapy (IGRT) requires only a few seconds and yields good results in localizing a tumor under rigid motion in the head and neck and under respiratory deformation in the lung, using one treatment-time imaging 2D projection or a small set thereof.

TREK: an integrated system architecture for intraoperative cone-beam CT-guided surgery

PURPOSE

A system architecture has been developed for integration of intraoperative 3D imaging [viz., mobile C-arm cone-beam CT (CBCT)] with surgical navigation (e.g., trackers, endoscopy, and preoperative image and planning data). The goal of this paper is to describe the architecture and its handling of a broad variety of data sources in modular tool development for streamlined use of CBCT guidance in application-specific surgical scenarios.

METHODS

The architecture builds on two proven open-source software packages, namely the cisst package (Johns Hopkins University, Baltimore, MD) and 3D Slicer (Brigham and Women's Hospital, Boston, MA), and combines data sources common to image-guided procedures with intraoperative 3D imaging. Integration at the software component level is achieved through language bindings to a scripting language (Python) and an object-oriented approach to abstract and simplify the use of devices with varying characteristics. The platform aims to minimize offline data processing and to expose quantitative tools that analyze and communicate factors of geometric precision online. Modular tools are defined to accomplish specific surgical tasks, demonstrated in three clinical scenarios (temporal bone, skull base, and spine surgery) that involve a progressively increased level of complexity in toolset requirements.

RESULTS

The resulting architecture (referred to as "TREK") hosts a collection of modules developed according to application-specific surgical tasks, emphasizing streamlined integration with intraoperative CBCT. These include multi-modality image display; 3D-3D rigid and deformable registration to bring preoperative image and planning data to the most up-to-date CBCT; 3D-2D registration of planning and image data to real-time fluoroscopy; infrared, electromagnetic, and video-based trackers used individually or in hybrid arrangements; augmented overlay of image and planning data in endoscopic or in-room video; and real-time "virtual fluoroscopy" computed from GPU-accelerated digitally reconstructed radiographs (DRRs). Application in three preclinical scenarios (temporal bone, skull base, and spine surgery) demonstrates the utility of the modular, task-specific approach in progressively complex tasks.

CONCLUSIONS

The design and development of a system architecture for image-guided surgery has been reported, demonstrating enhanced utilization of intraoperative CBCT in surgical applications with vastly different requirements. The system integrates C-arm CBCT with a broad variety of data sources in a modular fashion that streamlines the interface to application-specific tools, accommodates distinct workflow scenarios, and accelerates testing and translation of novel toolsets to clinical use. The modular architecture was shown to adapt to and satisfy the requirements of distinct surgical scenarios from a common code-base, leveraging software components arising from over a decade of effort within the imaging and computer-assisted interventions community.

A feasibility study of a molecular-based patient setup verification method using a parallel-plane PET system

A feasibility study of a novel PET-based molecular image guided radiation therapy (m-IGRT) system was conducted by comparing PET-based digitally reconstructed planar image (PDRI) registration with radiographic registration. We selected a pair of opposing parallel-plane PET systems for the practical implementation of this system. Planar images along the in-plane and cross-plane directions were reconstructed from the parallel-plane PET data. The in-plane and cross-plane FWHM of the profile of 2 mm diameter sources was approximately 1.8 and 8.1 mm, respectively. Therefore, only the reconstructed in-plane image from the parallel-plane PET data was used in the PDRI registration. In the image registration, five different sizes of (18)F cylindrical sources (diameter: 8, 12, 16, 24, 32 mm) were used to determine setup errors. The data acquisition times were 1, 3 and 5 min. Image registration was performed by five observers to determine the setup errors from PDRI registration and radiographic registration. The majority of the mean registration errors obtained from the PDRI registration were not significantly different from those obtained from the radiographic registration. Acquisition time did not appear to result in significant differences in the mean registration error. The mean registration error for the PDRI registration was found to be 0.93 ± 0.33 mm. This is not statistically different from the radiographic registration which had a mean registration error of 0.92 ± 0.27 mm. Our results suggest that m-IGRT image registration using PET-based reconstructed planar images along the in-plane direction is feasible for clinical use if PDRI registration is performed at two orthogonal gantry angles.

Multiple-object 2-D-3-D registration for noninvasive pose identification of fracture fragments

This paper presents a multiple-object 2-D-3-D registration technique for noninvasively identifying the poses of fracture fragments in the space of a preoperative treatment plan. The plan is made by manipulating and aligning computer models of individual fracture fragments that are segmented from a diagnostic computed tomography. The registration technique iteratively updates the treatment plan and matches its digitally reconstructed radiographs to a small number of intraoperative fluoroscopic images. The proposed approach combines an image similarity metric that integrates edge information with mutual information, and a global-local optimization scheme, to deal with challenges associated with the registration of multiple small fragments and limited imaging orientations in the operating room. The method is easy to use as minimum user interaction is required. Experiments on simulated fractures and two distal radius fracture phantoms demonstrate clinically acceptable target registration errors with capture range as large as 10 mm.

Robust initialization of 2D-3D image registration using the projection-slice theorem and phase correlation

PURPOSE

The image registration literature comprises many methods for 2D-3D registration for which accuracy has been established in a variety of applications. However, clinical application is limited by a small capture range. Initial offsets outside the capture range of a registration method will not converge to a successful registration. Previously reported capture ranges, defined as the 95% success range, are in the order of 4-11 mm mean target registration error. In this article, a relatively computationally inexpensive and robust estimation method is proposed with the objective to enlarge the capture range.

METHODS

The method uses the projection-slice theorem in combination with phase correlation in order to estimate the transform parameters, which provides an initialization of the subsequent registration procedure.

RESULTS

The feasibility of the method was evaluated by experiments using digitally reconstructed radiographs generated from in vivo 3D-RX data. With these experiments it was shown that the projection-slice theorem provides successful estimates of the rotational transform parameters for perspective projections and in case of translational offsets. The method was further tested on ex vivo ovine x-ray data. In 95% of the cases, the method yielded successful estimates for initial mean target registration errors up to 19.5 mm. Finally, the method was evaluated as an initialization method for an intensity-based 2D-3D registration method. The uninitialized and initialized registration experiments had success rates of 28.8% and 68.6%, respectively.

CONCLUSIONS

The authors have shown that the initialization method based on the projection-slice theorem and phase correlation yields adequate initializations for existing registration methods, thereby substantially enlarging the capture range of these methods.

Automated 2D-3D registration of portal images and CT data using line-segment enhancement

In prostate radiotherapy, setup errors with respect to the patient's bony anatomy can be reduced by aligning 2D megavoltage (MV) portal images acquired during treatment to a reference 3D kilovoltage (kV) CT acquired for treatment planning purposes. The purpose of this study was to evaluate a fully automated 2D-3D registration algorithm to quantify setup errors in 3D through the alignment of line-enhanced portal images and digitally reconstructed radiographs computed from the CT. The line-enhanced images were obtained by correlating the images with a filter bank of short line segments, or "sticks" at different orientations. The proposed methods were validated on (1) accurately collected gold-standard data consisting of a 3D kV cone-beam CT scan of an anthropomorphic phantom of the pelvis and 2D MV portal images in the anterior-posterior (AP) view acquired at 15 different poses and (2) a conventional 3D kV CT scan and weekly 2D MV AP portal images of a patient over 8 weeks. The mean (and standard deviation) of the absolute registration error for rotations around the right-lateral (RL), inferior-superior (IS), and posterior-anterior (PA) axes were 0.212 degree (0.214 degree), 0.055 degree (0.033 degree) and 0.041 degree (0.039 degree), respectively. The corresponding registration errors for translations along the RL, IS, and PA axes were 0.161 (0.131) mm, 0.096 (0.033) mm, and 0.612 (0.485) mm. The mean (and standard deviation) of the total registration error was 0.778 (0.543) mm. Registration on the patient images was successful in all eight cases as determined visually. The results indicate that it is feasible to automatically enhance features in MV portal images of the pelvis for use within a completely automated 2D-3D registration framework for the accurate determination of patient setup errors. They also indicate that it is feasible to estimate all six transformation parameters from a 3D CT of the pelvis and a single portal image in the AP view.

Fast DRR splat rendering using common consumer graphics hardware

Digitally rendered radiographs (DRR) are a vital part of various medical image processing applications such as 2D/3D registration for patient pose determination in image-guided radiotherapy procedures. This paper presents a technique to accelerate DRR creation by using conventional graphics hardware for the rendering process. DRR computation itself is done by an efficient volume rendering method named wobbled splatting. For programming the graphics hardware, NVIDIAs C for Graphics (Cg) is used. The description of an algorithm used for rendering DRRs on the graphics hardware is presented, together with a benchmark comparing this technique to a CPU-based wobbled splatting program. Results show a reduction of rendering time by about 70%-90% depending on the amount of data. For instance, rendering a volume of 2 x 10(6) voxels is feasible at an update rate of 38 Hz compared to 6 Hz on a common Intel-based PC using the graphics processing unit (GPU) of a conventional graphics adapter. In addition, wobbled splatting using graphics hardware for DRR computation provides higher resolution DRRs with comparable image quality due to special processing characteristics of the GPU. We conclude that DRR generation on common graphics hardware using the freely available Cg environment is a major step toward 2D/3D registration in clinical routine.

Direct aperture optimization for online adaptive radiation therapy

This paper is the first investigation of using direct aperture optimization (DAO) for online adaptive radiation therapy (ART). A geometrical model representing the anatomy of a typical prostate case was created. To simulate interfractional deformations, four different anatomical deformations were created by systematically deforming the original anatomy by various amounts (0.25, 0.50, 0.75, and 1.00 cm). We describe a series of techniques where the original treatment plan was adapted in order to correct for the deterioration of dose distribution quality caused by the anatomical deformations. We found that the average time needed to adapt the original plan to arrive at a clinically acceptable plan is roughly half of the time needed for a complete plan regeneration, for all four anatomical deformations. Furthermore, through modification of the DAO algorithm the optimization search space was reduced and the plan adaptation was significantly accelerated. For the first anatomical deformation (0.25 cm), the plan adaptation was six times more efficient than the complete plan regeneration. For the 0.50 and 0.75 cm deformations, the optimization efficiency was increased by a factor of roughly 3 compared to the complete plan regeneration. However, for the anatomical deformation of 1.00 cm, the reduction of the optimization search space during plan adaptation did not result in any efficiency improvement over the original (nonmodified) plan adaptation. The anatomical deformation of 1.00 cm demonstrates the limit of this approach. We propose an innovative approach to online ART in which the plan adaptation and radiation delivery are merged together and performed concurrently-adaptive radiation delivery (ARD). A fundamental advantage of ARD is the fact that radiation delivery can start almost immediately after image acquisition and evaluation. Most of the original plan adaptation is done during the radiation delivery, so the time spent adapting the original plan does not increase the overall time the patient has to spend on the treatment couch. As a consequence, the effective time allotted for plan adaptation is drastically reduced. For the 0.25, 0.5, and 0.75 cm anatomical deformations, the treatment time was increased by only 2, 4, and 6 s, respectively, as compared to no plan adaptation. For the anatomical deformation of 1.0 cm the time increase was substantially larger. The anatomical deformation of 1.0 cm represents an extreme case, which is rarely observed for the prostate, and again demonstrates the limit of this approach. ARD shows great potential for an online adaptive method with minimal extension of treatment time.

Registration of DRRs and portal images for verification of stereotactic body radiotherapy: a feasibility study in lung cancer treatment

Image guidance has become a pre-requisite for hypofractionated radiotherapy where the applied dose per fraction is increased. Particularly in stereotactic body radiotherapy (SBRT) for lung tumours, one has to account for set-up errors and intrafraction tumour motion. In our feasibility study, we compared digitally reconstructed radiographs (DRRs) of lung lesions with MV portal images (PIs) to obtain the displacement of the tumour before irradiation. The verification of the tumour position was performed by rigid intensity based registration and three different merit functions such as the sum of squared pixel intensity differences, normalized cross correlation and normalized mutual information. The registration process then provided a translation vector that defines the displacement of the target in order to align the tumour with the isocentre. To evaluate the registration algorithms, 163 test images were created and subsequently, a lung phantom containing an 8 cm(3) tumour was built. In a further step, the registration process was applied on patient data, containing 38 tumours in 113 fractions. To potentially improve registration outcome, two filter types (histogram equalization and display equalization) were applied and their impact on the registration process was evaluated. Generated test images showed an increase in successful registrations when applying a histogram equalization filter whereas the lung phantom study proved the accuracy of the selected algorithms, i.e. deviations of the calculated translation vector for all test algorithms were below 1 mm. For clinical patient data, successful registrations occurred in about 59% of anterior-posterior (AP) and 46% of lateral projections, respectively. When patients with a clinical target volume smaller than 10 cm(3) were excluded, successful registrations go up to 90% in AP and 50% in lateral projection. In addition, a reliable identification of the tumour position was found to be difficult for clinical target volumes at the periphery of the lung, close to backbone or diaphragm. Moreover, tumour movement during shallow breathing strongly influences image acquisition for patient positioning. Recapitulating, 2D/3D image registration for lung tumours is an attractive alternative compared to conventional CT verification of the tumour position. Nevertheless, size and location of the tumour are limiting parameters for an accurate registration process.

Rigid 2D/3D slice-to-volume registration and its application on fluoroscopic CT images

Registration of single slices from FluoroCT, CineMR, or interventional magnetic resonance imaging to three dimensional (3D) volumes is a special aspect of the two-dimensional (2D)/3D registration problem. Rather than digitally rendered radiographs (DRR), single 2D slice images obtained during interventional procedures are compared to oblique reformatted slices from a high resolution 3D scan. Due to the lack of perspective information and the different imaging geometry, convergence behavior differs significantly from 2D/3D registration applications comparing DRR images with conventional x-ray images. We have implemented a number of merit functions and local and global optimization algorithms for slice-to-volume registration of computed tomography (CT) and FluoroCT images. These methods were tested on phantom images derived from clinical scans for liver biopsies. Our results indicate that good registration accuracy in the range of 0.50 and 1.0 mm is achievable using simple cross correlation and repeated application of local optimization algorithms. Typically, a registration took approximately 1 min on a standard personal computer. Other merit functions such as pattern intensity or normalized mutual information did not perform as well as cross correlation in this initial evaluation. Furthermore, it appears as if the use of global optimization algorithms such as simulated annealing does not improve reliability or accuracy of the registration process. These findings were also confirmed in a preliminary registration study on five clinical scans. These experiments have, however, shown that a strict breath-hold protocol is inevitable when using rigid registration techniques for lesion localization in image-guided biopsy retrieval. Finally, further possible applications of slice-to-volume registration are discussed.