Registering histologic and MR images of prostate for image-based cancer detection

Full resolution reconstruction of whole-mount sections from digitized individual tissue fragments

Whole-mount sectioning is a technique in histopathology where a full slice of tissue, such as a transversal cross-section of a prostate specimen, is prepared on a large microscope slide without further sectioning into smaller fragments. Although this technique can offer improved correlation with pre-operative imaging and is paramount for multimodal research, it is not commonly employed due to its technical difficulty, associated cost and cumbersome integration in (digital) pathology workflows. In this work, we present a computational tool named PythoStitcher which reconstructs artificial whole-mount sections from digitized tissue fragments, thereby bringing the benefits of whole-mount sections to pathology labs currently unable to employ this technique. Our proposed algorithm consists of a multi-step approach where it (i) automatically determines how fragments need to be reassembled, (ii) iteratively optimizes the stitch using a genetic algorithm and (iii) efficiently reconstructs the final artificial whole-mount section on full resolution (0.25 µm/pixel). PythoStitcher was validated on a total of 198 cases spanning five datasets with a varying number of tissue fragments originating from different organs from multiple centers. PythoStitcher successfully reconstructed the whole-mount section in 86-100% of cases for a given dataset with a residual registration mismatch of 0.65-2.76 mm on automatically selected landmarks. It is expected that our algorithm can aid pathology labs unable to employ whole-mount sectioning through faster clinical case evaluation and improved radiology-pathology correlation workflows.

Multi-Task Weakly-Supervised Attention Network for Dementia Status Estimation With Structural MRI

Accurate prediction of clinical scores (of neuropsychological tests) based on noninvasive structural magnetic resonance imaging (MRI) helps understand the pathological stage of dementia (e.g., Alzheimer's disease (AD)) and forecast its progression. Existing machine/deep learning approaches typically preselect dementia-sensitive brain locations for MRI feature extraction and model construction, potentially leading to undesired heterogeneity between different stages and degraded prediction performance. Besides, these methods usually rely on prior anatomical knowledge (e.g., brain atlas) and time-consuming nonlinear registration for the preselection of brain locations, thereby ignoring individual-specific structural changes during dementia progression because all subjects share the same preselected brain regions. In this article, we propose a multi-task weakly-supervised attention network (MWAN) for the joint regression of multiple clinical scores from baseline MRI scans. Three sequential components are included in MWAN: 1) a backbone fully convolutional network for extracting MRI features; 2) a weakly supervised dementia attention block for automatically identifying subject-specific discriminative brain locations; and 3) an attention-aware multitask regression block for jointly predicting multiple clinical scores. The proposed MWAN is an end-to-end and fully trainable deep learning model in which dementia-aware holistic feature learning and multitask regression model construction are integrated into a unified framework. Our MWAN method was evaluated on two public AD data sets for estimating clinical scores of mini-mental state examination (MMSE), clinical dementia rating sum of boxes (CDRSB), and AD assessment scale cognitive subscale (ADAS-Cog). Quantitative experimental results demonstrate that our method produces superior regression performance compared with state-of-the-art methods. Importantly, qualitative results indicate that the dementia-sensitive brain locations automatically identified by our MWAN method well retain individual specificities and are biologically meaningful.

Correlation of in-vivo imaging with histopathology: A review

Despite tremendous advancements in in vivo imaging modalities, there remains substantial uncertainty with respect to tumor delineation on in these images. Histopathology remains the gold standard for determining the extent of malignancy, with in vivo imaging to histopathologic correlation enabling spatial comparisons. In this review, the steps necessary for successful imaging to histopathologic correlation are described, including in vivo imaging, resection, fixation, specimen sectioning (sectioning technique, securing technique, orientation matching, slice matching), microtome sectioning and staining, correlation (including image registration) and performance evaluation. The techniques used for each of these steps are also discussed. Hundreds of publications from the past 20 years were surveyed, and 62 selected for detailed analysis. For these 62 publications, each stage of the correlative pathology process (and the sub-steps of specimen sectioning) are listed. A statistical analysis was conducted based on 19 studies that reported target registration error as their performance metric. While some methods promise greater accuracy, they may be expensive. Due to the complexity of the processes involved, correlative pathology studies generally include a small number of subjects, which hinders advanced developments in this field.

Asymmetric multi-task attention network for prostate bed segmentation in computed tomography images

Post-prostatectomy radiotherapy requires accurate annotation of the prostate bed (PB), i.e., the residual tissue after the operative removal of the prostate gland, to minimize side effects on surrounding organs-at-risk (OARs). However, PB segmentation in computed tomography (CT) images is a challenging task, even for experienced physicians. This is because PB is almost a "virtual" target with non-contrast boundaries and highly variable shapes depending on neighboring OARs. In this work, we propose an asymmetric multi-task attention network (AMTA-Net) for the concurrent segmentation of PB and surrounding OARs. Our AMTA-Net mimics experts in delineating the non-contrast PB by explicitly leveraging its critical dependency on the neighboring OARs (i.e., the bladder and rectum), which are relatively easy to distinguish in CT images. Specifically, we first adopt a U-Net as the backbone network for the low-level (or prerequisite) task of the OAR segmentation. Then, we build an attention sub-network upon the backbone U-Net with a series of cascaded attention modules, which can hierarchically transfer the OAR features and adaptively learn discriminative representations for the high-level (or primary) task of the PB segmentation. We comprehensively evaluate the proposed AMTA-Net on a clinical dataset composed of 186 CT images. According to the experimental results, our AMTA-Net significantly outperforms current clinical state-of-the-arts (i.e., atlas-based segmentation methods), indicating the value of our method in reducing time and labor in the clinical workflow. Our AMTA-Net also presents better performance than the technical state-of-the-arts (i.e., the deep learning-based segmentation methods), especially for the most indistinguishable and clinically critical part of the PB boundaries. Source code is released at https://github.com/superxuang/amta-net.

Synergistic learning of lung lobe segmentation and hierarchical multi-instance classification for automated severity assessment of COVID-19 in CT images

Understanding chest CT imaging of the coronavirus disease 2019 (COVID-19) will help detect infections early and assess the disease progression. Especially, automated severity assessment of COVID-19 in CT images plays an essential role in identifying cases that are in great need of intensive clinical care. However, it is often challenging to accurately assess the severity of this disease in CT images, due to variable infection regions in the lungs, similar imaging biomarkers, and large inter-case variations. To this end, we propose a synergistic learning framework for automated severity assessment of COVID-19 in 3D CT images, by jointly performing lung lobe segmentation and multi-instance classification. Considering that only a few infection regions in a CT image are related to the severity assessment, we first represent each input image by a bag that contains a set of 2D image patches (with each cropped from a specific slice). A multi-task multi-instance deep network (called M2 UNet) is then developed to assess the severity of COVID-19 patients and also segment the lung lobe simultaneously. Our M2 UNet consists of a patch-level encoder, a segmentation sub-network for lung lobe segmentation, and a classification sub-network for severity assessment (with a unique hierarchical multi-instance learning strategy). Here, the context information provided by segmentation can be implicitly employed to improve the performance of severity assessment. Extensive experiments were performed on a real COVID-19 CT image dataset consisting of 666 chest CT images, with results suggesting the effectiveness of our proposed method compared to several state-of-the-art methods.

Histological Validation of MRI: A Review of Challenges in Registration of Imaging and Whole-Mount Histopathology

Rigorous validation with ground truth information such as histology is needed to reliably assess the current and potential value of MRI techniques to characterize tissue and identify disease-related tissue alterations. Commonly used methods that aim to directly correlate histology and MRI data generally fall short of this goal due to spatial errors that preclude direct matching. Errors result from tissue deformation, differences in spatial resolution and slice thickness, non-coplanar and/or nonintersecting plane orientations, and different image contrast mechanisms. Some of these problems arise from limitations in standard protocols for clinical tissue processing and histology-based pathology reporting, and to some extent can be addressed by modifications to standard protocols without compromising the clinical process. Typical modifications include ex vivo specimen MRI, block-face photography, addition of fiducial markers, and 3D printed molds to constrain tissue deformation and guide sectioning. This review summarizes the advantages and limitations of MRI validation techniques based on coregistration of MRI with whole-mount histology of tissue specimens. LEVEL OF EVIDENCE: 4 TECHNICAL EFFICACY STAGE: 1.

Accurate validation of ultrasound imaging of prostate cancer: a review of challenges in registration of imaging and histopathology

As the development of modalities for prostate cancer (PCa) imaging advances, the challenge of accurate registration between images and histopathologic ground truth becomes more pressing. Localization of PCa, rather than detection, requires a pixel-to-pixel validation of imaging based on histopathology after radical prostatectomy. Such a registration procedure is challenging for ultrasound modalities; not only the deformations of the prostate after resection have to be taken into account, but also the deformation due to the employed transrectal probe and the mismatch in orientation between imaging planes and pathology slices. In this work, we review the latest techniques to facilitate accurate validation of PCa localization in ultrasound imaging studies and extrapolate a general strategy for implementation of a registration procedure.

Automatic deformable registration of histological slides to μCT volume data

Localizing a histological section in the three-dimensional dataset of a different imaging modality is a challenging 2D-3D registration problem. In the literature, several approaches have been proposed to solve this problem; however, they cannot be considered as fully automatic. Recently, we developed an automatic algorithm that could successfully find the position of a histological section in a micro computed tomography (μCT) volume. For the majority of the datasets, the result of localization corresponded to the manual results. However, for some datasets, the matching μCT slice was off the ground-truth position. Furthermore, elastic distortions, due to histological preparation, could not be accounted for in this framework. In the current study, we introduce two optimization frameworks based on normalized mutual information, which enabled us to accurately register histology slides to volume data. The rigid approach allocated 81 % of histological sections with a median position error of 8.4 μm in jaw bone datasets, and the deformable approach improved registration by 33 μm with respect to the median distance error for four histological slides in the cerebellum dataset.

Intensity-based volumetric registration of magnetic resonance images and whole-mount sections of the prostate

OBJECTIVE

Magnetic Resonance Imaging (MRI) of the prostate provides useful in vivo diagnostic tissue information such as tumor location and aggressiveness, but ex vivo histopathology remains the ground truth. There are several challenges related to the registration of MRI to histopathology. We present a method for registration of standard clinical T2-weighted MRI (T2W-MRI) and transverse histopathology whole-mount (WM) sections of the prostate.

METHODS

An isotropic volume stack was created from the WM sections using 2D rigid and deformable registration combined with linear interpolation. The prostate was segmented manually from the T2W-MRI volume and registered to the WM section volume using a combination of affine and deformable registration. The method was evaluated on a set of 12 patients who had undergone radical prostatectomy. Registration accuracy was assessed using volume overlap (Dice Coefficient, DC) and landmark distances.

RESULTS

The DC was 0.94 for the whole prostate, 0.63 for the peripheral zone and 0.77 for the remaining gland. The landmark distances were on average 5.4 mm.

CONCLUSION

The volume overlap for the whole prostate and remaining gland, as well as the landmark distances indicate good registration accuracy for the proposed method, and shows that it can be highly useful for registering clinical available MRI and WM sections of the prostate.

3D volume reconstruction from serial breast specimen radiographs for mapping between histology and 3D whole specimen imaging

PURPOSE

In breast imaging, radiological in vivo images, such as x-ray mammography and magnetic resonance imaging (MRI), are used for tumor detection, diagnosis, and size determination. After excision, the specimen is typically sliced into slabs and a small subset is sampled. Histopathological imaging of the stained samples is used as the gold standard for characterization of the tumor microenvironment. A 3D volume reconstruction of the whole specimen from the 2D slabs could facilitate bridging the gap between histology and in vivo radiological imaging. This task is challenging, however, due to the large deformation that the breast tissue undergoes after surgery and the significant undersampling of the specimen obtained in histology. In this work, we present a method to reconstruct a coherent 3D volume from 2D digital radiographs of the specimen slabs.

METHODS

To reconstruct a 3D breast specimen volume, we propose the use of multiple target neighboring slices, when deforming each 2D slab radiograph in the volume, rather than performing pairwise registrations. The algorithm combines neighborhood slice information with free-form deformations, which enables a flexible, nonlinear deformation to be computed subject to the constraint that a coherent 3D volume is obtained. The neighborhood information provides adequate constraints, without the need for any additional regularization terms.

RESULTS

The volume reconstruction algorithm is validated on clinical mastectomy samples using a quantitative assessment of the volume reconstruction smoothness and a comparison with a whole specimen 3D image acquired for validation before slicing. Additionally, a target registration error of 5 mm (comparable to the specimen slab thickness of 4 mm) was obtained for five cases. The error was computed using manual annotations from four observers as gold standard, with interobserver variability of 3.4 mm. Finally, we illustrate how the reconstructed volumes can be used to map histology images to a 3D specimen image of the whole sample (either MRI or CT).

CONCLUSIONS

Qualitative and quantitative assessment has illustrated the benefit of using our proposed methodology to reconstruct a coherent specimen volume from serial slab radiographs. To our knowledge, this is the first method that has been applied to clinical breast cases, with the goal of reconstructing a whole specimen sample. The algorithm can be used as part of the pipeline of mapping histology images to ex vivo and ultimately in vivo radiological images of the breast.

Deformable image registration between pathological images and MR image via an optical macro image

Computed tomography (CT) and magnetic resonance (MR) imaging have been widely used for visualizing the inside of the human body. However, in many cases, pathological diagnosis is conducted through a biopsy or resection of an organ to evaluate the condition of tissues as definitive diagnosis. To provide more advanced information onto CT or MR image, it is necessary to reveal the relationship between tissue information and image signals. We propose a registration scheme for a set of PT images of divided specimens and a 3D-MR image by reference to an optical macro image (OM image) captured by an optical camera. We conducted a fundamental study using a resected human brain after the death of a brain cancer patient. We constructed two kinds of registration processes using the OM image as the base for both registrations to make conversion parameters between the PT and MR images. The aligned PT images had shapes similar to the OM image. On the other hand, the extracted cross-sectional MR image was similar to the OM image. From these resultant conversion parameters, the corresponding region on the PT image could be searched and displayed when an arbitrary pixel on the MR image was selected. The relationship between the PT and MR images of the whole brain can be analyzed using the proposed method. We confirmed that same regions between the PT and MR images could be searched and displayed using resultant information obtained by the proposed method. In terms of the accuracy of proposed method, the TREs were 0.56±0.39mm and 0.87±0.42mm. We can analyze the relationship between tissue information and MR signals using the proposed method.

Slice correspondence estimation using SURF descriptors and context-based search for prostate whole-mount histology MRI registration

Registration of histopathology volumes to Magnetic Resonance Images(MRI) is a crucial step for finding correlations in Prostate Cancer (PCa) and assessing tumor agressivity. This paper proposes a two-stage framework aimed at registering both modalities. Firstly, Speeded-Up Robust Features (SURF) algorithm and a context-based search is used to automatically determine slice correspondences between MRI and histology volumes. This step initializes a multimodal nonrigid registration strategy, which allows to propagate histology slices to MRI. Evaluation was performed on 5 prospective studies using a slice index score and landmark distances. With respect to a manual ground truth, the first stage of the framework exhibited an average error of 1,54 slice index and 3,51 mm in the prostate specimen. The reconstruction of a three-dimensional Whole-Mount Histology (WMH) shows promising results aimed to perform later PCa pattern detection and staging.

Online updating of context-aware landmark detectors for prostate localization in daily treatment CT images

PURPOSE

In image guided radiation therapy, it is crucial to fast and accurately localize the prostate in the daily treatment images. To this end, the authors propose an online update scheme for landmark-guided prostate segmentation, which can fully exploit valuable patient-specific information contained in the previous treatment images and can achieve improved performance in landmark detection and prostate segmentation.

METHODS

To localize the prostate in the daily treatment images, the authors first automatically detect six anatomical landmarks on the prostate boundary by adopting a context-aware landmark detection method. Specifically, in this method, a two-layer regression forest is trained as a detector for each target landmark. Once all the newly detected landmarks from new treatment images are reviewed or adjusted (if necessary) by clinicians, they are further included into the training pool as new patient-specific information to update all the two-layer regression forests for the next treatment day. As more and more treatment images of the current patient are acquired, the two-layer regression forests can be continually updated by incorporating the patient-specific information into the training procedure. After all target landmarks are detected, a multiatlas random sample consensus (multiatlas RANSAC) method is used to segment the entire prostate by fusing multiple previously segmented prostates of the current patient after they are aligned to the current treatment image. Subsequently, the segmented prostate of the current treatment image is again reviewed (or even adjusted if needed) by clinicians before including it as a new shape example into the prostate shape dataset for helping localize the entire prostate in the next treatment image.

RESULTS

The experimental results on 330 images of 24 patients show the effectiveness of the authors' proposed online update scheme in improving the accuracies of both landmark detection and prostate segmentation. Besides, compared to the other state-of-the-art prostate segmentation methods, the authors' method achieves the best performance.

CONCLUSIONS

By appropriate use of valuable patient-specific information contained in the previous treatment images, the authors' proposed online update scheme can obtain satisfactory results for both landmark detection and prostate segmentation.

A hybrid biomechanical intensity based deformable image registration of lung 4DCT

Deformable image registration (DIR) has been extensively studied over the past two decades due to its essential role in many image-guided interventions (IGI). IGI demands a highly accurate registration that maintains its accuracy across the entire region of interest. This work evaluates the improvement in accuracy and consistency by refining the results of Morfeus, a biomechanical model-based DIR algorithm. A hybrid DIR algorithm is proposed based on, a biomechanical model-based DIR algorithm and a refinement step based on a B-spline intensity-based algorithm. Inhale and exhale reconstructions of four-dimensional computed tomography (4DCT) lung images from 31 patients were initially registered using the biomechanical DIR by modeling contact surface between the lungs and the chest cavity. The resulting deformations were then refined using the intensity-based algorithm to reduce any residual uncertainties. Important parameters in the intensity-based algorithm, including grid spacing, number of pyramids, and regularization coefficient, were optimized on 10 randomly-chosen patients (out of 31). Target registration error (TRE) was calculated by measuring the Euclidean distance of common anatomical points on both images after registration. For each patient a minimum of 30 points/lung were used. Grid spacing of 8 mm, 5 levels of grid pyramids, and regularization coefficient of 3.0 were found to provide optimal results on 10 randomly chosen patients. Overall the entire patient population (n = 31), the hybrid method resulted in mean ± SD (90th%) TRE of 1.5 ± 1.4 (2.9) mm compared to 3.1 ± 1.9 (5.6) using biomechanical DIR and 2.6 ± 2.5 (6.1) using intensity-based DIR alone. The proposed hybrid biomechanical modeling intensity based algorithm is a promising DIR technique which could be used in various IGI procedures. The current investigation shows the efficacy of this approach for the registration of 4DCT images of the lungs with average accuracy of 1.5 mm.

A MR-TRUS Registration Method for Ultrasound-Guided Prostate Interventions

In this paper, we report a MR-TRUS prostate registration method that uses a subject-specific prostate strain model to improve MR-targeted, US-guided prostate interventions (e.g., biopsy and radiotherapy). The proposed algorithm combines a subject-specific prostate biomechanical model with a B-spline transformation to register the prostate gland of the MRI to the TRUS images. The prostate biomechanical model was obtained through US elastography and a 3D strain map of the prostate was generated. The B-spline transformation was calculated by minimizing Euclidean distance between the normalized attribute vectors of landmarks on MR and TRUS prostate surfaces. This prostate tissue gradient map was used to constrain the B-spline-based transformation to predict and compensate for the internal prostate-gland deformation. This method was validated with a prostate-phantom experiment and a pilot study of 5 prostate-cancer patients. For the phantom study, the mean target registration error (TRE) was 1.3 mm. MR-TRUS registration was also successfully performed for 5 patients with a mean TRE less than 2 mm. The proposed registration method may provide an accurate and robust means of estimating internal prostate-gland deformation, and could be valuable for prostate-cancer diagnosis and treatment.

Prostate whole-mount histology reconstruction and registration to MRI for correlating in-vivo observations with biological findings

Multi-parametric magnetic resonance imaging (mMRI) is the standard exam for prostate cancer diagnosis, staging and risk assessment in current clinical routine. Correlating mMRI in-vivo observations with biological findings from radical prostatectomy specimen would improve the optimal therapy selection. Thus, we proposed a method for reconstructing and registering the prostate whole-mount histology (WMH) to the MRI, considering a thin slicing of the prostatectomy specimen. The method was evaluated on 3 patients, included in a prospective study, for which hematein-eosinsafran and immunohistochemistry stainings were performed. The registration error was assessed by measuring the Euclidean distance between landmarks, previously identified by an expert on both mMRI and histological slices. The mean error was 4:90α1:34 mm. Our method demonstrated promising results for registering prostate WMH to in-vivo mMRI, thus allowing for spatial accurate correlation between radiologic observations and biological information.

Prostate CT segmentation method based on nonrigid registration in ultrasound-guided CT-based HDR prostate brachytherapy

PURPOSE

The technological advances in real-time ultrasound image guidance for high-dose-rate (HDR) prostate brachytherapy have placed this treatment modality at the forefront of innovation in cancer radiotherapy. Prostate HDR treatment often involves placing the HDR catheters (needles) into the prostate gland under the transrectal ultrasound (TRUS) guidance, then generating a radiation treatment plan based on CT prostate images, and subsequently delivering high dose of radiation through these catheters. The main challenge for this HDR procedure is to accurately segment the prostate volume in the CT images for the radiation treatment planning. In this study, the authors propose a novel approach that integrates the prostate volume from 3D TRUS images into the treatment planning CT images to provide an accurate prostate delineation for prostate HDR treatment.

METHODS

The authors' approach requires acquisition of 3D TRUS prostate images in the operating room right after the HDR catheters are inserted, which takes 1-3 min. These TRUS images are used to create prostate contours. The HDR catheters are reconstructed from the intraoperative TRUS and postoperative CT images, and subsequently used as landmarks for the TRUS-CT image fusion. After TRUS-CT fusion, the TRUS-based prostate volume is deformed to the CT images for treatment planning. This method was first validated with a prostate-phantom study. In addition, a pilot study of ten patients undergoing HDR prostate brachytherapy was conducted to test its clinical feasibility. The accuracy of their approach was assessed through the locations of three implanted fiducial (gold) markers, as well as T2-weighted MR prostate images of patients.

RESULTS

For the phantom study, the target registration error (TRE) of gold-markers was 0.41 ± 0.11 mm. For the ten patients, the TRE of gold markers was 1.18 ± 0.26 mm; the prostate volume difference between the authors' approach and the MRI-based volume was 7.28% ± 0.86%, and the prostate volume Dice overlap coefficient was 91.89% ± 1.19%.

CONCLUSIONS

The authors have developed a novel approach to improve prostate contour utilizing intraoperative TRUS-based prostate volume in the CT-based prostate HDR treatment planning, demonstrated its clinical feasibility, and validated its accuracy with MRIs. The proposed segmentation method would improve prostate delineations, enable accurate dose planning and treatment delivery, and potentially enhance the treatment outcome of prostate HDR brachytherapy.

Quantitative identification of magnetic resonance imaging features of prostate cancer response following laser ablation and radical prostatectomy

Laser interstitial thermotherapy (LITT) is a relatively new focal therapy technique for the ablation of localized prostate cancer. In this study, for the first time, we are integrating ex vivo pathology and magnetic resonance imaging (MRI) to assess the imaging characteristics of prostate cancer and treatment changes following LITT. Via a unique clinical trial, which gave us the availability of ex vivo histology and pre- and post-LITT MRIs, (1) we investigated the imaging characteristics of treatment effects and residual disease, and (2) evaluated treatment-induced feature changes in the ablated area relative to the residual disease. First, a pathologist annotated the ablated area and the residual disease on the ex vivo histology. Subsequently, we transferred the annotations to the post-LITT MRI using a semi-automatic elastic registration. The pre- and post-LITT MRIs were registered and features were extracted. A scoring metric based on the change in median pre- and post-LITT feature values was introduced, which allowed us to identify the most treatment responsive features. Our results show that (1) image characteristics for treatment effects and residual disease are different, and (2) the change of feature values between pre- and post-LITT MRIs can be a quantitative biomarker for treatment response. Finally, using feature change improved discrimination between the residual disease and treatment effects.

Comparative evaluation of registration algorithms in different brain databases with varying difficulty: results and insights

Evaluating various algorithms for the inter-subject registration of brain magnetic resonance images (MRI) is a necessary topic receiving growing attention. Existing studies evaluated image registration algorithms in specific tasks or using specific databases (e.g., only for skull-stripped images, only for single-site images, etc.). Consequently, the choice of registration algorithms seems task- and usage/parameter-dependent. Nevertheless, recent large-scale, often multi-institutional imaging-related studies create the need and raise the question whether some registration algorithms can 1) generally apply to various tasks/databases posing various challenges; 2) perform consistently well, and while doing so, 3) require minimal or ideally no parameter tuning. In seeking answers to this question, we evaluated 12 general-purpose registration algorithms, for their generality, accuracy and robustness. We fixed their parameters at values suggested by algorithm developers as reported in the literature. We tested them in 7 databases/tasks, which present one or more of 4 commonly-encountered challenges: 1) inter-subject anatomical variability in skull-stripped images; 2) intensity homogeneity, noise and large structural differences in raw images; 3) imaging protocol and field-of-view (FOV) differences in multi-site data; and 4) missing correspondences in pathology-bearing images. Totally 7,562 registrations were performed. Registration accuracies were measured by (multi-)expert-annotated landmarks or regions of interest (ROIs). To ensure reproducibility, we used public software tools, public databases (whenever possible), and we fully disclose the parameter settings. We show evaluation results, and discuss the performances in light of algorithms' similarity metrics, transformation models and optimization strategies. We also discuss future directions for the algorithm development and evaluations.

Registration of whole-mount histology and volumetric imaging of the prostate using particle filtering

Registration of histological slices to volumetric imaging of the prostate is an important task that can be used to optimize imaging for cancer detection. Such registration is challenging due to physical changes of the specimen during excision and fixation, and misalignment of the histological slices during preparation and digital scanning. In this work, we consider a multi-slice to volume registration method in which a stack of sparse, unaligned 2-D whole-mount histological slices is registered to a 3-D volumetric imaging of the prostate. We propose a particle filtering framework to contend with the high dimensionality of the search space and multimodal nature of the optimization. Such framework allows modeling of the uncertainty in the pose of the slices and in the imaged information, in order to derive optimal registration parameters in a Bayesian approach. Intensity-, region-, and point-based similarity metrics were incorporated into the registration algorithm to account for different imaging modalities. We demonstrate and evaluate our method on a diverse set of data that includes a synthetic volume, ex vivo and in vivo magnetic resonance imaging, and in vivo ultrasound.

Reconstruction of 3-dimensional histology volume and its application to study mouse mammary glands

Histology volume reconstruction facilitates the study of 3D shape and volume change of an organ at the level of macrostructures made up of cells. It can also be used to investigate and validate novel techniques and algorithms in volumetric medical imaging and therapies. Creating 3D high-resolution atlases of different organs(1,2,3) is another application of histology volume reconstruction. This provides a resource for investigating tissue structures and the spatial relationship between various cellular features. We present an image registration approach for histology volume reconstruction, which uses a set of optical blockface images. The reconstructed histology volume represents a reliable shape of the processed specimen with no propagated post-processing registration error. The Hematoxylin and Eosin (H&E) stained sections of two mouse mammary glands were registered to their corresponding blockface images using boundary points extracted from the edges of the specimen in histology and blockface images. The accuracy of the registration was visually evaluated. The alignment of the macrostructures of the mammary glands was also visually assessed at high resolution. This study delineates the different steps of this image registration pipeline, ranging from excision of the mammary gland through to 3D histology volume reconstruction. While 2D histology images reveal the structural differences between pairs of sections, 3D histology volume provides the ability to visualize the differences in shape and volume of the mammary glands.

Deformable segmentation of 3D MR prostate images via distributed discriminative dictionary and ensemble learning

PURPOSE

Automatic prostate segmentation from MR images is an important task in various clinical applications such as prostate cancer staging and MR-guided radiotherapy planning. However, the large appearance and shape variations of the prostate in MR images make the segmentation problem difficult to solve. Traditional Active Shape/Appearance Model (ASM/AAM) has limited accuracy on this problem, since its basic assumption, i.e., both shape and appearance of the targeted organ follow Gaussian distributions, is invalid in prostate MR images. To this end, the authors propose a sparse dictionary learning method to model the image appearance in a nonparametric fashion and further integrate the appearance model into a deformable segmentation framework for prostate MR segmentation.

METHODS

To drive the deformable model for prostate segmentation, the authors propose nonparametric appearance and shape models. The nonparametric appearance model is based on a novel dictionary learning method, namely distributed discriminative dictionary (DDD) learning, which is able to capture fine distinctions in image appearance. To increase the differential power of traditional dictionary-based classification methods, the authors' DDD learning approach takes three strategies. First, two dictionaries for prostate and nonprostate tissues are built, respectively, using the discriminative features obtained from minimum redundancy maximum relevance feature selection. Second, linear discriminant analysis is employed as a linear classifier to boost the optimal separation between prostate and nonprostate tissues, based on the representation residuals from sparse representation. Third, to enhance the robustness of the authors' classification method, multiple local dictionaries are learned for local regions along the prostate boundary (each with small appearance variations), instead of learning one global classifier for the entire prostate. These discriminative dictionaries are located on different patches of the prostate surface and trained to adaptively capture the appearance in different prostate zones, thus achieving better local tissue differentiation. For each local region, multiple classifiers are trained based on the randomly selected samples and finally assembled by a specific fusion method. In addition to this nonparametric appearance model, a prostate shape model is learned from the shape statistics using a novel approach, sparse shape composition, which can model nonGaussian distributions of shape variation and regularize the 3D mesh deformation by constraining it within the observed shape subspace.

RESULTS

The proposed method has been evaluated on two datasets consisting of T2-weighted MR prostate images. For the first (internal) dataset, the classification effectiveness of the authors' improved dictionary learning has been validated by comparing it with three other variants of traditional dictionary learning methods. The experimental results show that the authors' method yields a Dice Ratio of 89.1% compared to the manual segmentation, which is more accurate than the three state-of-the-art MR prostate segmentation methods under comparison. For the second dataset, the MICCAI 2012 challenge dataset, the authors' proposed method yields a Dice Ratio of 87.4%, which also achieves better segmentation accuracy than other methods under comparison.

CONCLUSIONS

A new magnetic resonance image prostate segmentation method is proposed based on the combination of deformable model and dictionary learning methods, which achieves more accurate segmentation performance on prostate T2 MR images.

Registration of in vivo prostate MRI and pseudo-whole mount histology using Local Affine Transformations guided by Internal Structures (LATIS)

PURPOSE

To present a novel registration approach called LATIS (Local Affine Transformation guided by Internal Structures) for coregistering post prostatectomy pseudo-whole mount (PWM) pathological sections with in vivo MRI (magnetic resonance imaging) images.

MATERIALS AND METHODS

Thirty-five patients with biopsy-proven prostate cancer were imaged at 3T with an endorectal coil. Excised prostate specimens underwent quarter mount step-section pathologic processing, digitization, annotation, and assembly into a PWM. Manually annotated macro-structures on both pathology and MRI were used to assist registration using a relaxed local affine transformation approximation. Registration accuracy was assessed by calculation of the Dice similarity coefficient (DSC) between transformed and target capsule masks and least-square distance between transformed and target landmark positions.

RESULTS

LATIS registration resulted in a DSC value of 0.991 ± 0.004 and registration accuracy of 1.54 ± 0.64 mm based on identified landmarks common to both datasets. Image registration performed without the use of internal structures led to an 87% increase in landmark-based registration error. Derived transformation matrices were used to map regions of pathologically defined disease to MRI.

CONCLUSION

LATIS was used to successfully coregister digital pathology with in vivo MRI to facilitate improved correlative studies between pathologically identified features of prostate cancer and multiparametric MRI.

3D prostate histology reconstruction: an evaluation of image-based and fiducial-based algorithms

PURPOSE

Evaluation of in vivo prostate imaging modalities for determining the spatial distribution and aggressiveness of prostate cancer ideally requires accurate registration of images to an accepted reference standard, such as histopathological examination of radical prostatectomy specimens. Three-dimensional (3D) reconstruction of prostate histology facilitates these registration-based evaluations by reintroducing 3D spatial information lost during histology processing. Because the reconstruction accuracy may constrain the clinical questions that can be answered with these data, it is important to assess the tradeoffs between minimally disruptive methods based on intrinsic image information and potentially more robust methods based on extrinsic fiducial markers.

METHODS

Ex vivo magnetic resonance (MR) images and digitized whole-mount histology images from 12 radical prostatectomy specimens were used to evaluate four 3D histology reconstruction algorithms. 3D reconstructions were computed by registering each histology image to the corresponding ex vivo MR image using one of two similarity metrics (mutual information or fiducial registration error) and one of two search domains (affine transformations or a constrained subset thereof). The algorithms were evaluated for accuracy using the mean target registration error (TRE) computed from homologous intrinsic point landmarks (3-16 per histology section; 232 total) identified on histology and MR images, and for the sensitivity of TRE to rotational, translational, and scaling initialization errors.

RESULTS

The algorithms using fiducial registration error and mutual information had mean ± standard deviation TREs of 0.7 ± 0.4 and 1.2 ± 0.7 mm, respectively, and one algorithm using fiducial registration error and affine transforms had negligible sensitivities to initialization errors. The postoptimization values of the mutual information-based metric showed evidence of errors due to both the optimizer and the similarity metric, and variation of parameters of the mutual information-based metric did not improve its performance.

CONCLUSIONS

The extrinsic fiducial-based algorithm had lower mean TRE and lower sensitivity to initialization than the intrinsic intensity-based algorithm using mutual information. A model relating statistical power to registration error for certain imaging validation study designs estimated that a reconstruction algorithm with a mean TRE of 0.7 mm would require 27% fewer subjects than the method used to initialize the algorithms (mean TRE 1.3 ± 0.7 mm), suggesting the choice of reconstruction technique can have a substantial impact on the design of imaging validation studies, and on their overall cost.

Incremental learning with selective memory (ILSM): towards fast prostate localization for image guided radiotherapy

Image-guided radiotherapy (IGRT) requires fast and accurate localization of the prostate in 3-D treatment-guided radiotherapy, which is challenging due to low tissue contrast and large anatomical variation across patients. On the other hand, the IGRT workflow involves collecting a series of computed tomography (CT) images from the same patient under treatment. These images contain valuable patient-specific information yet are often neglected by previous works. In this paper, we propose a novel learning framework, namely incremental learning with selective memory (ILSM), to effectively learn the patient-specific appearance characteristics from these patient-specific images. Specifically, starting with a population-based discriminative appearance model, ILSM aims to "personalize" the model to fit patient-specific appearance characteristics. The model is personalized with two steps: backward pruning that discards obsolete population-based knowledge and forward learning that incorporates patient-specific characteristics. By effectively combining the patient-specific characteristics with the general population statistics, the incrementally learned appearance model can localize the prostate of a specific patient much more accurately. This work has three contributions: 1) the proposed incremental learning framework can capture patient-specific characteristics more effectively, compared to traditional learning schemes, such as pure patient-specific learning, population-based learning, and mixture learning with patient-specific and population data; 2) this learning framework does not have any parametric model assumption, hence, allowing the adoption of any discriminative classifier; and 3) using ILSM, we can localize the prostate in treatment CTs accurately (DSC  ∼ 0.89 ) and fast (  ∼ 4 s), which satisfies the real-world clinical requirements of IGRT.

Contrast enhanced-magnetic resonance imaging as a surrogate to map verteporfin delivery in photodynamic therapy

The use of in vivo contrast-enhanced magnetic resonance (MR) imaging as a surrogate for photosensitizer (verteporfin) dosimetry in photodynamic therapy of pancreas cancer is demonstrated by correlating MR contrast uptake to ex vivo fluorescence images on excised tissue. An orthotopic pancreatic xenograft mouse model was used for the study. A strong correlation (r = 0.57) was found for bulk intensity measurements of T1-weighted gadolinium enhancement and verteporfin fluorescence in the tumor region of interest. The use of contrast-enhanced MR imaging shows promise as a method for treatment planning and photosensitizer dosimetry in human photodynamic therapy (PDT) of pancreas cancer.

3D prostate histology image reconstruction: Quantifying the impact of tissue deformation and histology section location

Background:

Guidelines for localizing prostate cancer on imaging are ideally informed by registered post-prostatectomy histology. 3D histology reconstruction methods can support this by reintroducing 3D spatial information lost during histology processing. The need to register small, high-grade foci drives a need for high accuracy. Accurate 3D reconstruction method design is impacted by the answers to the following central questions of this work. (1) How does prostate tissue deform during histology processing? (2) What spatial misalignment of the tissue sections is induced by microtome cutting? (3) How does the choice of reconstruction model affect histology reconstruction accuracy?

Materials and Methods:

Histology, paraffin block face and magnetic resonance images were acquired for 18 whole mid-gland tissue slices from six prostates. 7-15 homologous landmarks were identified on each image. Tissue deformation due to histology processing was characterized using the target registration error (TRE) after landmark-based registration under four deformation models (rigid, similarity, affine and thin-plate-spline [TPS]). The misalignment of histology sections from the front faces of tissue slices was quantified using manually identified landmarks. The impact of reconstruction models on the TRE after landmark-based reconstruction was measured under eight reconstruction models comprising one of four deformation models with and without constraining histology images to the tissue slice front faces.

Results:

Isotropic scaling improved the mean TRE by 0.8-1.0 mm (all results reported as 95% confidence intervals), while skew or TPS deformation improved the mean TRE by <0.1 mm. The mean misalignment was 1.1-1.9^° (angle) and 0.9-1.3 mm (depth). Using isotropic scaling, the front face constraint raised the mean TRE by 0.6-0.8 mm.

Conclusions:

For sub-millimeter accuracy, 3D reconstruction models should not constrain histology images to the tissue slice front faces and should be flexible enough to model isotropic scaling.

Model-based registration of ex vivo and in vivo MRI of the prostate using elastography*

Registration of histopathology to in vivo magnetic resonance imaging (MRI) of the prostate is an important task that can be used to optimize in vivo imaging for cancer detection. Such registration is challenging due to the change in volume and deformation of the prostate during excision and fixation. One approach towards this problem involves the use of an ex vivo MRI of the excised prostate specimen, followed by in vivo to ex vivo MRI registration of the prostate. We propose a novel registration method that uses a patient-specific biomechanical model acquired using magnetic resonance elastography to deform the in vivo volume and match it to the surface of the ex vivo specimen. The forces that drive the deformations are derived from a region-based energy, with the elastic potential used for regularization. The incorporation of elastography data into the registration framework allows inhomogeneous elasticity to be assigned to the in vivo volume. We show that such inhomogeneity improves the registration results by providing a physical regularization of the deformation map. The method is demonstrated and evaluated on six clinical cases.

3D reconstruction of 2D fluorescence histology images and registration with in vivo MR images: application in a rodent stroke model

To validate and add value to non-invasive imaging techniques, the corresponding histology is required to establish biological correlates. We present an efficient, semi-automated image-processing pipeline that uses immunohistochemically stained sections to reconstruct a 3D brain volume from 2D histological images before registering these with the corresponding 3D in vivo magnetic resonance images (MRI). A multistep registration procedure that first aligns the "global" volume by using the centre of mass and then applies a rigid and affine alignment based on signal intensities is described. This technique was applied to a training set of three rat brain volumes before being validated on three normal brains. Application of the approach to register "abnormal" images from a rat model of stroke allowed the neurobiological correlates of the variations in the hyper-intense MRI signal intensity caused by infarction to be investigated. For evaluation, the corresponding anatomical landmarks in MR and histology were defined to measure the registration accuracy. A registration error of 0.249 mm (approximately one in-plane voxel dimension) was evident in healthy rat brains and of 0.323 mm in a rodent model of stroke. The proposed reconstruction and registration pipeline allowed for the precise analysis of non-invasive MRI and corresponding microstructural histological features in 3D. We were thus able to interrogate histology to deduce the cause of MRI signal variations in the lesion cavity and the peri-infarct area.

Model-based registration of ex vivo and in vivo MRI of the prostate using elastography

Registration of histopathology to in vivo magnetic resonance imaging (MRI) of the prostate is an important task that can be used to optimize in vivo imaging for cancer detection. Such registration is challenging due to the change in volume and deformation of the prostate during excision and fixation. One approach towards this problem involves the use of an ex vivo MRI of the excised prostate specimen, followed by in vivo to ex vivo MRI registration of the prostate. We propose a novel registration method that uses a patient-specific biomechanical model acquired using magnetic resonance elastography to deform the in vivo volume and match it to the surface of the ex vivo specimen. The forces that drive the deformations are derived from a region-based energy, with the elastic potential used for regularization. The incorporation of elastography data into the registration framework allows inhomogeneous elasticity to be assigned to the in vivo volume. We show that such inhomogeneity improves the registration results by providing a physical regularization of the deformation map. The method is demonstrated and evaluated on six clinical cases.

Sparse patch-based label propagation for accurate prostate localization in CT images

In this paper, we propose a new prostate computed tomography (CT) segmentation method for image guided radiation therapy. The main contributions of our method lie in the following aspects. 1) Instead of using voxel intensity information alone, patch-based representation in the discriminative feature space with logistic sparse LASSO is used as anatomical signature to deal with low contrast problem in prostate CT images. 2) Based on the proposed patch-based signature, a new multi-atlases label fusion method formulated under sparse representation framework is designed to segment prostate in the new treatment images, with guidance from the previous segmented images of the same patient. This method estimates the prostate likelihood of each voxel in the new treatment image from its nearby candidate voxels in the previous segmented images, based on the nonlocal mean principle and sparsity constraint. 3) A hierarchical labeling strategy is further designed to perform label fusion, where voxels with high confidence are first labeled for providing useful context information in the same image for aiding the labeling of the remaining voxels. 4) An online update mechanism is finally adopted to progressively collect more patient-specific information from newly segmented treatment images of the same patient, for adaptive and more accurate segmentation. The proposed method has been extensively evaluated on a prostate CT image database consisting of 24 patients where each patient has more than 10 treatment images, and further compared with several state-of-the-art prostate CT segmentation algorithms using various evaluation metrics. Experimental results demonstrate that the proposed method consistently achieves higher segmentation accuracy than any other methods under comparison.

Prostate segmentation by sparse representation based classification

PURPOSE

The segmentation of prostate in CT images is of essential importance to external beam radiotherapy, which is one of the major treatments for prostate cancer nowadays. During the radiotherapy, the prostate is radiated by high-energy x rays from different directions. In order to maximize the dose to the cancer and minimize the dose to the surrounding healthy tissues (e.g., bladder and rectum), the prostate in the new treatment image needs to be accurately localized. Therefore, the effectiveness and efficiency of external beam radiotherapy highly depend on the accurate localization of the prostate. However, due to the low contrast of the prostate with its surrounding tissues (e.g., bladder), the unpredicted prostate motion, and the large appearance variations across different treatment days, it is challenging to segment the prostate in CT images. In this paper, the authors present a novel classification based segmentation method to address these problems.

METHODS

To segment the prostate, the proposed method first uses sparse representation based classification (SRC) to enhance the prostate in CT images by pixel-wise classification, in order to overcome the limitation of poor contrast of the prostate images. Then, based on the classification results, previous segmented prostates of the same patient are used as patient-specific atlases to align onto the current treatment image and the majority voting strategy is finally adopted to segment the prostate. In order to address the limitations of the traditional SRC in pixel-wise classification, especially for the purpose of segmentation, the authors extend SRC from the following four aspects: (1) A discriminant subdictionary learning method is proposed to learn a discriminant and compact representation of training samples for each class so that the discriminant power of SRC can be increased and also SRC can be applied to the large-scale pixel-wise classification. (2) The L1 regularized sparse coding is replaced by the elastic net in order to obtain a smooth and clear prostate boundary in the classification result. (3) Residue-based linear regression is incorporated to improve the classification performance and to extend SRC from hard classification to soft classification. (4) Iterative SRC is proposed by using context information to iteratively refine the classification results.

RESULTS

The proposed method has been comprehensively evaluated on a dataset consisting of 330 CT images from 24 patients. The effectiveness of the extended SRC has been validated by comparing it with the traditional SRC based on the proposed four extensions. The experimental results show that our extended SRC can obtain not only more accurate classification results but also smoother and clearer prostate boundary than the traditional SRC. Besides, the comparison with other five state-of-the-art prostate segmentation methods indicates that our method can achieve better performance than other methods under comparison.

CONCLUSIONS

The authors have proposed a novel prostate segmentation method based on the sparse representation based classification, which can achieve considerably accurate segmentation results in CT prostate segmentation.

Decision support system for localizing prostate cancer based on multiparametric magnetic resonance imaging

PURPOSE

There is a growing need to localize prostate cancers on magnetic resonance imaging (MRI) to facilitate the use of image guided biopsy, focal therapy, and active surveillance follow up. Our goal was to develop a decision support system (DSS) for detecting and localizing peripheral zone prostate cancers by using machine learning approach to calculate a cancer probability map from multiparametric MR images (MP-MRI).

METHODS

This IRB approved Health Insurance Portability and Accountability Act compliant retrospective study consisted of 31 patients (mean age and serum prostate specific antigen of 60.4 and 6.62 ng∕ml, respectively) who had MP-MRI at 3 T followed by radical prostatectomy. Seven patients were excluded due to technical issues with their MP-MRI (e.g., motion artifact, failure to perform all sequences). Cancer and normal regions were identified in the peripheral zone by correlating them to whole mount histology slides of the excised prostatectomy specimens. To facilitate the correlation, tissue blocks matching the MR slices were obtained using a MR-based patient-specific mold. Segmented regions on the MP-MRI were correlated to histopathology and used as training sets for the learning system that generated the cancer probability maps. Leave-one-patient-out cross-validation on the cancer and normal regions was performed to determine the learning system's efficacy, an evolutionary strategies approach (also known as a genetic algorithm) was used to find the optimal values for a set of parameters, and finally a cancer probability map was generated.

RESULTS

For the 24 patients that were used in the study, 225 cancer and 264 noncancerous regions were identified from the region maps. The efficacy of DSS was first determined without optimizing support vector machines (SVM) parameters, where a region having a cancer probability greater than or equal to 50% was considered as a correct classification. The nonoptimized system had an f-measure of 85% and the Kappa coefficient of 71% (Rater's agreement, where raters are DSS and ground truth histology). The efficacy of the DSS after optimizing SVM parameters using a genetic algorithm had an f-measure of 89% and a Kappa coefficient of 80%. Thus, after optimization of the DSS there was a 4% increase in the f-measure and a 9% increase in the Kappa coefficient.

CONCLUSIONS

This DSS provides a cancer probability map for peripheral zone prostate tumors based on endorectal MP-MRI. These cancer probability maps can potentially aid radiologists in accurately localizing peripheral zone prostate cancers for planning targeted biopsies, focal therapy, and follow up for active surveillance.

Angiogenesis imaging by spatiotemporal analysis of ultrasound contrast agent dispersion kinetics

The key role of angiogenesis in cancer growth has motivated extensive research with the goal of noninvasive cancer detection by blood perfusion imaging. However, the results are still limited and the diagnosis of major forms of cancer, such as prostate cancer, are currently based on systematic biopsies. The difficulty in the detection of angiogenesis partly resides in a complex relationship between angiogenesis and perfusion. This may be overcome by analysis of the dispersion kinetics of ultrasound contrast agents. Determined by multipath trajectories through the microvasculature, dispersion permits a better characterization of the microvascular architecture and, therefore, more accurate detection of angiogenesis. In this paper, a novel dispersion analysis method is proposed for prostate cancer localization. An ultrasound contrast agent bolus is injected intravenously. Spatiotemporal analysis of the concentration evolution measured at different pixels in the prostate is used to assess the local dispersion kinetics of the injected agent. In particular, based on simulations of the convective diffusion equation, the similarity between the concentration evolutions at neighbor pixels is the adopted dispersion measure. Six measurements in patients, compared with the histology, provided a receiver operating characteristic curve integral equal to 0.87. This result was superior to that obtained by the previous approaches reported in the literature.

High-resolution MRI of excised human prostate specimens acquired with 9.4T in detection and identification of cancers: validation of a technique

PURPOSE

To evaluate feasibility of high-resolution, high-field ex vivo prostate magnetic resonance imaging (MRI) as an aid to guide pathologists' examination and develop in vivo MRI methods.

MATERIALS AND METHODS

Unfixed excised prostatectomy specimens (n = 9) were obtained and imaged immediately after radical prostatectomy under an Institutional Review Board-approved protocol. High-resolution T2-weighted (T2W) MRI of specimens were acquired with a Bruker 9.4 T scanner to correlate with whole-mount histology. Additionally, T2 and apparent diffusion coefficient (ADC) maps were generated.

RESULTS

By visual inspection of the nine prostate specimens imaged, high-resolution T2W MRI showed improved anatomical detail compared to published low-resolution images acquired at 4 T as published by other investigators. Benign prostatic hyperplasia, adenocarcinomas, curvilinear duct architecture distortion due to adenocarcinomas, and normal radial duct distribution were readily identified. T2 was ≈10 msec longer (P < 0.03) and the ADC was ≈1.4 times larger (P < 0.002) in the normal peripheral zone compared to the peripheral zone with prostate cancer.

CONCLUSION

Differences in T2 and ADC between benign and malignant tissue are consistent with in vivo data. High-resolution, high-field MRI has the potential to improve the detection and identification of prostate structures. The protocols and techniques developed in this study could augment routine pathological analysis of surgical specimens and guide treatment of prostate cancer patients.

Multiparametric MRI maps for detection and grading of dominant prostate tumors

PURPOSE

To develop an image-based technique capable of detection and grading of prostate cancer, which combines features extracted from multiparametric MRI into a single parameter map of cancer probability.

MATERIALS AND METHODS

A combination of features extracted from diffusion tensor MRI and dynamic contrast enhanced MRI was used to characterize biopsy samples from 29 patients. Support vector machines were used to separate the cancerous samples from normal biopsy samples and to compute a measure of cancer probability, presented in the form of a cancer colormap. The classification results were compared with the biopsy results and the classifier was tuned to provide the largest area under the receiver operating characteristic (ROC) curve. Based solely on the tuning of the classifier on the biopsy data, cancer colormaps were also created for whole-mount histopathology slices from four radical prostatectomy patients.

RESULTS

An area under ROC curve of 0.96 was obtained on the biopsy dataset and was validated by a "leave-one-patient-out" procedure. The proposed measure of cancer probability shows a positive correlation with Gleason score. The cancer colormaps created for the histopathology patients do display the dominant tumors. The colormap accuracy increases with measured tumor area and Gleason score.

CONCLUSION

Dynamic contrast enhanced imaging and diffusion tensor imaging, when used within the framework of supervised classification, can play a role in characterizing prostate cancer.

Biomechanical model-based deformable registration of MRI and histopathology for clinical prostatectomy

A biomechanical model-based deformable image registration incorporating specimen-specific changes in material properties is optimized and evaluated for correlating histology of clinical prostatectomy specimens with in vivo MRI. In this methodology, a three-step registration based on biomechanics calculates the transformations between histology and fixed, fixed and fresh, and fresh and in vivo states. A heterogeneous linear elastic material model is constructed based on magnetic resonance elastography (MRE) results. The ex vivo tissue MRE data provide specimen-specific information for the fresh and fixed tissue to account for the changes due to fixation. The accuracy of the algorithm was quantified by calculating the target registration error (TRE) by identifying naturally occurring anatomical points within the prostate in each image. TRE were improved with the deformable registration algorithm compared to rigid registration alone. The qualitative assessment also showed a good alignment between histology and MRI after the proposed deformable registration.

Robust automatic knee MR slice positioning through redundant and hierarchical anatomy detection

Diagnostic magnetic resonance (MR) image quality is highly dependent on the position and orientation of the slice groups, due to the intrinsic high in-slice and low through-slice resolutions of MR imaging. Hence, the higher speed, accuracy, and reproducibility of automatic slice positioning, make it highly desirable over manual slice positioning. However, imaging artifacts, diseases, joint articulation, variations across ages and demographics as well as the extremely high performance requirements prevent state-of-the-art methods, such as volumetric registration, to be an off-the-shelf solution. In this paper, we address all these issues through an automatic slice positioning framework based on redundant and hierarchical learning. Our method has two hallmarks that are specifically designed to achieve high robustness and accuracy. 1) A redundant set of anatomy detectors are learned to provide local appearance cues. These detections are pruned and assembled according to a distributed anatomy model, which captures group-wise spatial configurations among anatomy primitives. This strategy brings about a high level of robustness and works even if a large portion of the target is distorted, missing, or occluded. 2) The detectors are learned and invoked in a hierarchical fashion, with each local detection scheduled and iterated according to its intrinsic invariance property. This iterative alignment process is shown to dramatically improve alignment accuracy. The proposed system is extensively validated on a large dataset including 744 clinical MR scans. Compared to state-of-the-art methods, our method exhibits superior performance in terms of robustness, accuracy, and reproducibility. The methodology is general and can be applied to other anatomies and other imaging modalities.

Facilitating tumor functional assessment by spatially relating 3D tumor histology and in vivo MRI: image registration approach

Background

Magnetic resonance imaging (MRI), together with histology, is widely used to diagnose and to monitor treatment in oncology. Spatial correspondence between these modalities provides information about the ability of MRI to characterize cancerous tissue. However, registration is complicated by deformations during pathological processing, and differences in scale and information content.

Methodology/Principal Findings

This study proposes a methodology for establishing an accurate 3D relation between histological sections and high resolution in vivo MRI tumor data. The key features of the methodology are: 1) standardized acquisition and processing, 2) use of an intermediate ex vivo MRI, 3) use of a reference cutting plane, 4) dense histological sampling, 5) elastic registration, and 6) use of complete 3D data sets. Five rat pancreatic tumors imaged by T2*-w MRI were used to evaluate the proposed methodology. The registration accuracy was assessed by root mean squared (RMS) distances between manually annotated landmark points in both modalities. After elastic registration the average RMS distance decreased from 1.4 to 0.7 mm. The intermediate ex vivo MRI and the reference cutting plane shared by all three 3D images (in vivo MRI, ex vivo MRI, and 3D histology data) were found to be crucial for the accurate co-registration between the 3D histological data set and in vivo MRI. The MR intensity in necrotic regions, as manually annotated in 3D histology, was significantly different from other histologically confirmed regions (i.e., viable and hemorrhagic). However, the viable and the hemorrhagic regions showed a large overlap in T2^*-w MRI signal intensity.

Conclusions

The established 3D correspondence between tumor histology and in vivo MRI enables extraction of MRI characteristics for histologically confirmed regions. The proposed methodology allows the creation of a tumor database of spatially registered multi-spectral MR images and multi-stained 3D histology.

DRAMMS: Deformable registration via attribute matching and mutual-saliency weighting

A general-purpose deformable registration algorithm referred to as "DRAMMS" is presented in this paper. DRAMMS bridges the gap between the traditional voxel-wise methods and landmark/feature-based methods with primarily two contributions. First, DRAMMS renders each voxel relatively distinctively identifiable by a rich set of attributes, therefore largely reducing matching ambiguities. In particular, a set of multi-scale and multi-orientation Gabor attributes are extracted and the optimal components are selected, so that they form a highly distinctive morphological signature reflecting the anatomical and geometric context around each voxel. Moreover, the way in which the optimal Gabor attributes are constructed is independent of the underlying image modalities or contents, which renders DRAMMS generally applicable to diverse registration tasks. A second contribution of DRAMMS is that it modulates the registration by assigning higher weights to those voxels having higher ability to establish unique (hence reliable) correspondences across images, therefore reducing the negative impact of those regions that are less capable of finding correspondences (such as outlier regions). A continuously-valued weighting function named "mutual-saliency" is developed to reflect the matching uniqueness between a pair of voxels implied by the tentative transformation. As a result, voxels do not contribute equally as in most voxel-wise methods, nor in isolation as in landmark/feature-based methods. Instead, they contribute according to the continuously-valued mutual-saliency map, which dynamically evolves during the registration process. Experiments in simulated images, inter-subject images, single-/multi-modality images, from brain, heart, and prostate have demonstrated the general applicability and the accuracy of DRAMMS.

Elastic registration of multimodal prostate MRI and histology via multiattribute combined mutual information

PURPOSE

By performing registration of preoperative multiprotocol in vivo magnetic resonance (MR) images of the prostate with corresponding whole-mount histology (WMH) sections from postoperative radical prostatectomy specimens, an accurate estimate of the spatial extent of prostate cancer (CaP) on in vivo MR imaging (MRI) can be retrospectively established. This could allow for definition of quantitative image-based disease signatures and lead to development of classifiers for disease detection on multiprotocol in vivo MRI. Automated registration of MR and WMH images of the prostate is complicated by dissimilar image intensities, acquisition artifacts, and nonlinear shape differences.

METHODS

The authors present a method for automated elastic registration of multiprotocol in vivo MRI and WMH sections of the prostate. The method, multiattribute combined mutual information (MACMI), leverages all available multiprotocol image data to drive image registration using a multivariate formulation of mutual information.

RESULTS

Elastic registration using the multivariate MI formulation is demonstrated for 150 corresponding sets of prostate images from 25 patient studies with T2-weighted and dynamic-contrast enhanced MRI and 85 image sets from 15 studies with an additional functional apparent diffusion coefficient MRI series. Qualitative results of MACMI evaluation via visual inspection suggest that an accurate delineation of CaP extent on MRI is obtained. Results of quantitative evaluation on 150 clinical and 20 synthetic image sets indicate improved registration accuracy using MACMI compared to conventional pairwise mutual information-based approaches.

CONCLUSIONS

The authors' approach to the registration of in vivo multiprotocol MRI and ex vivo WMH of the prostate using MACMI is unique, in that (1) information from all available image protocols is utilized to drive the registration with histology, (2) no additional, intermediate ex vivo radiology or gross histology images need be obtained in addition to the routinely acquired in vivo MRI series, and (3) no corresponding anatomical landmarks are required to be identified manually or automatically on the images.

3D Non-rigid Registration Using Surface and Local Salient Features for Transrectal Ultrasound Image-guided Prostate Biopsy

We present a 3D non-rigid registration algorithm for the potential use in combining PET/CT and transrectal ultrasound (TRUS) images for targeted prostate biopsy. Our registration is a hybrid approach that simultaneously optimizes the similarities from point-based registration and volume matching methods. The 3D registration is obtained by minimizing the distances of corresponding points at the surface and within the prostate and by maximizing the overlap ratio of the bladder neck on both images. The hybrid approach not only capture deformation at the prostate surface and internal landmarks but also the deformation at the bladder neck regions. The registration uses a soft assignment and deterministic annealing process. The correspondences are iteratively established in a fuzzy-to-deterministic approach. B-splines are used to generate a smooth non-rigid spatial transformation. In this study, we tested our registration with pre- and post-biopsy TRUS images of the same patients. Registration accuracy is evaluated using manual defined anatomic landmarks, i.e. calcification. The root-mean-squared (RMS) of the difference image between the reference and floating images was decreased by 62.6±9.1% after registration. The mean target registration error (TRE) was 0.88±0.16 mm, i.e. less than 3 voxels with a voxel size of 0.38×0.38×0.38 mm3 for all five patients. The experimental results demonstrate the robustness and accuracy of the 3D non-rigid registration algorithm.