Spatially weighted mutual information (SWMI) for registration of digitally reconstructed ex vivo whole mount histology and in vivo prostate MRI

Multi-scale statistical deformation based co-registration of prostate MRI and post-surgical whole mount histopathology

BACKGROUND

Accurate delineations of regions of interest (ROIs) on multi-parametric magnetic resonance imaging (mpMRI) are crucial for development of automated, machine learning-based prostate cancer (PCa) detection and segmentation models. However, manual ROI delineations are labor-intensive and susceptible to inter-reader variability. Histopathology images from radical prostatectomy (RP) represent the "gold standard" in terms of the delineation of disease extents, for example, PCa, prostatitis, and benign prostatic hyperplasia (BPH). Co-registering digitized histopathology images onto pre-operative mpMRI enables automated mapping of the ground truth disease extents onto mpMRI, thus enabling the development of machine learning tools for PCa detection and risk stratification. Still, MRI-histopathology co-registration is challenging due to various artifacts and large deformation between in vivo MRI and ex vivo whole-mount histopathology images (WMHs). Furthermore, the artifacts on WMHs, such as tissue loss, may introduce unrealistic deformation during co-registration.

PURPOSE

This study presents a new registration pipeline, MSERgSDM, a multi-scale feature-based registration (MSERg) with a statistical deformation (SDM) constraint, which aims to improve accuracy of MRI-histopathology co-registration.

METHODS

In this study, we collected 85 pairs of MRI and WMHs from 48 patients across three cohorts. Cohort 1 (D1), comprised of a unique set of 3D printed mold data from six patients, facilitated the generation of ground truth deformations between ex vivo WMHs and in vivo MRI. The other two clinically acquired cohorts (D2 and D3) included 42 patients. Affine and nonrigid registrations were employed to minimize the deformation between ex vivo WMH and ex vivo T2-weighted MRI (T2WI) in D1. Subsequently, ground truth deformation between in vivo T2WI and ex vivo WMH was approximated as the deformation between in vivo T2WI and ex vivo T2WI. In D2 and D3, the prostate anatomical annotations, for example, tumor and urethra, were made by a pathologist and a radiologist in collaboration. These annotations included ROI boundary contours and landmark points. Before applying the registration, manual corrections were made for flipping and rotation of WMHs. MSERgSDM comprises two main components: (1) multi-scale representation construction, and (2) SDM construction. For the SDM construction, we collected N = 200 reasonable deformation fields generated using MSERg, verified through visual inspection. Three additional methods, including intensity-based registration, ProsRegNet, and MSERg, were also employed for comparison against MSERgSDM.

RESULTS

Our results suggest that MSERgSDM performed comparably to the ground truth (p > 0.05). Additionally, MSERgSDM (ROI Dice ratio = 0.61, landmark distance = 3.26 mm) exhibited significant improvement over MSERg (ROI Dice ratio = 0.59, landmark distance = 3.69 mm) and ProsRegNet (ROI Dice ratio = 0.56, landmark distance = 4.00 mm) in local alignment.

CONCLUSIONS

This study presents a novel registration method, MSERgSDM, for mapping ex vivo WMH onto in vivo prostate MRI. Our preliminary results demonstrate that MSERgSDM can serve as a valuable tool to map ground truth disease annotations from histopathology images onto MRI, thereby assisting in the development of machine learning models for PCa detection on MRI.

Prostate cancer detection and segmentation on MRI using non-local mask R-CNN with histopathological ground truth

BACKGROUND

Automatic detection and segmentation of intraprostatic lesions (ILs) on preoperative multiparametric-magnetic resonance images (mp-MRI) can improve clinical workflow efficiency and enhance the diagnostic accuracy of prostate cancer and is an essential step in dominant intraprostatic lesion boost.

PURPOSE

The goal is to improve the detection and segmentation accuracy of 3D ILs in MRI by a proposed a deep learning (DL)-based algorithm with histopathological ground truth.

METHODS

This retrospective study included 262 patients with in vivo prostate biparametric MRI (bp-MRI) scans and were divided into three cohorts based on their data analysis and annotation. Histopathological ground truth was established by using histopathology images as delineation reference standard on cohort 1, which consisted of 64 patients and was randomly split into 20 training, 12 validation, and 32 testing patients. Cohort 2 consisted of 158 patients with bp-MRI based lesion delineation, and was randomly split into 104 training, 15 validation, and 39 testing patients. Cohort 3 consisted of 40 unannotated patients, used in semi-supervised learning. We proposed a non-local Mask R-CNN and boosted its performance by applying different training techniques. The performance of non-local Mask R-CNN was compared with baseline Mask R-CNN, 3D U-Net and an experienced radiologist's delineation and was evaluated by detection rate, dice similarity coefficient (DSC), sensitivity, and Hausdorff Distance (HD).

RESULTS

The independent testing set consists of 32 patients with histopathological ground truth. With the training technique maximizing detection rate, the non-local Mask R-CNN achieved 80.5% and 94.7% detection rate; 0.548 and 0.604 DSC; 5.72 and 6.36 95 HD (mm); 0.613 and 0.580 sensitivity for ILs of all Gleason Grade groups (GGGs) and clinically significant ILs (GGG > 2), which outperformed baseline Mask R-CNN and 3D U-Net. For clinically significant ILs, the model segmentation accuracy was significantly higher than that of the experienced radiologist involved in the study, who achieved 0.512 DSC (p = 0.04), 8.21 (p = 0.041) 95 HD (mm), and 0.398 (p = 0.001) sensitivity.

CONCLUSION

The proposed DL model achieved state-of-art performance and has the potential to help improve radiotherapy treatment planning and noninvasive prostate cancer diagnosis.

3D Registration of pre-surgical prostate MRI and histopathology images via super-resolution volume reconstruction

The use of MRI for prostate cancer diagnosis and treatment is increasing rapidly. However, identifying the presence and extent of cancer on MRI remains challenging, leading to high variability in detection even among expert radiologists. Improvement in cancer detection on MRI is essential to reducing this variability and maximizing the clinical utility of MRI. To date, such improvement has been limited by the lack of accurately labeled MRI datasets. Data from patients who underwent radical prostatectomy enables the spatial alignment of digitized histopathology images of the resected prostate with corresponding pre-surgical MRI. This alignment facilitates the delineation of detailed cancer labels on MRI via the projection of cancer from histopathology images onto MRI. We introduce a framework that performs 3D registration of whole-mount histopathology images to pre-surgical MRI in three steps. First, we developed a novel multi-image super-resolution generative adversarial network (miSRGAN), which learns information useful for 3D registration by producing a reconstructed 3D MRI. Second, we trained the network to learn information between histopathology slices to facilitate the application of 3D registration methods. Third, we registered the reconstructed 3D histopathology volumes to the reconstructed 3D MRI, mapping the extent of cancer from histopathology images onto MRI without the need for slice-to-slice correspondence. When compared to interpolation methods, our super-resolution reconstruction resulted in the highest PSNR relative to clinical 3D MRI (32.15 dB vs 30.16 dB for BSpline interpolation). Moreover, the registration of 3D volumes reconstructed via super-resolution for both MRI and histopathology images showed the best alignment of cancer regions when compared to (1) the state-of-the-art RAPSODI approach, (2) volumes that were not reconstructed, or (3) volumes that were reconstructed using nearest neighbor, linear, or BSpline interpolations. The improved 3D alignment of histopathology images and MRI facilitates the projection of accurate cancer labels on MRI, allowing for the development of improved MRI interpretation schemes and machine learning models to automatically detect cancer on MRI.

Histological Validation of MRI: A Review of Challenges in Registration of Imaging and Whole-Mount Histopathology

Rigorous validation with ground truth information such as histology is needed to reliably assess the current and potential value of MRI techniques to characterize tissue and identify disease-related tissue alterations. Commonly used methods that aim to directly correlate histology and MRI data generally fall short of this goal due to spatial errors that preclude direct matching. Errors result from tissue deformation, differences in spatial resolution and slice thickness, non-coplanar and/or nonintersecting plane orientations, and different image contrast mechanisms. Some of these problems arise from limitations in standard protocols for clinical tissue processing and histology-based pathology reporting, and to some extent can be addressed by modifications to standard protocols without compromising the clinical process. Typical modifications include ex vivo specimen MRI, block-face photography, addition of fiducial markers, and 3D printed molds to constrain tissue deformation and guide sectioning. This review summarizes the advantages and limitations of MRI validation techniques based on coregistration of MRI with whole-mount histology of tissue specimens. LEVEL OF EVIDENCE: 4 TECHNICAL EFFICACY STAGE: 1.

Motion Correction in Optical Resolution Photoacoustic Microscopy

In this paper, we are proposing a novel motion correction algorithm for high-resolution OR-PAM imaging. Our algorithm combines a modified demons-based tracking approach with a newly developed multi-scale vascular feature matching method to track motion between adjacent B-scan images without needing any reference object. We first applied this algorithm to correct motion artifacts within one three-dimensional (3D) data segment of rat iris obtained with OR-PAM imaging. We then extended the application of this algorithm to correct motions to obtain vasculature imaging in the whole mouse back. In here, we stitched five adjacent 3D data segments (large field-of-view) obtained while changing the focus of OR-PAM differently for each subarea. The results showed that the motion artifacts of both large blood vessels and microvessels could be accurately corrected in both cases. Compared to the manually stitching method and the traditional SIFT algorithm, the algorithm proposed in this paper has better performance in stitching adjacent data segments. The high accuracy of the motion correction algorithm makes it valuable in OR-PAM for high-resolution imaging of large animals and for quantitative functional imaging.

Multiparametric MRI and radiomics in prostate cancer: a review

Multiparametric MRI (mpMRI) is an imaging modality that combines anatomical MR imaging with one or more functional MRI sequences. It has become a versatile tool for detecting and characterising prostate cancer (PCa). The traditional role of mpMRI was confined to PCa staging, but due to the advanced imaging techniques, its role has expanded to various stages in clinical practises including tumour detection, disease monitor during active surveillance and sequential imaging for patient follow-up. Meanwhile, with the growing speed of data generation and the increasing volume of imaging data, it is highly demanded to apply computerised methods to process mpMRI data and extract useful information. Hence quantitative analysis for imaging data using radiomics has become an emerging paradigm. The application of radiomics approaches in prostate cancer has not only enabled automatic localisation of the disease but also provided a non-invasive solution to assess tumour biology (e.g. aggressiveness and the presence of hypoxia). This article reviews mpMRI and its expanding role in PCa detection, staging and patient management. Following that, an overview of prostate radiomics will be provided, with a special focus on its current applications as well as its future directions.

Accurate validation of ultrasound imaging of prostate cancer: a review of challenges in registration of imaging and histopathology

As the development of modalities for prostate cancer (PCa) imaging advances, the challenge of accurate registration between images and histopathologic ground truth becomes more pressing. Localization of PCa, rather than detection, requires a pixel-to-pixel validation of imaging based on histopathology after radical prostatectomy. Such a registration procedure is challenging for ultrasound modalities; not only the deformations of the prostate after resection have to be taken into account, but also the deformation due to the employed transrectal probe and the mismatch in orientation between imaging planes and pathology slices. In this work, we review the latest techniques to facilitate accurate validation of PCa localization in ultrasound imaging studies and extrapolate a general strategy for implementation of a registration procedure.

3D Registration of mpMRI for Assessment of Prostate Cancer Focal Therapy

RATIONALE AND OBJECTIVES

This study aimed to assess a novel method of three-dimensional (3D) co-registration of prostate magnetic resonance imaging (MRI) examinations performed before and after prostate cancer focal therapy.

MATERIALS AND METHODS

We developed a software platform for automatic 3D deformable co-registration of prostate MRI at different time points and applied this method to 10 patients who underwent focal ablative therapy. MRI examinations were performed preoperatively, as well as 1 week and 6 months post treatment. Rigid registration served as reference for assessing co-registration accuracy and precision.

RESULTS

Segmentation of preoperative and postoperative prostate revealed a significant postoperative volume decrease of the gland that averaged 6.49 cc (P = .017). Applying deformable transformation based on mutual information from 120 pairs of MRI slices, we refined by 2.9 mm (max. 6.25 mm) the alignment of the ablation zone, segmented from contrast-enhanced images on the 1-week postoperative examination, to the 6-month postoperative T2-weighted images. This represented a 500% improvement over the rigid approach (P = .001), corrected by volume. The dissimilarity by Dice index of the mapped ablation zone using deformable transformation vs rigid control was significantly (P = .04) higher at the ablation site than in the whole gland.

CONCLUSIONS

Our findings illustrate our method's ability to correct for deformation at the ablation site. The preliminary analysis suggests that deformable transformation computed from mutual information of preoperative and follow-up MRI is accurate in co-registration of MRI examinations performed before and after focal therapy. The ability to localize the previously ablated tissue in 3D space may improve targeting for image-guided follow-up biopsy within focal therapy protocols.

Computer-aided Detection of Prostate Cancer with MRI: Technology and Applications

One in six men will develop prostate cancer in his lifetime. Early detection and accurate diagnosis of the disease can improve cancer survival and reduce treatment costs. Recently, imaging of prostate cancer has greatly advanced since the introduction of multiparametric magnetic resonance imaging (mp-MRI). Mp-MRI consists of T2-weighted sequences combined with functional sequences including dynamic contrast-enhanced MRI, diffusion-weighted MRI, and magnetic resonance spectroscopy imaging. Because of the big data and variations in imaging sequences, detection can be affected by multiple factors such as observer variability and visibility and complexity of the lesions. To improve quantitative assessment of the disease, various computer-aided detection systems have been designed to help radiologists in their clinical practice. This review paper presents an overview of literatures on computer-aided detection of prostate cancer with mp-MRI, which include the technology and its applications. The aim of the survey is threefold: an introduction for those new to the field, an overview for those working in the field, and a reference for those searching for literature on a specific application.

Weighted mutual information analysis substantially improves domain-based functional network models

Motivation: Functional protein–protein interaction (PPI) networks elucidate molecular pathways underlying complex phenotypes, including those of human diseases. Extrapolation of domain–domain interactions (DDIs) from known PPIs is a major domain-based method for inferring functional PPI networks. However, the protein domain is a functional unit of the protein. Therefore, we should be able to effectively infer functional interactions between proteins based on the co-occurrence of domains.

Results: Here, we present a method for inferring accurate functional PPIs based on the similarity of domain composition between proteins by weighted mutual information (MI) that assigned different weights to the domains based on their genome-wide frequencies. Weighted MI outperforms other domain-based network inference methods and is highly predictive for pathways as well as phenotypes. A genome-scale human functional network determined by our method reveals numerous communities that are significantly associated with known pathways and diseases. Domain-based functional networks may, therefore, have potential applications in mapping domain-to-pathway or domain-to-phenotype associations.

Availability and Implementation: Source code for calculating weighted mutual information based on the domain profile matrix is available from www.netbiolab.org/w/WMI.

Contact:Insuklee@yonsei.ac.kr

Supplementary information:Supplementary data are available at Bioinformatics online.

Building of a composite virtual slide from contiguous tissue samples

Background

Currently available microscope slide scanners produce whole slide images at various resolutions from histological sections. Nevertheless, acquisition area and so visualization of large tissue samples are limited by the standardized size of glass slides, used daily in pathology departments. The proposed solution has been developed to build composite virtual slides from images of large tumor fragments.

Materials and methods

Images of HES or immunostained histological sections of carefully labeled fragments from a representative slice of breast carcinoma were acquired with a digital slide scanner at a magnification of 20×. The tiling program involves three steps: the straightening of tissue fragment images using polynomial interpolation method, and the building and assembling of strips of contiguous tissue sample whole slide images in × and y directions. The final image is saved in a pyramidal BigTiff file format. The program has been tested on several tumor slices. A correlation quality control has been done on five images artificially cut.

Results

Sixty tumor slices from twenty surgical specimens, cut into two to twenty six pieces, were reconstructed. A median of 98.71% is obtained by computing the correlation coefficients between native and reconstructed images for quality control.

Conclusions

The proposed method is efficient and able to adapt itself to daily work conditions of classical pathology laboratories.

Registration of in vivo prostate MRI and pseudo-whole mount histology using Local Affine Transformations guided by Internal Structures (LATIS)

PURPOSE

To present a novel registration approach called LATIS (Local Affine Transformation guided by Internal Structures) for coregistering post prostatectomy pseudo-whole mount (PWM) pathological sections with in vivo MRI (magnetic resonance imaging) images.

MATERIALS AND METHODS

Thirty-five patients with biopsy-proven prostate cancer were imaged at 3T with an endorectal coil. Excised prostate specimens underwent quarter mount step-section pathologic processing, digitization, annotation, and assembly into a PWM. Manually annotated macro-structures on both pathology and MRI were used to assist registration using a relaxed local affine transformation approximation. Registration accuracy was assessed by calculation of the Dice similarity coefficient (DSC) between transformed and target capsule masks and least-square distance between transformed and target landmark positions.

RESULTS

LATIS registration resulted in a DSC value of 0.991 ± 0.004 and registration accuracy of 1.54 ± 0.64 mm based on identified landmarks common to both datasets. Image registration performed without the use of internal structures led to an 87% increase in landmark-based registration error. Derived transformation matrices were used to map regions of pathologically defined disease to MRI.

CONCLUSION

LATIS was used to successfully coregister digital pathology with in vivo MRI to facilitate improved correlative studies between pathologically identified features of prostate cancer and multiparametric MRI.

3D prostate histology reconstruction: an evaluation of image-based and fiducial-based algorithms

PURPOSE

Evaluation of in vivo prostate imaging modalities for determining the spatial distribution and aggressiveness of prostate cancer ideally requires accurate registration of images to an accepted reference standard, such as histopathological examination of radical prostatectomy specimens. Three-dimensional (3D) reconstruction of prostate histology facilitates these registration-based evaluations by reintroducing 3D spatial information lost during histology processing. Because the reconstruction accuracy may constrain the clinical questions that can be answered with these data, it is important to assess the tradeoffs between minimally disruptive methods based on intrinsic image information and potentially more robust methods based on extrinsic fiducial markers.

METHODS

Ex vivo magnetic resonance (MR) images and digitized whole-mount histology images from 12 radical prostatectomy specimens were used to evaluate four 3D histology reconstruction algorithms. 3D reconstructions were computed by registering each histology image to the corresponding ex vivo MR image using one of two similarity metrics (mutual information or fiducial registration error) and one of two search domains (affine transformations or a constrained subset thereof). The algorithms were evaluated for accuracy using the mean target registration error (TRE) computed from homologous intrinsic point landmarks (3-16 per histology section; 232 total) identified on histology and MR images, and for the sensitivity of TRE to rotational, translational, and scaling initialization errors.

RESULTS

The algorithms using fiducial registration error and mutual information had mean ± standard deviation TREs of 0.7 ± 0.4 and 1.2 ± 0.7 mm, respectively, and one algorithm using fiducial registration error and affine transforms had negligible sensitivities to initialization errors. The postoptimization values of the mutual information-based metric showed evidence of errors due to both the optimizer and the similarity metric, and variation of parameters of the mutual information-based metric did not improve its performance.

CONCLUSIONS

The extrinsic fiducial-based algorithm had lower mean TRE and lower sensitivity to initialization than the intrinsic intensity-based algorithm using mutual information. A model relating statistical power to registration error for certain imaging validation study designs estimated that a reconstruction algorithm with a mean TRE of 0.7 mm would require 27% fewer subjects than the method used to initialize the algorithms (mean TRE 1.3 ± 0.7 mm), suggesting the choice of reconstruction technique can have a substantial impact on the design of imaging validation studies, and on their overall cost.

Semiautomatic registration of digital histopathology images to in vivo MR images in molded and unmolded prostates

PURPOSE

To evaluate a semiautomatic software-based method of registering in vivo prostate MR images to digital histopathology images using two approaches: (i) in which the prostates were molded to simulate distortion due to the endorectal imaging coil before fixation, and (ii) in which the prostates were not molded.

MATERIALS AND METHODS

T2-weighted MR images and digitized whole-mount histopathology images were acquired for 26 patients with biopsy-confirmed prostate cancer who underwent radical prostatectomy. Ten excised prostates were molded before fixation. A semiautomatic method was used to align MR images to histopathology. Percent overlap between MR and histopathology images, as well as distances between corresponding anatomical landmarks were calculated and used to evaluate the registration technique for molded and unmolded cases.

RESULTS

The software successfully morphed histology-based prostate images into corresponding MR images. Percent overlap improved from 80.4 ± 5.8% before morphing to 99.7 ± 0.62% post morphing. Molded prostates had a smaller distance between landmarks (1.91 ± 0.75 mm) versus unmolded (2.34 ± 0.68 mm), P < 0.08.

CONCLUSION

Molding a prostate before fixation provided a better alignment of internal structures within the prostate, but this did not reach statistical significance. Software-based morphing allowed for nearly complete overlap between the pathology slides and the MR images.

Bayesian technique for image classifying registration

In this paper, we address a complex image registration issue arising while the dependencies between intensities of images to be registered are not spatially homogeneous. Such a situation is frequently encountered in medical imaging when a pathology present in one of the images modifies locally intensity dependencies observed on normal tissues. Usual image registration models, which are based on a single global intensity similarity criterion, fail to register such images, as they are blind to local deviations of intensity dependencies. Such a limitation is also encountered in contrast-enhanced images where there exist multiple pixel classes having different properties of contrast agent absorption. In this paper, we propose a new model in which the similarity criterion is adapted locally to images by classification of image intensity dependencies. Defined in a Bayesian framework, the similarity criterion is a mixture of probability distributions describing dependencies on two classes. The model also includes a class map which locates pixels of the two classes and weighs the two mixture components. The registration problem is formulated both as an energy minimization problem and as a maximum a posteriori estimation problem. It is solved using a gradient descent algorithm. In the problem formulation and resolution, the image deformation and the class map are estimated simultaneously, leading to an original combination of registration and classification that we call image classifying registration. Whenever sufficient information about class location is available in applications, the registration can also be performed on its own by fixing a given class map. Finally, we illustrate the interest of our model on two real applications from medical imaging: template-based segmentation of contrast-enhanced images and lesion detection in mammograms. We also conduct an evaluation of our model on simulated medical data and show its ability to take into account spatial variations of intensity dependencies while keeping a good registration accuracy.