Registration of PET and CT images based on multiresolution gradient of mutual information demons algorithm for positioning esophageal cancer patients

Multi-Modal Ensemble Deep Learning in Head and Neck Cancer HPV Sub-Typing

Oropharyngeal Squamous Cell Carcinoma (OPSCC) is one of the common forms of heterogeneity in head and neck cancer. Infection with human papillomavirus (HPV) has been identified as a major risk factor for OPSCC. Therefore, differentiating the HPV-positive and negative cases in OPSCC patients is an essential diagnostic factor influencing future treatment decisions. In this study, we investigated the accuracy of a deep learning-based method for image interpretation and automatically detected the HPV status of OPSCC in routinely acquired Computed Tomography (CT) and Positron Emission Tomography (PET) images. We introduce a 3D CNN-based multi-modal feature fusion architecture for HPV status prediction in primary tumor lesions. The architecture is composed of an ensemble of CNN networks and merges image features in a softmax classification layer. The pipeline separately learns the intensity, contrast variation, shape, texture heterogeneity, and metabolic assessment from CT and PET tumor volume regions and fuses those multi-modal features for final HPV status classification. The precision, recall, and AUC scores of the proposed method are computed, and the results are compared with other existing models. The experimental results demonstrate that the multi-modal ensemble model with soft voting outperformed single-modality PET/CT, with an AUC of 0.76 and F1 score of 0.746 on publicly available TCGA and MAASTRO datasets. In the MAASTRO dataset, our model achieved an AUC score of 0.74 over primary tumor volumes of interest (VOIs). In the future, more extensive cohort validation may suffice for better diagnostic accuracy and provide preliminary assessment before the biopsy.

General simultaneous motion estimation and image reconstruction (G-SMEIR)

To achieve better performance for 4D multi-frame reconstruction with the parametric motion model (MF-PMM), a general simultaneous motion estimation and image reconstruction (G-SMEIR) method is proposed. In G-SMEIR, projection domain motion estimation and image domain motion estimation are performed alternatively to achieve better 4D reconstruction. This method can mitigate the local optimum trapping problem in either domain. To improve computational efficiency, the image domain motion estimation is accelerated by adapting fast convergent algorithms and graphics processing unit (GPU) computing. The proposed G-SMEIR method is tested using a cone-beam computed tomography (CBCT) simulation study of 4D XCAT phantom at different dose levels and compared with 3D total variation-based reconstruction (3D TV), 4D reconstruction with image domain motion estimation (IM4D), and SMEIR. G-SMEIR shows strong denoising capability and achieves similar performance at regular dose and half dose. The root mean squared error (RMSE) of G-SMEIR is the best among the four methods and improved about 12% over SMEIR for all respiratory phase images at full dose. G-SMEIR also achieved the best structural similarity index (SSIM) values among all methods. More importantly, G-SMEIR leads to more than 40% improvement of the mean deviation from the phantom tumor motion over SMEIR. A preliminary patient CBCT image reconstruction also shows better image quality of G-SMEIR than that of the frame-by-frame reconstruction (3D TV) and MF-PMM either using image domain motion estimation (IM4D) or using projection domain motion estimation (SMEIR) alone. G-SMEIR with a flexible combination of image domain and projection domain motion estimation provides an effective tool for 4D tomographic reconstruction.

miRID: Multi-Modal Image Registration Using Modality-Independent and Rotation-Invariant Descriptor

Axiomatically, symmetry is a fundamental property of mathematical functions defining similarity measures, where similarity measures are important tools in many areas of computer science, including machine learning and image processing. In this paper, we investigate a new technique to measure the similarity between two images, a fixed image and a moving image, in multi-modal image registration (MIR). MIR in medical image processing is essential and useful in diagnosis and therapy guidance, but still a very challenging task due to the lack of robustness against the rotational variance in the image transformation process. Our investigation leads to a novel, local self-similarity descriptor, called the modality-independent and rotation-invariant descriptor (miRID). By relying on the mean of the intensity values, an miRID is simply computable and can effectively handle the complicated intensity relationship between multi-modal images. Moreover, it can also overcome the problem of rotational variance by sorting the numerical values, each of which is the absolute difference between each pixel’s intensity and the mean of all pixel intensities within a patch of the image. The experimental result shows that our method outperforms others in both multi-modal rigid and non-rigid image registrations.

Evaluation of Medical Image Registration Techniques Based on Nature and Domain of the Transformation

A lot of research has been done during the past 20 years in the area of medical image registration for obtaining detailed, important, and complementary information from two or more images and aligning them into a single, more informative image. Nature of the transformation and domain of the transformation are two important medical image registration techniques that deal with characters of objects (motions) in images. This article presents a detailed survey of the registration techniques that belong to both categories with detailed elaboration on their features, issues, and challenges. An investigation estimating similarity and dissimilarity measures and performance evaluation is the main objective of this work. This article also provides reference knowledge in a compact form for researchers and clinicians looking for the proper registration technique for a particular application.

Evaluation of GMI and PMI diffeomorphic-based demons algorithms for aligning PET and CT Images

Fusion of anatomic information in computed tomography (CT) and functional information in F18‐FDG positron emission tomography (PET) is crucial for accurate differentiation of tumor from benign masses, designing radiotherapy treatment plan and staging of cancer. Although current PET and CT images can be acquired from combined F18‐FDG PET/CT scanner, the two acquisitions are scanned separately and take a long time, which may induce potential positional errors in global and local caused by respiratory motion or organ peristalsis. So registration (alignment) of whole‐body PET and CT images is a prerequisite for their meaningful fusion. The purpose of this study was to assess the performance of two multimodal registration algorithms for aligning PET and CT images. The proposed gradient of mutual information (GMI)‐based demons algorithm, which incorporated the GMI between two images as an external force to facilitate the alignment, was compared with the point‐wise mutual information (PMI) diffeomorphic‐based demons algorithm whose external force was modified by replacing the image intensity difference in diffeomorphic demons algorithm with the PMI to make it appropriate for multimodal image registration. Eight patients with esophageal cancer(s) were enrolled in this IRB‐approved study. Whole‐body PET and CT images were acquired from a combined F18‐FDG PET/CT scanner for each patient. The modified Hausdorff distance (dMH) was used to evaluate the registration accuracy of the two algorithms. Of all patients, the mean values and standard deviations (SDs) of dMH were 6.65 (± 1.90) voxels and 6.01 (± 1.90) after the GMI‐based demons and the PMI diffeomorphic‐based demons registration algorithms respectively. Preliminary results on oncological patients showed that the respiratory motion and organ peristalsis in PET/CT esophageal images could not be neglected, although a combined F18‐FDG PET/CT scanner was used for image acquisition. The PMI diffeomorphic‐based demons algorithm was more accurate than the GMI‐based demons algorithm in registering PET/CT esophageal images.

PACS numbers: 87.57.nj, 87.57. Q‐, 87.57.uk