Deep Learning-Based Concurrent Brain Registration and Tumor Segmentation

Empowering brain cancer diagnosis: harnessing artificial intelligence for advanced imaging insights

Artificial intelligence (AI) is increasingly being used in the medical field, specifically for brain cancer imaging. In this review, we explore how AI-powered medical imaging can impact the diagnosis, prognosis, and treatment of brain cancer. We discuss various AI techniques, including deep learning and causality learning, and their relevance. Additionally, we examine current applications that provide practical solutions for detecting, classifying, segmenting, and registering brain tumors. Although challenges such as data quality, availability, interpretability, transparency, and ethics persist, we emphasise the enormous potential of intelligent applications in standardising procedures and enhancing personalised treatment, leading to improved patient outcomes. Innovative AI solutions have the power to revolutionise neuro-oncology by enhancing the quality of routine clinical practice.

SMILE: Siamese Multi-scale Interactive-representation LEarning for Hierarchical Diffeomorphic Deformable image registration

Deformable medical image registration plays an important role in many clinical applications. It aims to find a dense deformation field to establish point-wise correspondences between a pair of fixed and moving images. Recently, unsupervised deep learning-based registration methods have drawn more and more attention because of fast inference at testing stage. Despite remarkable progress, existing deep learning-based methods suffer from several limitations including: (a) they often overlook the explicit modeling of feature correspondences due to limited receptive fields; (b) the performance on image pairs with large spatial displacements is still limited since the dense deformation field is regressed from features learned by local convolutions; and (c) desirable properties, including topology-preservation and the invertibility of transformation, are often ignored. To address above limitations, we propose a novel Convolutional Neural Network (CNN) consisting of a Siamese Multi-scale Interactive-representation LEarning (SMILE) encoder and a Hierarchical Diffeomorphic Deformation (HDD) decoder. Specifically, the SMILE encoder aims for effective feature representation learning and spatial correspondence establishing while the HDD decoder seeks to regress the dense deformation field in a coarse-to-fine manner. We additionally propose a novel Local Invertible Loss (LIL) to encourage topology-preservation and local invertibility of the regressed transformation while keeping high registration accuracy. Extensive experiments conducted on two publicly available brain image datasets demonstrate the superiority of our method over the state-of-the-art (SOTA) approaches. Specifically, on the Neurite-OASIS dataset, our method achieved an average DSC of 0.815 and an average ASSD of 0.633 mm.

Unsupervised deep learning registration model for multimodal brain images

Multimodal image registration is a key for many clinical image-guided interventions. However, it is a challenging task because of complicated and unknown relationships between different modalities. Currently, deep supervised learning is the state-of-theart method at which the registration is conducted in end-to-end manner and one-shot. Therefore, a huge ground-truth data is required to improve the results of deep neural networks for registration. Moreover, supervised methods may yield models that bias towards annotated structures. Here, to deal with above challenges, an alternative approach is using unsupervised learning models. In this study, we have designed a novel deep unsupervised Convolutional Neural Network (CNN)-based model based on computer tomography/magnetic resonance (CT/MR) co-registration of brain images in an affine manner. For this purpose, we created a dataset consisting of 1100 pairs of CT/MR slices from the brain of 110 neuropsychic patients with/without tumor. At the next step, 12 landmarks were selected by a well-experienced radiologist and annotated on each slice resulting in the computation of series of metrics evaluation, target registration error (TRE), Dice similarity, Hausdorff, and Jaccard coefficients. The proposed method could register the multimodal images with TRE 9.89, Dice similarity 0.79, Hausdorff 7.15, and Jaccard 0.75 that are appreciable for clinical applications. Moreover, the approach registered the images in an acceptable time 203 ms and can be appreciable for clinical usage due to the short registration time and high accuracy. Here, the results illustrated that our proposed method achieved competitive performance against other related approaches from both reasonable computation time and the metrics evaluation.

Investigation and benchmarking of U-Nets on prostate segmentation tasks

In healthcare, a growing number of physicians and support staff are striving to facilitate personalized radiotherapy regimens for patients with prostate cancer. This is because individual patient biology is unique, and employing a single approach for all is inefficient. A crucial step for customizing radiotherapy planning and gaining fundamental information about the disease, is the identification and delineation of targeted structures. However, accurate biomedical image segmentation is time-consuming, requires considerable experience and is prone to observer variability. In the past decade, the use of deep learning models has significantly increased in the field of medical image segmentation. At present, a vast number of anatomical structures can be demarcated on a clinician's level with deep learning models. These models would not only unload work, but they can offer unbiased characterization of the disease. The main architectures used in segmentation are the U-Net and its variants, that exhibit outstanding performances. However, reproducing results or directly comparing methods is often limited by closed source of data and the large heterogeneity among medical images. With this in mind, our intention is to provide a reliable source for assessing deep learning models. As an example, we chose the challenging task of delineating the prostate gland in multi-modal images. First, this paper provides a comprehensive review of current state-of-the-art convolutional neural networks for 3D prostate segmentation. Second, utilizing public and in-house CT and MR datasets of varying properties, we created a framework for an objective comparison of automatic prostate segmentation algorithms. The framework was used for rigorous evaluations of the models, highlighting their strengths and weaknesses.

Learn2Reg: Comprehensive Multi-Task Medical Image Registration Challenge, Dataset and Evaluation in the Era of Deep Learning

Image registration is a fundamental medical image analysis task, and a wide variety of approaches have been proposed. However, only a few studies have comprehensively compared medical image registration approaches on a wide range of clinically relevant tasks. This limits the development of registration methods, the adoption of research advances into practice, and a fair benchmark across competing approaches. The Learn2Reg challenge addresses these limitations by providing a multi-task medical image registration data set for comprehensive characterisation of deformable registration algorithms. A continuous evaluation will be possible at https://learn2reg.grand-challenge.org. Learn2Reg covers a wide range of anatomies (brain, abdomen, and thorax), modalities (ultrasound, CT, MR), availability of annotations, as well as intra- and inter-patient registration evaluation. We established an easily accessible framework for training and validation of 3D registration methods, which enabled the compilation of results of over 65 individual method submissions from more than 20 unique teams. We used a complementary set of metrics, including robustness, accuracy, plausibility, and runtime, enabling unique insight into the current state-of-the-art of medical image registration. This paper describes datasets, tasks, evaluation methods and results of the challenge, as well as results of further analysis of transferability to new datasets, the importance of label supervision, and resulting bias. While no single approach worked best across all tasks, many methodological aspects could be identified that push the performance of medical image registration to new state-of-the-art performance. Furthermore, we demystified the common belief that conventional registration methods have to be much slower than deep-learning-based methods.

A segmentation-informed deep learning framework to register dynamic two-dimensional magnetic resonance images of the vocal tract during speech

Objective

Dynamic magnetic resonance (MR) imaging enables visualisation of articulators during speech. There is growing interest in quantifying articulator motion in two-dimensional MR images of the vocal tract, to better understand speech production and potentially inform patient management decisions. Image registration is an established way to achieve this quantification. Recently, segmentation-informed deformable registration frameworks have been developed and have achieved state-of-the-art accuracy. This work aims to adapt such a framework and optimise it for estimating displacement fields between dynamic two-dimensional MR images of the vocal tract during speech.

Methods

A deep-learning-based registration framework was developed and compared with current state-of-the-art registration methods and frameworks (two traditional methods and three deep-learning-based frameworks, two of which are segmentation informed). The accuracy of the methods and frameworks was evaluated using the Dice coefficient (DSC), average surface distance (ASD) and a metric based on velopharyngeal closure. The metric evaluated if the fields captured a clinically relevant and quantifiable aspect of articulator motion.

Results

The segmentation-informed frameworks achieved higher DSCs and lower ASDs and captured more velopharyngeal closures than the traditional methods and the framework that was not segmentation informed. All segmentation-informed frameworks achieved similar DSCs and ASDs. However, the proposed framework captured the most velopharyngeal closures.

Conclusions

A framework was successfully developed and found to more accurately estimate articulator motion than five current state-of-the-art methods and frameworks.

Significance

The first deep-learning-based framework specifically for registering dynamic two-dimensional MR images of the vocal tract during speech has been developed and evaluated.

SegPC-2021: A challenge & dataset on segmentation of Multiple Myeloma plasma cells from microscopic images

Multiple Myeloma (MM) is an emerging ailment of global concern. Its diagnosis at the early stages is critical for recovery. Therefore, efforts are underway to produce digital pathology tools with human-level intelligence that are efficient, scalable, accessible, and cost-effective. Following the trend, a medical imaging challenge on "Segmentation of Multiple Myeloma Plasma Cells in Microscopic Images (SegPC-2021)" was organized at the IEEE International Symposium on Biomedical Imaging (ISBI), 2021, France. The challenge addressed the problem of cell segmentation in microscopic images captured from the slides prepared from the bone marrow aspirate of patients diagnosed with Multiple Myeloma. The challenge released a total of 775 images with 690 and 85 images of sizes 2040×1536 and 1920×2560 pixels, respectively, captured from two different (microscope and camera) setups. The participants had to segment the plasma cells with a separate label on each cell's nucleus and cytoplasm. This problem comprises many challenges, including a reduced color contrast between the cytoplasm and the background, and the clustering of cells with a feeble boundary separation of individual cells. To our knowledge, the SegPC-2021 challenge dataset is the largest publicly available annotated data on plasma cell segmentation in MM so far. The challenge targets a semi-automated tool to ensure the supervision of medical experts. It was conducted for a span of five months, from November 2020 to April 2021. Initially, the data was shared with 696 people from 52 teams, of which 41 teams submitted the results of their models on the evaluation portal in the validation phase. Similarly, 20 teams qualified for the last round, of which 16 teams submitted the results in the final test phase. All the top-5 teams employed DL-based approaches, and the best mIoU obtained on the final test set of 277 microscopic images was 0.9389. All these five models have been analyzed and discussed in detail. This challenge task is a step towards the target of creating an automated MM diagnostic tool.

Quantitative evaluation of the influence of multiple MRI sequences and of pathological tissues on the registration of longitudinal data acquired during brain tumor treatment

Registration methods facilitate the comparison of multiparametric magnetic resonance images acquired at different stages of brain tumor treatments. Image-based registration solutions are influenced by the sequences chosen to compute the distance measure, and the lack of image correspondences due to the resection cavities and pathological tissues. Nonetheless, an evaluation of the impact of these input parameters on the registration of longitudinal data is still missing. This work evaluates the influence of multiple sequences, namely T1-weighted (T1), T2-weighted (T2), contrast enhanced T1-weighted (T1-CE), and T2 Fluid Attenuated Inversion Recovery (FLAIR), and the exclusion of the pathological tissues on the non-rigid registration of pre- and post-operative images. We here investigate two types of registration methods, an iterative approach and a convolutional neural network solution based on a 3D U-Net. We employ two test sets to compute the mean target registration error (mTRE) based on corresponding landmarks. In the first set, markers are positioned exclusively in the surroundings of the pathology. The methods employing T1-CE achieves the lowest mTREs, with a improvement up to 0.8 mm for the iterative solution. The results are higher than the baseline when using the FLAIR sequence. When excluding the pathology, lower mTREs are observable for most of the methods. In the second test set, corresponding landmarks are located in the entire brain volumes. Both solutions employing T1-CE obtain the lowest mTREs, with a decrease up to 1.16 mm for the iterative method, whereas the results worsen using the FLAIR. When excluding the pathology, an improvement is observable for the CNN method using T1-CE. Both approaches utilizing the T1-CE sequence obtain the best mTREs, whereas the FLAIR is the least informative to guide the registration process. Besides, the exclusion of pathology from the distance measure computation improves the registration of the brain tissues surrounding the tumor. Thus, this work provides the first numerical evaluation of the influence of these parameters on the registration of longitudinal magnetic resonance images, and it can be helpful for developing future algorithms.

3D Lightweight Network for Simultaneous Registration and Segmentation of Organs-at-Risk in CT Images of Head and Neck Cancer

Image-guided radiation therapy (IGRT) is the most effective treatment for head and neck cancer. The successful implementation of IGRT requires accurate delineation of organ-at-risk (OAR) in the computed tomography (CT) images. In routine clinical practice, OARs are manually segmented by oncologists, which is time-consuming, laborious, and subjective. To assist oncologists in OAR contouring, we proposed a three-dimensional (3D) lightweight framework for simultaneous OAR registration and segmentation. The registration network was designed to align a selected OAR template to a new image volume for OAR localization. A region of interest (ROI) selection layer then generated ROIs of OARs from the registration results, which were fed into a multiview segmentation network for accurate OAR segmentation. To improve the performance of registration and segmentation networks, a centre distance loss was designed for the registration network, an ROI classification branch was employed for the segmentation network, and further, context information was incorporated to iteratively promote both networks' performance. The segmentation results were further refined with shape information for final delineation. We evaluated registration and segmentation performances of the proposed framework using three datasets. On the internal dataset, the Dice similarity coefficient (DSC) of registration and segmentation was 69.7% and 79.6%, respectively. In addition, our framework was evaluated on two external datasets and gained satisfactory performance. These results showed that the 3D lightweight framework achieved fast, accurate and robust registration and segmentation of OARs in head and neck cancer. The proposed framework has the potential of assisting oncologists in OAR delineation.

Machine Learning Algorithms in Neuroimaging: An Overview

Machine learning (ML) and artificial intelligence (AI) applications in the field of neuroimaging have been on the rise in recent years, and their clinical adoption is increasing worldwide. Deep learning (DL) is a field of ML that can be defined as a set of algorithms enabling a computer to be fed with raw data and progressively discover-through multiple layers of representation-more complex and abstract patterns in large data sets. The combination of ML and radiomics, namely the extraction of features from medical images, has proven valuable, too: Radiomic information can be used for enhanced image characterization and prognosis or outcome prediction. This chapter summarizes the basic concepts underlying ML application for neuroimaging and discusses technical aspects of the most promising algorithms, with a specific focus on Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), in order to provide the readership with the fundamental theoretical tools to better understand ML in neuroimaging. Applications are highlighted from a practical standpoint in the last section of the chapter, including: image reconstruction and restoration, image synthesis and super-resolution, registration, segmentation, classification, and outcome prediction.

Advances in micro-CT imaging of small animals

PURPOSE

Micron-scale computed tomography (micro-CT) imaging is a ubiquitous, cost-effective, and non-invasive three-dimensional imaging modality. We review recent developments and applications of micro-CT for preclinical research.

METHODS

Based on a comprehensive review of recent micro-CT literature, we summarize features of state-of-the-art hardware and ongoing challenges and promising research directions in the field.

RESULTS

Representative features of commercially available micro-CT scanners and some new applications for both in vivo and ex vivo imaging are described. New advancements include spectral scanning using dual-energy micro-CT based on energy-integrating detectors or a new generation of photon-counting x-ray detectors (PCDs). Beyond two-material discrimination, PCDs enable quantitative differentiation of intrinsic tissues from one or more extrinsic contrast agents. When these extrinsic contrast agents are incorporated into a nanoparticle platform (e.g. liposomes), novel micro-CT imaging applications are possible such as combined therapy and diagnostic imaging in the field of cancer theranostics. Another major area of research in micro-CT is in x-ray phase contrast (XPC) imaging. XPC imaging opens CT to many new imaging applications because phase changes are more sensitive to density variations in soft tissues than standard absorption imaging. We further review the impact of deep learning on micro-CT. We feature several recent works which have successfully applied deep learning to micro-CT data, and we outline several challenges specific to micro-CT.

CONCLUSIONS

All of these advancements establish micro-CT imaging at the forefront of preclinical research, able to provide anatomical, functional, and even molecular information while serving as a testbench for translational research.

Biomarker Technologies to Support Early Clinical Immuno-oncology Development: Advances and Interpretation

Today, there is a huge effort to develop cancer immunotherapeutics capable of combating cancer cells as well as the biological environment in which they can grow, adapt, and survive. For such treatments to benefit more patients, there is a great need to dissect the complex interplays between tumor cells and the host's immune system. Monitoring mechanisms of resistance to immunotherapeutics can delineate the evolution of key players capable of driving an efficacious antitumor immune response. In doing so, simultaneous and systematic interrogation of multiple biomarkers beyond single biomarker approaches needs to be undertaken. Zooming into cell-to-cell interactions using technological advancements with unprecedented cellular resolution such as single-cell spatial transcriptomics, advanced tissue histology approaches, and new molecular immune profiling tools promises to provide a unique level of molecular granularity of the tumor environment and may support better decision-making during drug development. This review will focus on how such technological tools are applied in clinical settings, to inform the underlying tumor-immune biology of patients and offer a deeper understanding of cancer immune responsiveness to immuno-oncology treatments.

Artificial intelligence in radiation oncology

Artificial intelligence (AI) is a computer science that tries to mimic human-like intelligence in machines that use computer software and algorithms to perform specific tasks without direct human input. Machine learning (ML) is a subunit of AI that uses data-driven algorithms that learn to imitate human behavior based on a previous example or experience. Deep learning is an ML technique that uses deep neural networks to create a model. The growth and sharing of data, increasing computing power, and developments in AI have initiated a transformation in healthcare. Advances in radiation oncology have produced a significant amount of data that must be integrated with computed tomography imaging, dosimetry, and imaging performed before each fraction. Of the many algorithms used in radiation oncology, has advantages and limitations with different computational power requirements. The aim of this review is to summarize the radiotherapy (RT) process in workflow order by identifying specific areas in which quality and efficiency can be improved by ML. The RT stage is divided into seven stages: patient evaluation, simulation, contouring, planning, quality control, treatment application, and patient follow-up. A systematic evaluation of the applicability, limitations, and advantages of AI algorithms has been done for each stage.

Longitudinal diffusion MRI analysis using Segis-Net: A single-step deep-learning framework for simultaneous segmentation and registration

This work presents a single-step deep-learning framework for longitudinal image analysis, coined Segis-Net. To optimally exploit information available in longitudinal data, this method concurrently learns a multi-class segmentation and nonlinear registration. Segmentation and registration are modeled using a convolutional neural network and optimized simultaneously for their mutual benefit. An objective function that optimizes spatial correspondence for the segmented structures across time-points is proposed. We applied Segis-Net to the analysis of white matter tracts from N=8045 longitudinal brain MRI datasets of 3249 elderly individuals. Segis-Net approach showed a significant increase in registration accuracy, spatio-temporal segmentation consistency, and reproducibility compared with two multistage pipelines. This also led to a significant reduction in the sample-size that would be required to achieve the same statistical power in analyzing tract-specific measures. Thus, we expect that Segis-Net can serve as a new reliable tool to support longitudinal imaging studies to investigate macro- and microstructural brain changes over time.

Diffusion and perfusion MRI radiomics obtained from deep learning segmentation provides reproducible and comparable diagnostic model to human in post-treatment glioblastoma

OBJECTIVES

Deep learning-based automatic segmentation (DLAS) helps the reproducibility of radiomics features, but its effect on radiomics modeling is unknown. We therefore evaluated whether DLAS can robustly extract anatomical and physiological MRI features, thereby assisting in the accurate assessment of treatment response in glioblastoma patients.

METHODS

A DLAS model was trained on 238 glioblastomas and validated on an independent set of 98 pre- and 86 post-treatment glioblastomas from two tertiary hospitals. A total of 1618 radiomics features from contrast-enhanced T1-weighted images (CE-T1w) and histogram features from apparent diffusion coefficient (ADC) and cerebral blood volume (CBV) mapping were extracted. The diagnostic performance of radiomics features and ADC and CBV parameters for identifying treatment response was tested using area under the curve (AUC) from receiver operating characteristics analysis. Feature reproducibility was tested using a 0.80 cutoff for concordance correlation coefficients.

RESULTS

Reproducibility was excellent for ADC and CBV features (ICC, 0.82-0.99) and first-order features (pre- and post-treatment, 100% and 94.1% remained), but lower for texture (79.0% and 69.1% remained) and wavelet-transformed (81.8% and 74.9% remained) features of CE-T1w. DLAS-based radiomics showed similar performance to human-performed segmentations in internal validation (AUC, 0.81 [95% CI, 0.64-0.99] vs. AUC, 0.81 [0.60-1.00], p = 0.80), but slightly lower performance in external validation (AUC, 0.78 [0.61-0.95] vs. AUC, 0.65 [0.46-0.84], p = 0.23).

CONCLUSION

DLAS-based feature extraction showed high reproducibility for first-order features from anatomical and physiological MRI, and comparable diagnostic performance to human manual segmentations in the identification of pseudoprogression, supporting the utility of DLAS in quantitative MRI analysis.

KEY POINTS

• Deep learning-based automatic segmentation (DLAS) enables fast and robust feature extraction from diffusion- and perfusion-weighted MRI. • DLAS showed high reproducibility in first-order feature extraction from anatomical, diffusion, and perfusion MRI across two centers. • DLAS-based radiomics features showed comparable diagnostic accuracy to manual segmentations in post-treatment glioblastoma.

Molecular subgrouping of medulloblastoma based on few-shot learning of multitasking using conventional MR images: a retrospective multicenter study

Background

The determination of molecular subgroups—wingless (WNT), sonic hedgehog (SHH), Group 3, and Group 4—of medulloblastomas is very important for prognostication and risk-adaptive treatment strategies. Due to the rare disease characteristics of medulloblastoma, we designed a unique multitask framework for the few-shot scenario to achieve noninvasive molecular subgrouping with high accuracy.

Methods

We introduced a multitask technique based on mask regional convolutional neural network (Mask-RCNN). By effectively utilizing the comprehensive information including genotyping, tumor mask, and prognosis, multitask technique, on the one hand, realized multi-purpose modeling and simultaneously, on the other hand, promoted the accuracy of the molecular subgrouping. One hundred and thirteen medulloblastoma cases were collected from 4 hospitals during the 8-year period in the retrospective study, which were divided into 3-fold cross-validation cohorts (N = 74) from 2 hospitals and independent testing cohort (N = 39) from the other 2 hospitals. Comparative experiments of different auxiliary tasks were designed to illustrate the effect of multitasking in molecular subgrouping.

Results

Compared to the single-task framework, the multitask framework that combined 3 tasks increased the average accuracy of molecular subgrouping from 0.84 to 0.93 in cross-validation and from 0.79 to 0.85 in independent testing. The average area under the receiver operating characteristic curves (AUCs) of molecular subgrouping were 0.97 in cross-validation and 0.92 in independent testing. The average AUCs of prognostication also reached to 0.88 in cross-validation and 0.79 in independent testing. The tumor segmentation results achieved the Dice coefficient of 0.90 in both cohorts.

Conclusions

The multitask Mask-RCNN is an effective method for the molecular subgrouping and prognostication of medulloblastomas with high accuracy in few-shot learning.