Multi-atlas segmentation of biomedical images: A survey

MR-CT multi-atlas registration guided by fully automated brain structure segmentation with CNNs

PURPOSE

Computed tomography (CT) is widely used to identify anomalies in brain tissues because their localization is important for diagnosis and therapy planning. Due to the insufficient soft tissue contrast of CT, the division of the brain into anatomical meaningful regions is challenging and is commonly done with magnetic resonance imaging (MRI).

METHODS

We propose a multi-atlas registration approach to propagate anatomical information from a standard MRI brain atlas to CT scans. This translation will enable a detailed automated reporting of brain CT exams. We utilize masks of the lateral ventricles and the brain volume of CT images as adjuvant input to guide the registration process. Besides using manual annotations to test the registration in a first step, we then verify that convolutional neural networks (CNNs) are a reliable solution for automatically segmenting structures to enhance the registration process.

RESULTS

The registration method obtains mean Dice values of 0.92 and 0.99 in brain ventricles and parenchyma on 22 healthy test cases when using manually segmented structures as guidance. When guiding with automatically segmented structures, the mean Dice values are 0.87 and 0.98, respectively.

CONCLUSION

Our registration approach is a fully automated solution to register MRI atlas images to CT scans and thus obtain detailed anatomical information. The proposed CNN segmentation method can be used to obtain masks of ventricles and brain volume which guide the registration.

Automatic Time-Resolved Cardiovascular Segmentation of 4D Flow MRI Using Deep Learning

BACKGROUND

Segmenting the whole heart over the cardiac cycle in 4D flow MRI is a challenging and time-consuming process, as there is considerable motion and limited contrast between blood and tissue.

PURPOSE

To develop and evaluate a deep learning-based segmentation method to automatically segment the cardiac chambers and great thoracic vessels from 4D flow MRI.

STUDY TYPE

Retrospective.

SUBJECTS

A total of 205 subjects, including 40 healthy volunteers and 165 patients with a variety of cardiac disorders were included. Data were randomly divided into training (n = 144), validation (n = 20), and testing (n = 41) sets.

FIELD STRENGTH/SEQUENCE

A 3 T/time-resolved velocity encoded 3D gradient echo sequence (4D flow MRI).

ASSESSMENT

A 3D neural network based on the U-net architecture was trained to segment the four cardiac chambers, aorta, and pulmonary artery. The segmentations generated were compared to manually corrected atlas-based segmentations. End-diastolic (ED) and end-systolic (ES) volumes of the four cardiac chambers were calculated for both segmentations.

STATISTICAL TESTS

Dice score, Hausdorff distance, average surface distance, sensitivity, precision, and miss rate were used to measure segmentation accuracy. Bland-Altman analysis was used to evaluate agreement between volumetric parameters.

RESULTS

The following evaluation metrics were computed: mean Dice score (0.908 ± 0.023) (mean ± SD), Hausdorff distance (1.253 ± 0.293 mm), average surface distance (0.466 ± 0.136 mm), sensitivity (0.907 ± 0.032), precision (0.913 ± 0.028), and miss rate (0.093 ± 0.032). Bland-Altman analyses showed good agreement between volumetric parameters for all chambers. Limits of agreement as percentage of mean chamber volume (LoA%), left ventricular: 9.3%, 13.5%, left atrial: 12.4%, 16.9%, right ventricular: 9.9%, 15.6%, and right atrial: 18.7%, 14.4%; for ED and ES, respectively.

DATA CONCLUSION

The addition of this technique to the 4D flow MRI assessment pipeline could expedite and improve the utility of this type of acquisition in the clinical setting.

EVIDENCE LEVEL

4 TECHNICAL EFFICACY: Stage 1.

Using MRI to measure position and anatomy changes and assess their impact on the accuracy of hyperthermia treatment planning for cervical cancer

PURPOSE

We studied the differences between planning and treatment position, their impact on the accuracy of hyperthermia treatment planning (HTP) predictions, and the relevance of including true treatment anatomy and position in HTP based on magnetic resonance (MR) images.

MATERIALS AND METHODS

All volunteers were scanned with an MR-compatible hyperthermia device, including a filled waterbolus, to replicate the treatment setup. In the planning setup, the volunteers were scanned without the device to reproduce the imaging in the current HTP. First, we used rigid registration to investigate the patient position displacements between the planning and treatment setup. Second, we performed HTP for the planning anatomy at both positions and the treatment mimicking anatomy to study the effects of positioning and anatomy on the quality of the simulated hyperthermia treatment. Treatment quality was evaluated using SAR-based parameters.

RESULTS

We found an average displacement of 2 cm between planning and treatment positions. These displacements caused average absolute differences of ∼12% for TC25 and 10.4%-15.9% in THQ. Furthermore, we found that including the accurate treatment position and anatomy in treatment planning led to an improvement of 2% in TC25 and 4.6%-10.6% in THQ.

CONCLUSIONS

This study showed that precise patient position and anatomy are relevant since these affect the accuracy of HTP predictions. The major part of improved accuracy is related to implementing the correct position of the patient in the applicator. Hence, our study shows a clear incentive to accurately match the patient position in HTP with the actual treatment.

Automated landmarking via multiple templates

Manually collecting landmarks for quantifying complex morphological phenotypes can be laborious and subject to intra and interobserver errors. However, most automated landmarking methods for efficiency and consistency fall short of landmarking highly variable samples due to the bias introduced by the use of a single template. We introduce a fast and open source automated landmarking pipeline (MALPACA) that utilizes multiple templates for accommodating large-scale variations. We also introduce a K-means method of choosing the templates that can be used in conjunction with MALPACA, when no prior information for selecting templates is available. Our results confirm that MALPACA significantly outperforms single-template methods in landmarking both single and multi-species samples. K-means based template selection can also avoid choosing the worst set of templates when compared to random template selection. We further offer an example of post-hoc quality check for each individual template for further refinement. In summary, MALPACA is an efficient and reproducible method that can accommodate large morphological variability, such as those commonly found in evolutionary studies. To support the research community, we have developed open-source and user-friendly software tools for performing K-means multi-templates selection and MALPACA.

A novel multi-atlas segmentation approach under the semi-supervised learning framework: Application to knee cartilage segmentation

BACKGROUND AND OBJECTIVE

Multi-atlas based segmentation techniques, which rely on an atlas library comprised of training images labeled by an expert, have proven their effectiveness in multiple automatic segmentation applications. However, the usage of exhaustive patch libraries combined with the voxel-wise labeling incur a large computational cost in terms of memory requirements and execution times.

METHODS

To confront this shortcoming, we propose a novel two-stage multi-atlas approach designed under the Semi-Supervised Learning (SSL) framework. The main properties of our method are as follows: First, instead of the voxel-wise labeling approach, the labeling of target voxels is accomplished here by exploiting the spectral content of globally sampled datasets from the target image, along with their spatially correspondent data collected from the atlases. Following SSL, voxels classification is boosted by incorporating unlabeled data from the target image, in addition to the labeled ones from atlas library. Our scheme integrates constructively fruitful concepts, including sparse reconstructions of voxels from linear neighborhoods, HOG feature descriptors of patches/regions, and label propagation via sparse graph constructions. Segmentation of the target image is carried out in two stages: stage-1 focuses on the sampling and labeling of global data, while stage-2 undertakes the above tasks for the out-of-sample data. Finally, we propose different graph-based methods for the labeling of global data, while these methods are extended to deal with the out-of-sample voxels.

RESULTS

A thorough experimental investigation is conducted on 76 subjects provided by the publicly accessible Osteoarthritis Initiative (OAI) repository. Comparative results and statistical analysis demonstrate that the suggested methodology exhibits superior segmentation performance compared to the existing patch-based methods, across all evaluation metrics (DSC:88.89%, Precision: 89.86%, Recall: 88.12%), while at the same time it requires a considerably reduced computational load (>70% reduction on average execution time with respect to other patch-based). In addition, our approach is favorably compared against other non patch-based and deep learning methods in terms of performance accuracy (on the 3-class problem). A final experimentation on a 5-class setting of the problems demonstrates that our approach is capable of achieving performance comparable to existing state-of-the-art knee cartilage segmentation methods (DSC:88.22% and DSC:85.84% for femoral and tibial cartilage respectively).

An Efficient Optimization Approach for Glioma Tumor Segmentation in Brain MRI

Glioma is an aggressive type of cancer that develops in the brain or spinal cord. Due to many differences in its shape and appearance, accurate segmentation of glioma for identifying all parts of the tumor and its surrounding cancerous tissues is a challenging task. In recent researches, the combination of multi-atlas segmentation and machine learning methods provides robust and accurate results by learning from annotated atlas datasets. To overcome the side effects of limited existing information on atlas-based segmentation, and the long training phase of learning methods, we proposed a semi-supervised unified framework for multi-label segmentation that formulates this problem in terms of a Markov Random Field energy optimization on a parametric graph. To evaluate the proposed framework, we apply it to publicly available BRATS datasets, including low- and high-grade glioma tumors. Experimental results indicate competitive performance compared to the state-of-the-art methods. Compared with the top ranked methods, the proposed framework obtains the best dice score for segmenting of "whole tumor" (WT), "tumor core" (TC ) and "enhancing active tumor" (ET) regions. The achieved accuracy is 94[Formula: see text] characterized by the mean dice score. The motivation of using MRF graph is to map the segmentation problem to an optimization model in a graphical environment. Therefore, by defining perfect graph structure and optimum constraints and flows in the continuous max-flow model, the segmentation is performed precisely.

The role of artificial intelligence in veterinary radiation oncology

Veterinary radiation oncology regularly deploys sophisticated contouring, image registration, and treatment planning optimization software for patient care. Over the past decade, advances in computing power and the rapid development of neural networks, open-source software packages, and data science have been realized and resulted in new research and clinical applications of artificial intelligent (AI) systems in radiation oncology. These technologies differ from conventional software in their level of complexity and ability to learn from representative and local data. We provide clinical and research application examples of AI in human radiation oncology and their potential applications in veterinary medicine throughout the patient's care-path: from treatment simulation, deformable registration, auto-segmentation, automated treatment planning and plan selection, quality assurance, adaptive radiotherapy, and outcomes modeling. These technologies have the potential to offer significant time and cost savings in the veterinary setting; however, since the range of usefulness of these technologies have not been well studied nor understood, care must be taken if adopting AI technologies in clinical practice. Over the next several years, some practical and realizable applications of AI in veterinary radiation oncology include automated segmentation of normal tissues and tumor volumes, deformable registration, multi-criteria plan optimization, and adaptive radiotherapy. Keys in achieving success in adopting AI in veterinary radiation oncology include: establishing "truth-data"; data harmonization; multi-institutional data and collaborations; standardized dose reporting and taxonomy; adopting an open access philosophy, data collection and curation; open-source algorithm development; and transparent and platform-independent code development.

Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis

Recent self-supervised advances in medical computer vision exploit the global and local anatomical self-similarity for pretraining prior to downstream tasks such as segmentation. However, current methods assume i.i.d. image acquisition, which is invalid in clinical study designs where follow-up longitudinal scans track subject-specific temporal changes. Further, existing self-supervised methods for medically-relevant image-to-image architectures exploit only spatial or temporal self-similarity and do so via a loss applied only at a single image-scale, with naive multi-scale spatiotemporal extensions collapsing to degenerate solutions. To these ends, this paper makes two contributions: (1) It presents a local and multi-scale spatiotemporal representation learning method for image-to-image architectures trained on longitudinal images. It exploits the spatiotemporal self-similarity of learned multi-scale intra-subject image features for pretraining and develops several feature-wise regularizations that avoid degenerate representations; (2) During finetuning, it proposes a surprisingly simple self-supervised segmentation consistency regularization to exploit intra-subject correlation. Benchmarked across various segmentation tasks, the proposed framework outperforms both well-tuned randomly-initialized baselines and current self-supervised techniques designed for both i.i.d. and longitudinal datasets. These improvements are demonstrated across both longitudinal neurodegenerative adult MRI and developing infant brain MRI and yield both higher performance and longitudinal consistency.

An end-to-end multi-task system of automatic lesion detection and anatomical localization in whole-body bone scintigraphy by deep learning

SUMMARY

Limited by spatial resolution and visual contrast, bone scintigraphy interpretation is susceptible to subjective factors, which considerably affects the accuracy and repeatability of lesion detection and anatomical localization. In this work, we design and implement an end-to-end multi-task deep learning model to perform automatic lesion detection and anatomical localization in whole-body bone scintigraphy. A total of 617 whole-body bone scintigraphy cases including anterior and posterior views were retrospectively analyzed. The proposed semi-supervised model consists of two task flows. The first one, the lesion segmentation flow, received image patches and was trained in a supervised way. The other one, skeleton segmentation flow, was trained on as few as five labeled images in conjunction with the multi-atlas approach, in a semi-supervised way. The two flows joint in their encoder layers so each flow can capture more generalized distribution of the sample space and extract more abstract deep features. The experimental results show that the architecture achieved the highest precision in the finest bone segmentation task in both anterior and posterior images of whole-body scintigraphy. Such an end-to-end approach with very few manual annotation requirement would be suitable for algorithm deployment. Moreover, the proposed approach reliably balances unsupervised labels construction and supervised learning, providing useful insight for weakly labeled image analysis.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

An image registration method for voxel-wise analysis of whole-body oncological PET-CT

Whole-body positron emission tomography-computed tomography (PET-CT) imaging in oncology provides comprehensive information of each patient's disease status. However, image interpretation of volumetric data is a complex and time-consuming task. In this work, an image registration method targeted towards computer-aided voxel-wise analysis of whole-body PET-CT data was developed. The method used both CT images and tissue segmentation masks in parallel to spatially align images step-by-step. To evaluate its performance, a set of baseline PET-CT images of 131 classical Hodgkin lymphoma (cHL) patients and longitudinal image series of 135 head and neck cancer (HNC) patients were registered between and within subjects according to the proposed method. Results showed that major organs and anatomical structures generally were registered correctly. Whole-body inverse consistency vector and intensity magnitude errors were on average less than 5 mm and 45 Hounsfield units respectively in both registration tasks. Image registration was feasible in time and the nearly automatic pipeline enabled efficient image processing. Metabolic tumor volumes of the cHL patients and registration-derived therapy-related tissue volume change of the HNC patients mapped to template spaces confirmed proof-of-concept. In conclusion, the method established a robust point-correspondence and enabled quantitative visualization of group-wise image features on voxel level.

High-resolution atlasing and segmentation of the subcortex: Review and perspective on challenges and opportunities created by machine learning

This paper reviews almost three decades of work on atlasing and segmentation methods for subcortical structures in human brain MRI. In writing this survey, we have three distinct aims. First, to document the evolution of digital subcortical atlases of the human brain, from the early MRI templates published in the nineties, to the complex multi-modal atlases at the subregion level that are available today. Second, to provide a detailed record of related efforts in the automated segmentation front, from earlier atlas-based methods to modern machine learning approaches. And third, to present a perspective on the future of high-resolution atlasing and segmentation of subcortical structures in in vivo human brain MRI, including open challenges and opportunities created by recent developments in machine learning.

Clinical target volume segmentation based on gross tumor volume using deep learning for head and neck cancer treatment

Accurate clinical target volume (CTV) delineation is important for head and neck intensity-modulated radiation therapy. However, delineation is time-consuming and susceptible to interobserver variability (IOV). Based on a manual contouring process commonly used in clinical practice, we developed a deep learning (DL)-based method to delineate a low-risk CTV with computed tomography (CT) and gross tumor volume (GTV) input and compared it with a CT-only input. A total of 310 patients with oropharynx cancer were randomly divided into the training set (250) and test set (60). The low-risk CTV and primary GTV contours were used to generate label data for the input and ground truth. A 3D U-Net with a two-channel input of CT and GTV (U-Net_GTV) was proposed and its performance was compared with a U-Net with only CT input (U-Net_CT). The Dice similarity coefficient (DSC) and average Hausdorff distance (AHD) were evaluated. The time required to predict the CTV was 0.86 s per patient. U-Net_GTV showed a significantly higher mean DSC value than U-Net_CT (0.80 ± 0.03 and 0.76 ± 0.05) and a significantly lower mean AHD value (3.0 ± 0.5 mm vs 3.5 ± 0.7 mm). Compared to the existing DL method with only CT input, the proposed GTV-based segmentation using DL showed a more precise low-risk CTV segmentation for head and neck cancer. Our findings suggest that the proposed method could reduce the contouring time of a low-risk CTV, allowing the standardization of target delineations for head and neck cancer.

The role of the hippocampus in statistical learning and language recovery in persons with post stroke aphasia

Although several studies have aimed for accurate predictions of language recovery in post stroke aphasia, individual language outcomes remain hard to predict. Large-scale prediction models are built using data from patients mainly in the chronic phase after stroke, although it is clinically more relevant to consider data from the acute phase. Previous research has mainly focused on deficits, i.e., behavioral deficits or specific brain damage, rather than compensatory mechanisms, i.e., intact cognitive skills or undamaged brain regions. One such unexplored brain region that might support language (re)learning in aphasia is the hippocampus, a region that has commonly been associated with an individual's learning potential, including statistical learning. This refers to a set of mechanisms upon which we rely heavily in daily life to learn a range of regularities across cognitive domains. Against this background, thirty-three patients with aphasia (22 males and 11 females, M = 69.76 years, SD = 10.57 years) were followed for 1 year in the acute (1-2 weeks), subacute (3-6 months) and chronic phase (9-12 months) post stroke. We evaluated the unique predictive value of early structural hippocampal measures for short-term and long-term language outcomes (measured by the ANELT). In addition, we investigated whether statistical learning abilities were intact in patients with aphasia using three different tasks: an auditory-linguistic and visual task based on the computation of transitional probabilities and a visuomotor serial reaction time task. Finally, we examined the association of individuals' statistical learning potential with acute measures of hippocampal gray and white matter. Using Bayesian statistics, we found moderate evidence for the contribution of left hippocampal gray matter in the acute phase to the prediction of long-term language outcomes, over and above information on the lesion and the initial language deficit (measured by the ScreeLing). Non-linguistic statistical learning in patients with aphasia, measured in the subacute phase, was intact at the group level compared to 23 healthy older controls (8 males and 15 females, M = 74.09 years, SD = 6.76 years). Visuomotor statistical learning correlated with acute hippocampal gray and white matter. These findings reveal that particularly left hippocampal gray matter in the acute phase is a potential marker of language recovery after stroke, possibly through its statistical learning ability.

Multi-atlas segmentation and quantification of muscle, bone and subcutaneous adipose tissue in the lower leg using peripheral quantitative computed tomography

Accurate and reproducible tissue identification is essential for understanding structural and functional changes that may occur naturally with aging, or because of a chronic disease, or in response to intervention therapies. Peripheral quantitative computed tomography (pQCT) is regularly employed for body composition studies, especially for the structural and material properties of the bone. Furthermore, pQCT acquisition requires low radiation dose and the scanner is compact and portable. However, pQCT scans have limited spatial resolution and moderate SNR. pQCT image quality is frequently degraded by involuntary subject movement during image acquisition. These limitations may often compromise the accuracy of tissue quantification, and emphasize the need for automated and robust quantification methods. We propose a tissue identification and quantification methodology that addresses image quality limitations and artifacts, with increased interest in subject movement. We introduce a multi-atlas image segmentation (MAIS) framework for semantic segmentation of hard and soft tissues in pQCT scans at multiple levels of the lower leg. We describe the stages of statistical atlas generation, deformable registration and multi-tissue classifier fusion. We evaluated the performance of our methodology using multiple deformable registration approaches against reference tissue masks. We also evaluated the performance of conventional model-based segmentation against the same reference data to facilitate comparisons. We studied the effect of subject movement on tissue segmentation quality. We also applied the top performing method to a larger out-of-sample dataset and report the quantification results. The results show that multi-atlas image segmentation with diffeomorphic deformation and probabilistic label fusion produces very good quality over all tissues, even for scans with significant quality degradation. The application of our technique to the larger dataset reveals trends of age-related body composition changes that are consistent with the literature. Because of its robustness to subject motion artifacts, our MAIS methodology enables analysis of larger number of scans than conventional state-of-the-art methods. Automated analysis of both soft and hard tissues in pQCT is another contribution of this work.

AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem?

With the unprecedented developments in deep learning, automatic segmentation of main abdominal organs seems to be a solved problem as state-of-the-art (SOTA) methods have achieved comparable results with inter-rater variability on many benchmark datasets. However, most of the existing abdominal datasets only contain single-center, single-phase, single-vendor, or single-disease cases, and it is unclear whether the excellent performance can generalize on diverse datasets. This paper presents a large and diverse abdominal CT organ segmentation dataset, termed AbdomenCT-1K, with more than 1000 (1K) CT scans from 12 medical centers, including multi-phase, multi-vendor, and multi-disease cases. Furthermore, we conduct a large-scale study for liver, kidney, spleen, and pancreas segmentation and reveal the unsolved segmentation problems of the SOTA methods, such as the limited generalization ability on distinct medical centers, phases, and unseen diseases. To advance the unsolved problems, we further build four organ segmentation benchmarks for fully supervised, semi-supervised, weakly supervised, and continual learning, which are currently challenging and active research topics. Accordingly, we develop a simple and effective method for each benchmark, which can be used as out-of-the-box methods and strong baselines. We believe the AbdomenCT-1K dataset will promote future in-depth research towards clinical applicable abdominal organ segmentation methods.

Deep active learning for suggestive segmentation of biomedical image stacks via optimisation of Dice scores and traced boundary length

Manual segmentation of stacks of 2D biomedical images (e.g., histology) is a time-consuming task which can be sped up with semi-automated techniques. In this article, we present a suggestive deep active learning framework that seeks to minimise the annotation effort required to achieve a certain level of accuracy when labelling such a stack. The framework suggests, at every iteration, a specific region of interest (ROI) in one of the images for manual delineation. Using a deep segmentation neural network and a mixed cross-entropy loss function, we propose a principled strategy to estimate class probabilities for the whole stack, conditioned on heterogeneous partial segmentations of the 2D images, as well as on weak supervision in the form of image indices that bound each ROI. Using the estimated probabilities, we propose a novel active learning criterion based on predictions for the estimated segmentation performance and delineation effort, measured with average Dice scores and total delineated boundary length, respectively, rather than common surrogates such as entropy. The query strategy suggests the ROI that is expected to maximise the ratio between performance and effort, while considering the adjacency of structures that may have already been labelled - which decrease the length of the boundary to trace. We provide quantitative results on synthetically deformed MRI scans and real histological data, showing that our framework can reduce labelling effort by up to 60-70% without compromising accuracy.

Advances and Innovations in Ablative Head and Neck Oncologic Surgery Using Mixed Reality Technologies in Personalized Medicine

The benefit of computer-assisted planning in head and neck ablative and reconstructive surgery has been extensively documented over the last decade. This approach has been proven to offer a more secure surgical procedure. In the treatment of cancer of the head and neck, computer-assisted surgery can be used to visualize and estimate the location and extent of the tumor mass. Nowadays, some software tools even allow the visualization of the structures of interest in a mixed reality environment. However, the precise integration of mixed reality systems into a daily clinical routine is still a challenge. To date, this technology is not yet fully integrated into clinical settings such as the tumor board, surgical planning for head and neck tumors, or medical and surgical education. As a consequence, the handling of these systems is still of an experimental nature, and decision-making based on the presented data is not yet widely used. The aim of this paper is to present a novel, user-friendly 3D planning and mixed reality software and its potential application for ablative and reconstructive head and neck surgery.

Integrated 3d flow-based multi-atlas brain structure segmentation

MRI brain structure segmentation plays an important role in neuroimaging studies. Existing methods either spend much CPU time, require considerable annotated data, or fail in segmenting volumes with large deformation. In this paper, we develop a novel multi-atlas-based algorithm for 3D MRI brain structure segmentation. It consists of three modules: registration, atlas selection and label fusion. Both registration and label fusion leverage an integrated flow based on grayscale and SIFT features. We introduce an effective and efficient strategy for atlas selection by employing the accompanying energy generated in the registration step. A 3D sequential belief propagation method and a 3D coarse-to-fine flow matching approach are developed in both registration and label fusion modules. The proposed method is evaluated on five public datasets. The results show that it has the best performance in almost all the settings compared to competitive methods such as ANTs, Elastix, Learning to Rank and Joint Label Fusion. Moreover, our registration method is more than 7 times as efficient as that of ANTs SyN, while our label transfer method is 18 times faster than Joint Label Fusion in CPU time. The results on the ADNI dataset demonstrate that our method is applicable to image pairs that require a significant transformation in registration. The performance on a composite dataset suggests that our method succeeds in a cross-modality manner. The results of this study show that the integrated 3D flow-based method is effective and efficient for brain structure segmentation. It also demonstrates the power of SIFT features, multi-atlas segmentation and classical machine learning algorithms for a medical image analysis task. The experimental results on public datasets show the proposed method's potential for general applicability in various brain structures and settings.

Evaluation of auto-segmentation for EBRT planning structures using deep learning-based workflow on cervical cancer

Deep learning (DL) based approach aims to construct a full workflow solution for cervical cancer with external beam radiation therapy (EBRT) and brachytherapy (BT). The purpose of this study was to evaluate the accuracy of EBRT planning structures derived from DL based auto-segmentation compared with standard manual delineation. Auto-segmentation model based on convolutional neural networks (CNN) was developed to delineate clinical target volumes (CTVs) and organs at risk (OARs) in cervical cancer radiotherapy. A total of 300 retrospective patients from multiple cancer centers were used to train and validate the model, and 75 independent cases were selected as testing data. The accuracy of auto-segmented contours were evaluated using geometric and dosimetric metrics including dice similarity coefficient (DSC), 95% hausdorff distance (95%HD), jaccard coefficient (JC) and dose-volume index (DVI). The correlation between geometric metrics and dosimetric difference was performed by Spearman’s correlation analysis. The right and left kidney, bladder, right and left femoral head showed superior geometric accuracy (DSC: 0.88–0.93; 95%HD: 1.03 mm–2.96 mm; JC: 0.78–0.88), and the Bland–Altman test obtained dose agreement for these contours (P > 0.05) between manual and DL based methods. Wilcoxon’s signed-rank test indicated significant dosimetric differences in CTV, spinal cord and pelvic bone (P < 0.001). A strong correlation between the mean dose of pelvic bone and its 95%HD (R = 0.843, P < 0.001) was found in Spearman’s correlation analysis, and the remaining structures showed weak link between dosimetric difference and all of geometric metrics. Our auto-segmentation achieved a satisfied agreement for most EBRT planning structures, although the clinical acceptance of CTV was a concern. DL based auto-segmentation was an essential component in cervical cancer workflow which would generate the accurate contouring.

Joint attention in infants at high familial risk for autism spectrum disorder and the association with thalamic and hippocampal macrostructure

Autism spectrum disorder (ASD) is a heritable neurodevelopmental disorder. Infants diagnosed with ASD can show impairments in spontaneous gaze-following and will seldom engage in joint attention (JA). The ability to initiate JA (IJA) can be more significantly impaired than the ability to respond to JA (RJA). In a longitudinal study, 101 infants who had a familial risk for ASD were enrolled (62% males). Participants completed magnetic resonance imaging scans at 4 or 6 months of age. Subcortical volumes (thalamus, hippocampus, amygdala, basal ganglia, ventral diencephalon, and cerebellum) were automatically extracted. Early gaze and JA behaviors were assessed with standardized measures. The majority of infants were IJA nonresponders (n = 93, 92%), and over half were RJA nonresponders (n = 50, 52%). In the nonresponder groups, models testing the association of subcortical volumes with later ASD diagnosis accounted for age, sex, and cerebral volumes. In the nonresponder IJA group, using regression method, the left hippocampus (B = −0.009, aOR = 0.991, P = 0.025), the right thalamus (B = −0.016, aOR = 0.984, P = 0.026), as well as the left thalamus (B = 0.015, aOR = 1.015, P = 0.019), predicted later ASD diagnosis. Alterations in thalamic and hippocampal macrostructure in at-risk infants who do not engage in IJA may reflect an enhanced vulnerability and may be the key predictors of later ASD development.

Comparison of skeletal segmentation by deep learning-based and atlas-based segmentation in prostate cancer patients

OBJECTIVE

We aimed to compare the deep learning-based (VSBONE BSI) and atlas-based (BONENAVI) segmentation accuracy that have been developed to measure the bone scan index based on skeletal segmentation.

METHODS

We retrospectively conducted bone scans for 383 patients with prostate cancer. These patients were divided into two groups: 208 patients were injected with ^99mTc-hydroxymethylene diphosphonate processed by VSBONE BSI, and 175 patients were injected with ^99mTc-methylene diphosphonate processed by BONENAVI. Three observers classified the skeletal segmentations as either a "Match" or "Mismatch" in the following regions: the skull, cervical vertebrae, thoracic vertebrae, lumbar vertebrae, pelvis, sacrum, humerus, rib, sternum, clavicle, scapula, and femur. Segmentation error was defined if two or more observers selected "Mismatch" in the same region. We calculated the segmentation error rate according to each administration group and evaluated the presence of hot spots suspected bone metastases in "Mismatch" regions. Multivariate logistic regression analysis was used to determine the association between segmentation error and variables like age, uptake time, total counts, extent of disease, and gamma cameras.

RESULTS

The regions of "Mismatch" were more common in the long tube bones for VSBONE BSI and in the pelvis and axial skeletons for BONENAVI. Segmentation error was observed in 49 cases (23.6%) with VSBONE BSI and 58 cases (33.1%) with BONENAVI. VSBONE BSI tended that "Mismatch" regions contained hot spots suspected of bone metastases in patients with multiple bone metastases and showed that patients with higher extent of disease (odds ratio = 8.34) were associated with segmentation error in multivariate logistic regression analysis.

CONCLUSIONS

VSBONE BSI has a potential to be higher segmentation accuracy compared with BONENAVI. However, the segmentation error in VSBONE BSI occurred dependent on bone metastases burden. We need to be careful when evaluating multiple bone metastases using VSBONE BSI.

Synthesizing pseudo-T2w images to recapture missing data in neonatal neuroimaging with applications in rs-fMRI

T1- and T2-weighted (T1w and T2w) images are essential for tissue classification and anatomical localization in Magnetic Resonance Imaging (MRI) analyses. However, these anatomical data can be challenging to acquire in non-sedated neonatal cohorts, which are prone to high amplitude movement and display lower tissue contrast than adults. As a result, one of these modalities may be missing or of such poor quality that they cannot be used for accurate image processing, resulting in subject loss. While recent literature attempts to overcome these issues in adult populations using synthetic imaging approaches, evaluation of the efficacy of these methods in pediatric populations and the impact of these techniques in conventional MR analyses has not been performed. In this work, we present two novel methods to generate pseudo-T2w images: the first is based in deep learning and expands upon previous models to 3D imaging without the requirement of paired data, the second is based in nonlinear multi-atlas registration providing a computationally lightweight alternative. We demonstrate the anatomical accuracy of pseudo-T2w images and their efficacy in existing MR processing pipelines in two independent neonatal cohorts. Critically, we show that implementing these pseudo-T2w methods in resting-state functional MRI analyses produces virtually identical functional connectivity results when compared to those resulting from T2w images, confirming their utility in infant MRI studies for salvaging otherwise lost subject data.

Automated segmentation of the larynx on computed tomography images: a review

The larynx, or the voice-box, is a common site of occurrence of Head and Neck cancers. Yet, automated segmentation of the larynx has been receiving very little attention. Segmentation of organs is an essential step in cancer treatment-planning. Computed Tomography scans are routinely used to assess the extent of tumor spread in the Head and Neck as they are fast to acquire and tolerant to some movement. This paper reviews various automated detection and segmentation methods used for the larynx on Computed Tomography images. Image registration and deep learning approaches to segmenting the laryngeal anatomy are compared, highlighting their strengths and shortcomings. A list of available annotated laryngeal computed tomography datasets is compiled for encouraging further research. Commercial software currently available for larynx contouring are briefed in our work. We conclude that the lack of standardisation on larynx boundaries and the complexity of the relatively small structure makes automated segmentation of the larynx on computed tomography images a challenge. Reliable computer aided intervention in the contouring and segmentation process will help clinicians easily verify their findings and look for oversight in diagnosis. This review is useful for research that works with artificial intelligence in Head and Neck cancer, specifically that deals with the segmentation of laryngeal anatomy.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13534-022-00221-3.

Deep learning-based segmentation of knee MRI for fully automatic subregional morphological assessment of cartilage tissues: Data from the Osteoarthritis Initiative

Morphological changes in knee cartilage subregions are valuable imaging-based biomarkers for understanding progression of osteoarthritis, and they are typically detected from magnetic resonance imaging (MRI). So far, accurate segmentation of cartilage has been done manually. Deep learning approaches show high promise in automating the task; however, they lack clinically relevant evaluation. We introduce a fully automatic method for segmentation and subregional assessment of articular cartilage, and evaluate its predictive power in context of radiographic osteoarthritis progression. Two data sets of 3D double-echo steady-state (DESS) MRI derived from the Osteoarthritis Initiative were used: first, n = 88; second, n = 600, 0-/12-/24-month visits. Our method performed deep learning-based segmentation of knee cartilage tissues, their subregional division via multi-atlas registration, and extraction of subregional volume and thickness. The segmentation model was developed and assessed on the first data set. Subsequently, on the second data set, the morphological measurements from our and the prior methods were analyzed in correlation and agreement, and, eventually, by their discriminative power of radiographic osteoarthritis progression over 12 and 24 months, retrospectively. The segmentation model showed very high correlation (r > 0.934) and agreement (mean difference < 116 mm³ ) in volumetric measurements with the reference segmentations. Comparison of our and manual segmentation methods yielded r = 0.845-0.973 and mean differences = 262-501 mm³ for weight-bearing cartilage volume, and r = 0.770-0.962 and mean differences = 0.513-1.138 mm for subregional cartilage thickness. With regard to osteoarthritis progression, our method found most of the significant associations identified using the manual segmentation method, for both 12- and 24-month subregional cartilage changes. The method may be effectively applied in osteoarthritis progression studies to extract cartilage-related imaging biomarkers.

Efficient Johnson-SB Mixture Model for Segmentation of CT Liver Image

To overcome the problem that the traditional Gaussian mixture model (GMM) cannot well describe the skewness distribution of the gray-level histogram of a liver CT slice, we propose a novel segmentation method for liver CT images by introducing the Johnson-SB mixture model (J_SBMM). The Johnson-SB model not only has a flexible asymmetrical distribution but also covers a variety of other distributions as well. In this article, the parameter optimization formulas for J_SBMM were derived by employing the expectation-maximization (EM) algorithm and maximum likelihood. The implementation process of the J_SBMM-based segmentation algorithm is provided in detail. To make better use of the skewness of Johnson-SB and improve the segmentation accuracy, we devise an idea to divide the histogram into two parts and calculate the segmentation threshold for each part, respectively, which is called J_SBMM-TDH. By analyzing and comparing the segmentation thresholds with different cluster numbers, it is illustrated that the segmentation threshold of J_SBMM-TDH will tend to be stable with the increasing of cluster number, while that of GMM is sensitive to different cluster numbers. The proposed J_SBMM-TDH is applied to segment four randomly obtained abdominal CT image sequences, and the segmentation results and robustness have been compared between J_SBMM-TDH and GMM. It is verified that J_SBMM-TDH has preferable segmentation results and better robustness than GMM for the segmentation of liver CT images.

MRA-free intracranial vessel localization on MR vessel wall images

Analysis of vessel morphology is important in assessing intracranial atherosclerosis disease (ICAD). Recently, magnetic resonance (MR) vessel wall imaging (VWI) has been introduced to image ICAD and characterize morphology for atherosclerotic lesions. In order to automatically perform quantitative analysis on VWI data, MR angiography (MRA) acquired in the same imaging session is typically used to localize the vessel segments of interest. However, MRA may be unavailable caused by the lack or failure of the sequence in a VWI protocol. This study aims to investigate the feasibility to infer the vessel location directly from VWI. We propose to synergize an atlas-based method to preserve general vessel structure topology with a deep learning network in the motion field domain to correct the residual geometric error. Performance is quantified by examining the agreement between the extracted vessel structures from the pair-acquired and alignment-corrected angiogram, and the estimated output using a cross-validation scheme. Our proposed pipeline yields clinically feasible performance in localizing intracranial vessels, demonstrating the promise of performing vessel morphology analysis using VWI alone.

Multiscale Local Enhancement Deep Convolutional Networks for the Automated 3D Segmentation of Gross Tumor Volumes in Nasopharyngeal Carcinoma: A Multi-Institutional Dataset Study

Purpose

Accurate segmentation of gross target volume (GTV) from computed tomography (CT) images is a prerequisite in radiotherapy for nasopharyngeal carcinoma (NPC). However, this task is very challenging due to the low contrast at the boundary of the tumor and the great variety of sizes and morphologies of tumors between different stages. Meanwhile, the data source also seriously affect the results of segmentation. In this paper, we propose a novel three-dimensional (3D) automatic segmentation algorithm that adopts cascaded multiscale local enhancement of convolutional neural networks (CNNs) and conduct experiments on multi-institutional datasets to address the above problems.

Materials and Methods

In this study, we retrospectively collected CT images of 257 NPC patients to test the performance of the proposed automatic segmentation model, and conducted experiments on two additional multi-institutional datasets. Our novel segmentation framework consists of three parts. First, the segmentation framework is based on a 3D Res-UNet backbone model that has excellent segmentation performance. Then, we adopt a multiscale dilated convolution block to enhance the receptive field and focus on the target area and boundary for segmentation improvement. Finally, a central localization cascade model for local enhancement is designed to concentrate on the GTV region for fine segmentation to improve the robustness. The Dice similarity coefficient (DSC), positive predictive value (PPV), sensitivity (SEN), average symmetric surface distance (ASSD) and 95% Hausdorff distance (HD95) are utilized as qualitative evaluation criteria to estimate the performance of our automated segmentation algorithm.

Results

The experimental results show that compared with other state-of-the-art methods, our modified version 3D Res-UNet backbone has excellent performance and achieves the best results in terms of the quantitative metrics DSC, PPR, ASSD and HD95, which reached 74.49 ± 7.81%, 79.97 ± 13.90%, 1.49 ± 0.65 mm and 5.06 ± 3.30 mm, respectively. It should be noted that the receptive field enhancement mechanism and cascade architecture can have a great impact on the stable output of automatic segmentation results with high accuracy, which is critical for an algorithm. The final DSC, SEN, ASSD and HD95 values can be increased to 76.23 ± 6.45%, 79.14 ± 12.48%, 1.39 ± 5.44mm, 4.72 ± 3.04mm. In addition, the outcomes of multi-institution experiments demonstrate that our model is robust and generalizable and can achieve good performance through transfer learning.

Conclusions

The proposed algorithm could accurately segment NPC in CT images from multi-institutional datasets and thereby may improve and facilitate clinical applications.

Browsing Multiple Subjects When the Atlas Adaptation Cannot Be Achieved via a Warping Strategy

Brain mapping studies often need to identify brain structures or functional circuits into a set of individual brains. To this end, multiple atlases have been published to represent such structures based on different modalities, subject sets, and techniques. The mainstream approach to exploit these atlases consists in spatially deforming each individual data onto a given atlas using dense deformation fields, which supposes the existence of a continuous mapping between atlases and individuals. However, this continuity is not always verified, and this "iconic" approach has limits. We present in this study an alternative, complementary, "structural" approach, which consists in extracting structures from the individual data, and comparing them without deformation. A "structural atlas" is thus a collection of annotated individual data with a common structure nomenclature. It may be used to characterize structure shape variability across individuals or species, or to train machine learning systems. This study exhibits Anatomist, a powerful structural 3D visualization software dedicated to building, exploring, and editing structural atlases involving a large number of subjects. It has been developed primarily to decipher the cortical folding variability; cortical sulci vary enormously in both size and shape, and some may be missing or have various topologies, which makes iconic approaches inefficient to study them. We, therefore, had to build structural atlases for cortical sulci, and use them to train sulci identification algorithms. Anatomist can display multiple subject data in multiple views, supports all kinds of neuroimaging data, including compound structural object graphs, handles arbitrary coordinate transformation chains between data, and has multiple display features. It is designed as a programming library in both C++ and Python languages, and may be extended or used to build dedicated custom applications. Its generic design makes all the display and structural aspects used to explore the variability of the cortical folding pattern work in other applications, for instance, to browse axonal fiber bundles, deep nuclei, functional activations, or other kinds of cortical parcellations. Multimodal, multi-individual, or inter-species display is supported, and adaptations to large scale screen walls have been developed. These very original features make it a unique viewer for structural atlas browsing.

Perivascular space dilation is associated with vascular amyloid-β accumulation in the overlying cortex

Perivascular spaces (PVS) are compartments surrounding cerebral blood vessels that become visible on MRI when enlarged. Enlarged PVS (EPVS) are commonly seen in patients with cerebral small vessel disease (CSVD) and have been suggested to reflect dysfunctional perivascular clearance of soluble waste products from the brain. In this study, we investigated histopathological correlates of EPVS and how they relate to vascular amyloid-β (Aβ) in cerebral amyloid angiopathy (CAA), a form of CSVD that commonly co-exists with Alzheimer's disease (AD) pathology. We used ex vivo MRI, semi-automatic segmentation and validated deep-learning-based models to quantify EPVS and associated histopathological abnormalities. Severity of MRI-visible PVS during life was significantly associated with severity of MRI-visible PVS on ex vivo MRI in formalin fixed intact hemispheres and corresponded with PVS enlargement on histopathology in the same areas. EPVS were located mainly around the white matter portion of perforating cortical arterioles and their burden was associated with CAA severity in the overlying cortex. Furthermore, we observed markedly reduced smooth muscle cells and increased vascular Aβ accumulation, extending into the WM, in individually affected vessels with an EPVS. Overall, these findings are consistent with the notion that EPVS reflect impaired outward flow along arterioles and have implications for our understanding of perivascular clearance mechanisms, which play an important role in the pathophysiology of CAA and AD.

Automatic Segmentation of Bone Selective MR Images for Visualization and Craniometry of the Cranial Vault

RATIONALE AND OBJECTIVES

Solid-state MRI has been shown to provide a radiation-free alternative imaging strategy to CT. However, manual image segmentation to produce bone-selective MR-based 3D renderings is time and labor intensive, thereby acting as a bottleneck in clinical practice. The objective of this study was to evaluate an automatic multi-atlas segmentation pipeline for use on cranial vault images entirely circumventing prior manual intervention, and to assess concordance of craniometric measurements between pipeline produced MRI and CT-based 3D skull renderings.

MATERIALS AND METHODS

Dual-RF, dual-echo, 3D UTE pulse sequence MR data were obtained at 3T on 30 healthy subjects along with low-dose CT images between December 2018 to January 2020 for this prospective study. The four-point MRI datasets (two RF pulse widths and two echo times) were combined to produce bone-specific images. CT images were thresholded and manually corrected to segment the cranial vault. CT images were then rigidly registered to MRI using mutual information. The corresponding cranial vault segmentations were then transformed to MRI. The "ground truth" segmentations served as reference for the MR images. Subsequently, an automated multi-atlas pipeline was used to segment the bone-selective images. To compare manually and automatically segmented MR images, the Dice similarity coefficient (DSC) and Hausdorff distance (HD) were computed, and craniometric measurements between CT and automated-pipeline MRI-based segmentations were examined via Lin's concordance coefficient (LCC).

RESULTS

Automated segmentation reduced the need for an expert to obtain segmentation. Average DSC was 90.86 ± 1.94%, and average 95th percentile HD was 1.65 ± 0.44 mm between ground truth and automated segmentations. MR-based measurements differed from CT-based measurements by 0.73-1.2 mm on key craniometric measurements. LCC for distances between CT and MR-based landmarks were vertex-basion: 0.906, left-right frontozygomatic suture: 0.780, and glabella-opisthocranium: 0.956 for the three measurements.

CONCLUSION

Good agreement between CT and automated MR-based 3D cranial vault renderings has been achieved, thereby eliminating the laborious manual segmentation process. Target applications comprise craniofacial surgery as well as imaging of traumatic injuries and masses involving both bone and soft tissue.

Convolutional Neural Networks Enable Robust Automatic Segmentation of the Rat Hippocampus in MRI After Traumatic Brain Injury

Registration-based methods are commonly used in the automatic segmentation of magnetic resonance (MR) brain images. However, these methods are not robust to the presence of gross pathologies that can alter the brain anatomy and affect the alignment of the atlas image with the target image. In this work, we develop a robust algorithm, MU-Net-R, for automatic segmentation of the normal and injured rat hippocampus based on an ensemble of U-net-like Convolutional Neural Networks (CNNs). MU-Net-R was trained on manually segmented MR images of sham-operated rats and rats with traumatic brain injury (TBI) by lateral fluid percussion. The performance of MU-Net-R was quantitatively compared with methods based on single and multi-atlas registration using MR images from two large preclinical cohorts. Automatic segmentations using MU-Net-R and multi-atlas registration were of excellent quality, achieving cross-validated Dice scores above 0.90 despite the presence of brain lesions, atrophy, and ventricular enlargement. In contrast, the performance of single-atlas segmentation was unsatisfactory (cross-validated Dice scores below 0.85). Interestingly, the registration-based methods were better at segmenting the contralateral than the ipsilateral hippocampus, whereas MU-Net-R segmented the contralateral and ipsilateral hippocampus equally well. We assessed the progression of hippocampal damage after TBI by using our automatic segmentation tool. Our data show that the presence of TBI, time after TBI, and whether the hippocampus was ipsilateral or contralateral to the injury were the parameters that explained hippocampal volume.

Cross-Modality Multi-Atlas Segmentation Using Deep Neural Networks

Multi-atlas segmentation (MAS) is a promising framework for medical image segmentation. Generally, MAS methods register multiple atlases, i.e., medical images with corresponding labels, to a target image; and the transformed atlas labels can be combined to generate target segmentation via label fusion schemes. Many conventional MAS methods employed the atlases from the same modality as the target image. However, the number of atlases with the same modality may be limited or even missing in many clinical applications. Besides, conventional MAS methods suffer from the computational burden of registration or label fusion procedures. In this work, we design a novel cross-modality MAS framework, which uses available atlases from a certain modality to segment a target image from another modality. To boost the computational efficiency of the framework, both the image registration and label fusion are achieved by well-designed deep neural networks. For the atlas-to-target image registration, we propose a bi-directional registration network (BiRegNet), which can efficiently align images from different modalities. For the label fusion, we design a similarity estimation network (SimNet), which estimates the fusion weight of each atlas by measuring its similarity to the target image. SimNet can learn multi-scale information for similarity estimation to improve the performance of label fusion. The proposed framework was evaluated by the left ventricle and liver segmentation tasks on the MM-WHS and CHAOS datasets, respectively. Results have shown that the framework is effective for cross-modality MAS in both registration and label fusion. The code will be released publicly on https://github.com/NanYoMy/cmmas once the manuscript is accepted.

Pancreas MRI Segmentation Into Head, Body, and Tail Enables Regional Quantitative Analysis of Heterogeneous Disease

BACKGROUND

Quantitative imaging studies of the pancreas have often targeted the three main anatomical segments, head, body, and tail, using manual region of interest strategies to assess geographic heterogeneity. Existing automated analyses have implemented whole-organ segmentation, providing overall quantification but failing to address spatial heterogeneity.

PURPOSE

To develop and validate an automated method for pancreas segmentation into head, body, and tail subregions in abdominal MRI.

STUDY TYPE

Retrospective.

SUBJECTS

One hundred and fifty nominally healthy subjects from UK Biobank (100 subjects for method development and 50 subjects for validation). A separate 390 UK Biobank triples of subjects including type 2 diabetes mellitus (T2DM) subjects and matched nondiabetics.

FIELD STRENGTH/SEQUENCE

A 1.5 T, three-dimensional two-point Dixon sequence (for segmentation and volume assessment) and a two-dimensional axial multiecho gradient-recalled echo sequence.

ASSESSMENT

Pancreas segments were annotated by four raters on the validation cohort. Intrarater agreement and interrater agreement were reported using Dice overlap (Dice similarity coefficient [DSC]). A segmentation method based on template registration was developed and evaluated against annotations. Results on regional pancreatic fat assessment are also presented, by intersecting the three-dimensional parts segmentation with one available proton density fat fraction (PDFF) image.

STATISTICAL TEST

Wilcoxon signed rank test and Mann-Whitney U-test for comparisons. DSC and volume differences for evaluation. A P value < 0.05 was considered statistically significant.

RESULTS

Good intrarater (DSC mean, head: 0.982, body: 0.940, tail: 0.961) agreement and interrater (DSC mean, head: 0.968, body: 0.905, tail: 0.943) agreement were observed. No differences (DSC, head: P = 0.4358, body: P = 0.0992, tail: P = 0.1080) were observed between the manual annotations and our method's segmentations (DSC mean, head: 0.965, body: 0.893, tail: 0.934). Pancreatic body PDFF was different between T2DM and nondiabetics matched by body mass index.

DATA CONCLUSION

The developed segmentation's performance was no different from manual annotations. Application on type 2 diabetes subjects showed potential for assessing pancreatic disease heterogeneity.

LEVEL OF EVIDENCE

4 TECHNICAL EFFICACY STAGE: 3.

Deep learning-based segmentation of the thorax in mouse micro-CT scans

For image-guided small animal irradiations, the whole workflow of imaging, organ contouring, irradiation planning, and delivery is typically performed in a single session requiring continuous administration of anaesthetic agents. Automating contouring leads to a faster workflow, which limits exposure to anaesthesia and thereby, reducing its impact on experimental results and on animal wellbeing. Here, we trained the 2D and 3D U-Net architectures of no-new-Net (nnU-Net) for autocontouring of the thorax in mouse micro-CT images. We trained the models only on native CTs and evaluated their performance using an independent testing dataset (i.e., native CTs not included in the training and validation). Unlike previous studies, we also tested the model performance on an external dataset (i.e., contrast-enhanced CTs) to see how well they predict on CTs completely different from what they were trained on. We also assessed the interobserver variability using the generalized conformity index ([Formula: see text]) among three observers, providing a stronger human baseline for evaluating automated contours than previous studies. Lastly, we showed the benefit on the contouring time compared to manual contouring. The results show that 3D models of nnU-Net achieve superior segmentation accuracy and are more robust to unseen data than 2D models. For all target organs, the mean surface distance (MSD) and the Hausdorff distance (95p HD) of the best performing model for this task (nnU-Net 3d_fullres) are within 0.16 mm and 0.60 mm, respectively. These values are below the minimum required contouring accuracy of 1 mm for small animal irradiations, and improve significantly upon state-of-the-art 2D U-Net-based AIMOS method. Moreover, the conformity indices of the 3d_fullres model also compare favourably to the interobserver variability for all target organs, whereas the 2D models perform poorly in this regard. Importantly, the 3d_fullres model offers 98% reduction in contouring time.

Machine Learning for Auto-Segmentation in Radiotherapy Planning

Manual segmentation of target structures and organs at risk is a crucial step in the radiotherapy workflow. It has the disadvantages that it can require several hours of clinician time per patient and is prone to inter- and intra-observer variability. Automatic segmentation (auto-segmentation), using computer algorithms, seeks to address these issues. Advances in machine learning and computer vision have led to the development of methods for accurate and efficient auto-segmentation. This review surveys auto-segmentation techniques and applications in radiotherapy planning. It provides an overview of traditional approaches to auto-segmentation, including intensity analysis, shape modelling and atlas-based methods. The focus, though, is on uses of machine learning and deep learning, including convolutional neural networks. Finally, the future of machine-learning-driven auto-segmentation in clinical settings is considered, and the barriers that must be overcome for it to be widely accepted into routine practice are highlighted.

Clinical validation of an automatic atlas‐based segmentation tool for male pelvis CT images

Purpose

This retrospective work aims to evaluate the possible impact on intra‐ and inter‐observer variability, contouring time, and contour accuracy of introducing a pelvis computed tomography (CT) auto‐segmentation tool in radiotherapy planning workflow.

Methods

Tests were carried out on five structures (bladder, rectum, pelvic lymph‐nodes, and femoral heads) of six previously treated subjects, enrolling five radiation oncologists (ROs) to manually re‐contour and edit auto‐contours generated with a male pelvis CT atlas created with the commercial software MIM MAESTRO. The ROs first delineated manual contours (M). Then they modified the auto‐contours, producing automatic‐modified (AM) contours. The procedure was repeated to evaluate intra‐observer variability, producing M1, M2, AM1, and AM2 contour sets (each comprising 5 structures × 6 test patients × 5 ROs = 150 contours), for a total of 600 contours. Potential time savings was evaluated by comparing contouring and editing times. Structure contours were compared to a reference standard by means of Dice similarity coefficient (DSC) and mean distance to agreement (MDA), to assess intra‐ and inter‐observer variability. To exclude any automation bias, ROs evaluated both M and AM sets as “clinically acceptable” or “to be corrected” in a blind test.

Results

Comparing AM to M sets, a significant reduction of both inter‐observer variability (p < 0.001) and contouring time (‐45% whole pelvis, p < 0.001) was obtained. Intra‐observer variability reduction was significant only for bladder and femoral heads (p < 0.001). The statistical test showed no significant bias.

Conclusion

Our atlas‐based workflow proved to be effective for clinical practice as it can improve contour reproducibility and generate time savings. Based on these findings, institutions are encouraged to implement their auto‐segmentation method.

Generalizing deep learning brain segmentation for skull removal and intracranial measurements

Total intracranial volume (TICV) and posterior fossa volume (PFV) are essential covariates for brain volumetric analyses with structural magnetic resonance imaging (MRI). Detailed whole brain segmentation provides a non-invasive way to measure brain regions. Furthermore, increasing neuroimaging data are distributed in a skull-stripped manner for privacy protection. Therefore, generalizing deep learning brain segmentation for skull removal and intracranial measurements is an appealing task. However, data availability is challenging due to a limited set of manually traced atlases with whole brain and TICV/PFV labels. In this paper, we employ U-Net tiles to achieve automatic TICV estimation and whole brain segmentation simultaneously on brains w/and w/o the skull. To overcome the scarcity of manually traced whole brain volumes, a transfer learning method is introduced to estimate additional TICV and PFV labels during whole brain segmentation in T1-weighted MRI. Specifically, U-Net tiles are first pre-trained using large-scale BrainCOLOR atlases without TICV and PFV labels, which are created by multi-atlas segmentation. Then the pre-trained models are refined by training the additional TICV and PFV labels using limited BrainCOLOR atlases. We also extend our method to handle skull-stripped brain MR images. From the results, our method provides promising whole brain segmentation and volume estimation results for both brains w/and w/o skull in terms of mean Dice similarity coefficients and mean surface distance and absolute volume similarity. This method has been made available in open source (https://github.com/MASILab/SLANTbrainSeg_skullstripped).

Znet: Deep Learning Approach for 2D MRI Brain Tumor Segmentation

Background: Detection and segmentation of brain tumors using MR images are challenging and valuable tasks in the medical field. Early diagnosing and localizing of brain tumors can save lives and provide timely options for physicians to select efficient treatment plans. Deep learning approaches have attracted researchers in medical imaging due to their capacity, performance, and potential to assist in accurate diagnosis, prognosis, and medical treatment technologies. Methods and procedures: This paper presents a novel framework for segmenting 2D brain tumors in MR images using deep neural networks (DNN) and utilizing data augmentation strategies. The proposed approach (Znet) is based on the idea of skip-connection, encoder-decoder architectures, and data amplification to propagate the intrinsic affinities of a relatively smaller number of expert delineated tumors, e.g., hundreds of patients of the low-grade glioma (LGG), to many thousands of synthetic cases. Results: Our experimental results showed high values of the mean dice similarity coefficient (dice = 0.96 during model training and dice = 0.92 for the independent testing dataset). Other evaluation measures were also relatively high, e.g., pixel accuracy = 0.996, F1 score = 0.81, and Matthews Correlation Coefficient, MCC = 0.81. The results and visualization of the DNN-derived tumor masks in the testing dataset showcase the ZNet model’s capability to localize and auto-segment brain tumors in MR images. This approach can further be generalized to 3D brain volumes, other pathologies, and a wide range of image modalities. Conclusion: We can confirm the ability of deep learning methods and the proposed Znet framework to detect and segment tumors in MR images. Furthermore, pixel accuracy evaluation may not be a suitable evaluation measure for semantic segmentation in case of class imbalance in MR images segmentation. This is because the dominant class in ground truth images is the background. Therefore, a high value of pixel accuracy can be misleading in some computer vision applications. On the other hand, alternative evaluation metrics, such as dice and IoU (Intersection over Union), are more factual for semantic segmentation. Clinical impact: Artificial intelligence (AI) applications in medicine are advancing swiftly, however, there is a lack of deployed techniques in clinical practice. This research demonstrates a practical example of AI applications in medical imaging, which can be deployed as a tool for auto-segmentation of tumors in MR images.

Bayesian Spatial Binary Regression for Label Fusion in Structural Neuroimaging

Alzheimer's disease is a neurodegenerative condition that accelerates cognitive decline relative to normal aging. It is of critical scientific importance to gain a better understanding of early disease mechanisms in the brain to facilitate effective, targeted therapies. The volume of the hippocampus is often used in diagnosis and monitoring of the disease. Measuring this volume via neuroimaging is difficult since each hippocampus must either be manually identified or automatically delineated, a task referred to as segmentation. Automatic hippocampal segmentation often involves mapping a previously manually segmented image to a new brain image and propagating the labels to obtain an estimate of where each hippocampus is located in the new image. A more recent approach to this problem is to propagate labels from multiple manually segmented atlases and combine the results using a process known as label fusion. To date, most label fusion algorithms employ voting procedures with voting weights assigned directly or estimated via optimization. We propose using a fully Bayesian spatial regression model for label fusion that facilitates direct incorporation of covariate information while making accessible the entire posterior distribution. Our results suggest that incorporating tissue classification (e.g, gray matter) into the label fusion procedure can greatly improve segmentation when relatively homogeneous, healthy brains are used as atlases for diseased brains. The fully Bayesian approach also produces meaningful uncertainty measures about hippocampal volumes, information which can be leveraged to detect significant, scientifically meaningful differences between healthy and diseased populations, improving the potential for early detection and tracking of the disease.

Improving predictive CTV segmentation on CT and CBCT for cervical cancer by diffeomorphic registration of a prior

PURPOSE

Automatic cervix-uterus segmentation of the clinical target volume (CTV) on CT and cone beam CT (CBCT) scans is challenged by the limited visibility and the non-anatomical definition of certain border regions. We study potential performance gain of convolutional neural networks by regulating the segmentation predictions as diffeomorphic deformations of a segmentation prior.

METHODS

We introduce a 3D convolutional neural network (CNN) which segments the target scan by joint voxel-wise classification and the registration of a given prior. We compare this network to two other 3D baseline models: one treating segmentation as a classification problem (segmentation-only), the other as a registration problem (deformation-only). For reference and to highlight benefits of a 3D model, these models are also benchmarked against a 2D segmentation model. Network performances are reported for CT and CBCT segmentation of the cervix-uterus CTV. We train the networks on data of 84 patients. The prior is provided by the CTV segmentation of a planning CT. Repeat CT or CBCT scans constitute the target scans to be segmented.

RESULTS

All 3D models outperformed the 2D segmentation model. For CT segmentation, combining classification and registration in the proposed joint model proved beneficial, achieving a Dice score of 0.87 and a mean squared error (MSE) of the surface distance below 1.7 mm. No such synergy was observed for CBCT segmentation, for which the joint and the deformation-only model performed similarly, achieving a Dice score of about 0.80 and a MSE surface distance of 2.5 mm. However, the segmentation-only model performed notably worse in this low contrast regime. Visual inspection revealed that this performance drop translated into geometric inconsistencies between the prior and target segmentation. Such inconsistencies where not observed for the deformation-based models.

CONCLUSION

Constraining the solution space of admissible segmentation predictions to those reachable by a diffeomorphic deformation of the prior proved beneficial as it improved geometric consistency. Especially for CBCT, with its poor soft tissue contrast, this type of regularization becomes important as shown by quantitative and qualitative evaluation. This article is protected by copyright. All rights reserved.

Residual refinement for interactive skin lesion segmentation

BACKGROUND

Image segmentation is a difficult and classic problem. It has a wide range of applications, one of which is skin lesion segmentation. Numerous researchers have made great efforts to tackle the problem, yet there is still no universal method in various application domains.

RESULTS

We propose a novel approach that combines a deep convolutional neural network with a grabcut-like user interaction to tackle the interactive skin lesion segmentation problem. Slightly deviating from grabcut user interaction, our method uses boxes and clicks. In addition, contrary to existing interactive segmentation algorithms that combine the initial segmentation task with the following refinement task, we explicitly separate these tasks by designing individual sub-networks. One network is SBox-Net, and the other is Click-Net. SBox-Net is a full-fledged segmentation network that is built upon a pre-trained, state-of-the-art segmentation model, while Click-Net is a simple yet powerful network that combines feature maps extracted from SBox-Net and user clicks to residually refine the mistakes made by SBox-Net. Extensive experiments on two public datasets, PH2 and ISIC, confirm the effectiveness of our approach.

CONCLUSIONS

We present an interactive two-stage pipeline method for skin lesion segmentation, which was demonstrated to be effective in comprehensive experiments.

Multi-COVID-Net: Multi-objective optimized network for COVID-19 diagnosis from chest X-ray images

Coronavirus Disease 2019 (COVID-19) had already spread worldwide, and healthcare services have become limited in many countries. Efficient screening of hospitalized individuals is vital in the struggle toward COVID-19 through chest radiography, which is one of the important assessment strategies. This allows researchers to understand medical information in terms of chest X-ray (CXR) images and evaluate relevant irregularities, which may result in a fully automated identification of the disease. Due to the rapid growth of cases every day, a relatively small number of COVID-19 testing kits are readily accessible in health care facilities. Thus it is imperative to define a fully automated detection method as an instant alternate treatment possibility to limit the occurrence of COVID-19 among individuals. In this paper, a two-step Deep learning (DL) architecture has been proposed for COVID-19 diagnosis using CXR. The proposed DL architecture consists of two stages, “feature extraction and classification”. The “Multi-Objective Grasshopper Optimization Algorithm (MOGOA)” is presented to optimize the DL network layers; hence, these networks have named as “Multi-COVID-Net”. This model classifies the Non-COVID-19, COVID-19, and pneumonia patient images automatically. The Multi-COVID-Net has been tested by utilizing the publicly available datasets, and this model provides the best performance results than other state-of-the-art methods.

A Semi-Supervised Reduced-Space Method for Hyperspectral Imaging Segmentation

The development of the hyperspectral remote sensor technology allows the acquisition of images with a very detailed spectral information for each pixel. Because of this, hyperspectral images (HSI) potentially possess larger capabilities in solving many scientific and practical problems in agriculture, biomedical, ecological, geological, hydrological studies. However, their analysis requires developing specialized and fast algorithms for data processing, due the high dimensionality of the data. In this work, we propose a new semi-supervised method for multilabel segmentation of HSI that combines a suitable linear discriminant analysis, a similarity index to compare different spectra, and a random walk based model with a direct label assignment. The user-marked regions are used for the projection of the original high-dimensional feature space to a lower dimensional space, such that the class separation is maximized. This allows to retain in an automatic way the most informative features, lightening the successive computational burden. The part of the random walk is related to a combinatorial Dirichlet problem involving a weighted graph, where the nodes are the projected pixel of the original HSI, and the positive weights depend on the distances between these nodes. We then assign to each pixel of the original image a probability quantifying the likelihood that the pixel (node) belongs to some subregion. The computation of the spectral distance involves both the coordinates in a features space of a pixel and of its neighbors. The final segmentation process is therefore reduced to a suitable optimization problem coupling the probabilities from the random walker computation, and the similarity with respect the initially labeled pixels. We discuss the properties of the new method with experimental results carried on benchmark images.

SOMA: Subject-, object-, and modality-adapted precision atlas approach for automatic anatomy recognition and delineation in medical images

PURPOSE

In the multi-atlas segmentation (MAS) method, a large enough atlas set, which can cover the complete spectrum of the whole population pattern of the target object will benefit the segmentation quality. However, the difficulty in obtaining and generating such a large set of atlases and the computational burden required in the segmentation procedure make this approach impractical. In this paper, we propose a method called SOMA to select subject-, object-, and modality-adapted precision atlases for automatic anatomy recognition in medical images with pathology, following the idea that different regions of the target object in a novel image can be recognized by different atlases with regionally best similarity, so that effective atlases have no need to be globally similar to the target subject and also have no need to be overall similar to the target object.

METHODS

The SOMA method consists of three main components: atlas building, object recognition, and object delineation. Considering the computational complexity, we utilize an all-to-template strategy to align all images to the same image space belonging to the root image determined by the minimum spanning tree (MST) strategy among a subset of radiologically near-normal images. The object recognition process is composed of two stages: rough recognition and refined recognition. In rough recognition, subimage matching is conducted between the test image and each image of the whole atlas set, and only the atlas corresponding to the best-matched subimage contributes to the recognition map regionally. The frequency of best match for each atlas is recorded by a counter, and the atlases with the highest frequencies are selected as the precision atlases. In refined recognition, only the precision atlases are examined, and the subimage matching is conducted in a nonlocal manner of searching to further increase the accuracy of boundary matching. Delineation is based on a U-net-based deep learning network, where the original gray scale image together with the fuzzy map from refined recognition compose a two-channel input to the network, and the output is a segmentation map of the target object.

RESULTS

Experiments are conducted on computed tomography (CT) images with different qualities in two body regions - head and neck (H&N) and thorax, from 298 subjects with nine objects and 241 subjects with six objects, respectively. Most objects achieve a localization error within two voxels after refined recognition, with marked improvement in localization accuracy from rough to refined recognition of 0.6-3 mm in H&N and 0.8-4.9 mm in thorax, and also in delineation accuracy (Dice coefficient) from refined recognition to delineation of 0.01-0.11 in H&N and 0.01-0.18 in thorax.

CONCLUSIONS

The SOMA method shows high accuracy and robustness in anatomy recognition and delineation. The improvements from rough to refined recognition and further to delineation, as well as immunity of recognition accuracy to varying image and object qualities, demonstrate the core principles of SOMA where segmentation accuracy increases with precision atlases and gradually refined object matching.

Automatic Multi-Atlas Liver Segmentation and Couinaud Classification from CT Volumes

Primary Live Cancer (PLC) is the sixth most common cancer worldwide and its occurrence predominates in patients with chronic liver diseases and other risk factors like hepatitis B and C. Treatment of PLC and malignant liver tumors depend both in tumor characteristics and the functional status of the organ, thus must be individualized for each patient. Liver segmentation and classification according to Couinaud's classification is essential for computer-aided diagnosis and treatment planning, however, manual segmentation of the liver volume slice by slice can be a time-consuming and challenging task and it is highly dependent on the experience of the user. We propose an alternative automatic segmentation method that allows accuracy and time consumption amelioration. The procedure pursues a multi-atlas based classification for Couinaud segmentation. Our algorithm was implemented on 20 subjects from the IRCAD 3D data base in order to segment and classify the liver volume in its Couinaud segments, obtaining an average DICE coefficient of 0.94.Clinical Relevance- The final purpose of this work is to provide an automatic multi-atlas liver segmentation and Couinaud classification by means of CT image analysis.

Automatic Segmentation of Pelvic Cancers Using Deep Learning: State-of-the-Art Approaches and Challenges

The recent rise of deep learning (DL) and its promising capabilities in capturing non-explicit detail from large datasets have attracted substantial research attention in the field of medical image processing. DL provides grounds for technological development of computer-aided diagnosis and segmentation in radiology and radiation oncology. Amongst the anatomical locations where recent auto-segmentation algorithms have been employed, the pelvis remains one of the most challenging due to large intra- and inter-patient soft-tissue variabilities. This review provides a comprehensive, non-systematic and clinically-oriented overview of 74 DL-based segmentation studies, published between January 2016 and December 2020, for bladder, prostate, cervical and rectal cancers on computed tomography (CT) and magnetic resonance imaging (MRI), highlighting the key findings, challenges and limitations.