Learning to rank atlases for multiple-atlas segmentation

Integrated 3d flow-based multi-atlas brain structure segmentation

MRI brain structure segmentation plays an important role in neuroimaging studies. Existing methods either spend much CPU time, require considerable annotated data, or fail in segmenting volumes with large deformation. In this paper, we develop a novel multi-atlas-based algorithm for 3D MRI brain structure segmentation. It consists of three modules: registration, atlas selection and label fusion. Both registration and label fusion leverage an integrated flow based on grayscale and SIFT features. We introduce an effective and efficient strategy for atlas selection by employing the accompanying energy generated in the registration step. A 3D sequential belief propagation method and a 3D coarse-to-fine flow matching approach are developed in both registration and label fusion modules. The proposed method is evaluated on five public datasets. The results show that it has the best performance in almost all the settings compared to competitive methods such as ANTs, Elastix, Learning to Rank and Joint Label Fusion. Moreover, our registration method is more than 7 times as efficient as that of ANTs SyN, while our label transfer method is 18 times faster than Joint Label Fusion in CPU time. The results on the ADNI dataset demonstrate that our method is applicable to image pairs that require a significant transformation in registration. The performance on a composite dataset suggests that our method succeeds in a cross-modality manner. The results of this study show that the integrated 3D flow-based method is effective and efficient for brain structure segmentation. It also demonstrates the power of SIFT features, multi-atlas segmentation and classical machine learning algorithms for a medical image analysis task. The experimental results on public datasets show the proposed method's potential for general applicability in various brain structures and settings.

SOMA: Subject-, object-, and modality-adapted precision atlas approach for automatic anatomy recognition and delineation in medical images

PURPOSE

In the multi-atlas segmentation (MAS) method, a large enough atlas set, which can cover the complete spectrum of the whole population pattern of the target object will benefit the segmentation quality. However, the difficulty in obtaining and generating such a large set of atlases and the computational burden required in the segmentation procedure make this approach impractical. In this paper, we propose a method called SOMA to select subject-, object-, and modality-adapted precision atlases for automatic anatomy recognition in medical images with pathology, following the idea that different regions of the target object in a novel image can be recognized by different atlases with regionally best similarity, so that effective atlases have no need to be globally similar to the target subject and also have no need to be overall similar to the target object.

METHODS

The SOMA method consists of three main components: atlas building, object recognition, and object delineation. Considering the computational complexity, we utilize an all-to-template strategy to align all images to the same image space belonging to the root image determined by the minimum spanning tree (MST) strategy among a subset of radiologically near-normal images. The object recognition process is composed of two stages: rough recognition and refined recognition. In rough recognition, subimage matching is conducted between the test image and each image of the whole atlas set, and only the atlas corresponding to the best-matched subimage contributes to the recognition map regionally. The frequency of best match for each atlas is recorded by a counter, and the atlases with the highest frequencies are selected as the precision atlases. In refined recognition, only the precision atlases are examined, and the subimage matching is conducted in a nonlocal manner of searching to further increase the accuracy of boundary matching. Delineation is based on a U-net-based deep learning network, where the original gray scale image together with the fuzzy map from refined recognition compose a two-channel input to the network, and the output is a segmentation map of the target object.

RESULTS

Experiments are conducted on computed tomography (CT) images with different qualities in two body regions - head and neck (H&N) and thorax, from 298 subjects with nine objects and 241 subjects with six objects, respectively. Most objects achieve a localization error within two voxels after refined recognition, with marked improvement in localization accuracy from rough to refined recognition of 0.6-3 mm in H&N and 0.8-4.9 mm in thorax, and also in delineation accuracy (Dice coefficient) from refined recognition to delineation of 0.01-0.11 in H&N and 0.01-0.18 in thorax.

CONCLUSIONS

The SOMA method shows high accuracy and robustness in anatomy recognition and delineation. The improvements from rough to refined recognition and further to delineation, as well as immunity of recognition accuracy to varying image and object qualities, demonstrate the core principles of SOMA where segmentation accuracy increases with precision atlases and gradually refined object matching.

A robust discriminative multi-atlas label fusion method for hippocampus segmentation from MR image

Accurate and automatic segmentation of the hippocampus plays a vital role in the diagnosis and treatment of nervous system diseases. However, due to the anatomical variability of different subjects, the registered atlas images are not always perfectly aligned with the target image. This makes the segmentation of the hippocampus still face great challenges. In this paper, we propose a robust discriminative label fusion method under the multi-atlas framework. It is a patch embedding label fusion method based on conditional random field (CRF) model that integrates the metric learning and the graph cuts by an integrated formulation. Unlike most current label fusion methods with fixed (non-learning) distance metrics, a novel distance metric learning is presented to enhance discriminative observation and embed it into the unary potential function. In particular, Bayesian inference is utilized to extend a classic distance metric learning, in which large margin constraints are instead of pairwise constraints to obtain a more robust distance metric. And the pairwise homogeneity is fully considered in the spatial prior term based on classification labels and voxel intensity. The resulting integrated formulation is globally minimized by the efficient graph cuts algorithm. Further, sparse patch based method is utilized to polish the obtained segmentation results in label space. The proposed method is evaluated on IABA dataset and ADNI dataset for hippocampus segmentation. The Dice scores achieved by our method are 87.2%, 87.8%, 88.2% and 88.9% on left and right hippocampus on both two datasets, while the best Dice scores obtained by other methods are 86.0%, 86.9%, 86.8% and 88.0% on IABA dataset and ADNI dataset respectively. Experiments show that our approach achieves higher accuracy than state-of-the-art methods. We hope the proposed model can be transferred to combine with other promising distance measurement algorithms.

Anatomy Recognition in CT Images of Head & Neck Region via Precision Atlases

Multi-atlas segmentation methods will benefit from atlases covering the complete spectrum of population patterns, while the difficulties in generating such large enough datasets and the computation burden required in the segmentation procedure reduce its practicality in clinical application. In this work, we start from a viewpoint that different parts of the target object can be recognized by different atlases and propose a precision atlas selection strategy. By comparing regional similarity between target image and atlases, precision atlases are ranked and selected by the frequency of regional best match, which have no need to be globally similar to the target subject at either image-level or object-level, largely increasing the implicit patterns contained in the atlas set. In the proposed anatomy recognition method, atlas building is first achieved by all-to-template registration, where the minimum spanning tree (MST) strategy is used to select a registration template from a subset of radiologically near-normal images. Then, a two-stage recognition process is conducted: in rough recognition, sub-image level similarity is calculated between the test image and each image of the whole atlas set, and only the atlas with the highest similarity contributes to the recognition map regionally; in refined recognition, the atlases with the highest frequencies of best match are selected as the precision atlases and are utilized to further increase the accuracy of boundary matching. The proposed method is demonstrated on 298 computed tomography (CT) images and 9 organs in the Head & Neck (H&N) body region. Experimental results illustrate that our method is effective for organs with different segmentation challenge and samples with different image quality, where remarkable improvement in boundary interpretation is made by refined recognition and most objects achieve a localization error within 2 voxels.

FCN Based Label Correction for Multi-Atlas Guided Organ Segmentation

Segmentation of medical images using multiple atlases has recently gained immense attention due to their augmented robustness against variabilities across different subjects. These atlas-based methods typically comprise of three steps: atlas selection, image registration, and finally label fusion. Image registration is one of the core steps in this process, accuracy of which directly affects the final labeling performance. However, due to inter-subject anatomical variations, registration errors are inevitable. The aim of this paper is to develop a deep learning-based confidence estimation method to alleviate the potential effects of registration errors. We first propose a fully convolutional network (FCN) with residual connections to learn the relationship between the image patch pair (i.e., patches from the target subject and the atlas) and the related label confidence patch. With the obtained label confidence patch, we can identify the potential errors in the warped atlas labels and correct them. Then, we use two label fusion methods to fuse the corrected atlas labels. The proposed methods are validated on a publicly available dataset for hippocampus segmentation. Experimental results demonstrate that our proposed methods outperform the state-of-the-art segmentation methods.

Nonlinear interaction between APOE ε4 allele load and age in the hippocampal surface of cognitively intact individuals

The ε4 allele of the gene Apolipoprotein E is the major genetic risk factor for Alzheimer's Disease. APOE ε4 has been associated with changes in brain structure in cognitively impaired and unimpaired subjects, including atrophy of the hippocampus, which is one of the brain structures that is early affected by AD. In this work we analyzed the impact of APOE ε4 gene dose and its association with age, on hippocampal shape assessed with multivariate surface analysis, in a ε4‐enriched cohort of n = 479 cognitively healthy individuals. Furthermore, we sought to replicate our findings on an independent dataset of n = 969 individuals covering the entire AD spectrum. We segmented the hippocampus of the subjects with a multi‐atlas‐based approach, obtaining high‐dimensional meshes that can be analyzed in a multivariate way. We analyzed the effects of different factors including APOE, sex, and age (in both cohorts) as well as clinical diagnosis on the local 3D hippocampal surface changes. We found specific regions on the hippocampal surface where the effect is modulated by significant APOE ε4 linear and quadratic interactions with age. We compared between APOE and diagnosis effects from both cohorts, finding similarities between APOE ε4 and AD effects on specific regions, and suggesting that age may modulate the effect of APOE ε4 and AD in a similar way.

An Evaluation of Atlas Selection Methods for Atlas-Based Automatic Segmentation in Radiotherapy Treatment Planning

Atlas-based automatic segmentation is used in radiotherapy planning to accelerate the delineation of organs at risk (OARs). Atlas selection has been proposed as a way to improve the accuracy and execution time of segmentation, assuming that, the more similar the atlas is to the patient, the better the results will be. This paper presents an analysis of atlas selection methods in the context of radiotherapy treatment planning. For a range of commonly contoured OARs, a thorough comparison of a large class of typical atlas selection methods has been performed. For this evaluation, clinically contoured CT images of the head and neck ( N=316 ) and thorax ( N=280 ) were used. The state-of-the-art intensity and deformation similarity-based atlas selection methods were found to compare poorly to perfect atlas selection. Counter-intuitively, atlas selection methods based on a fixed set of representative atlases outperformed atlas selection methods based on the patient image. This study suggests that atlas-based segmentation with currently available selection methods compares poorly to the potential best performance, hampering the clinical utility of atlas-based segmentation. Effective atlas selection remains an open challenge in atlas-based segmentation for radiotherapy planning.

Dynamic multiatlas selection-based consensus segmentation of head and neck structures from CT images

PURPOSE

Manual delineation of head and neck (H&N) organ-at-risk (OAR) structures for radiation therapy planning is time consuming and highly variable. Therefore, we developed a dynamic multiatlas selection-based approach for fast and reproducible segmentation.

METHODS

Our approach dynamically selects and weights the appropriate number of atlases for weighted label fusion and generates segmentations and consensus maps indicating voxel-wise agreement between different atlases. Atlases were selected for a target as those exceeding an alignment weight called dynamic atlas attention index. Alignment weights were computed at the image level and called global weighted voting (GWV) or at the structure level and called structure weighted voting (SWV) by using a normalized metric computed as the sum of squared distances of computed tomography (CT)-radiodensity and modality-independent neighborhood descriptors (extracting edge information). Performance comparisons were performed using 77 H&N CT images from an internal Memorial Sloan-Kettering Cancer Center dataset (N = 45) and an external dataset (N = 32) using Dice similarity coefficient (DSC), Hausdorff distance (HD), 95th percentile of HD, median of maximum surface distance, and volume ratio error against expert delineation. Pairwise DSC accuracy comparisons of proposed (GWV, SWV) vs single best atlas (BA) or majority voting (MV) methods were performed using Wilcoxon rank-sum tests.

RESULTS

Both SWV and GWV methods produced significantly better segmentation accuracy than BA (P < 0.001) and MV (P < 0.001) for all OARs within both datasets. SWV generated the most accurate segmentations with DSC of: 0.88 for oral cavity, 0.85 for mandible, 0.84 for cord, 0.76 for brainstem and parotids, 0.71 for larynx, and 0.60 for submandibular glands. SWV's accuracy exceeded GWV's for submandibular glands (DSC = 0.60 vs 0.52, P = 0.019).

CONCLUSIONS

The contributed SWV and GWV methods generated more accurate automated segmentations than the other two multiatlas-based segmentation techniques. The consensus maps could be combined with segmentations to visualize voxel-wise consensus between atlases within OARs during manual review.

VoteNet: A Deep Learning Label Fusion Method for Multi-Atlas Segmentation

Deep learning (DL) approaches are state-of-the-art for many medical image segmentation tasks. They offer a number of advantages: they can be trained for specific tasks, computations are fast at test time, and segmentation quality is typically high. In contrast, previously popular multi-atlas segmentation (MAS) methods are relatively slow (as they rely on costly registrations) and even though sophisticated label fusion strategies have been proposed, DL approaches generally outperform MAS. In this work, we propose a DL-based label fusion strategy (VoteNet) which locally selects a set of reliable atlases whose labels are then fused via plurality voting. Experiments on 3D brain MRI data show that by selecting a good initial atlas set MAS with VoteNet significantly outperforms a number of other label fusion strategies as well as a direct DL segmentation approach. We also provide an experimental analysis of the upper performance bound achievable by our method. While unlikely achievable in practice, this bound suggests room for further performance improvements. Lastly, to address the runtime disadvantage of standard MAS, all our results make use of a fast DL registration approach.

How many models/atlases are needed as priors for capturing anatomic population variations?

Many medical image processing and analysis operations can benefit a great deal from prior information encoded in the form of models/atlases to capture variations over a population in form, shape, anatomic layout, and image appearance of objects. However, two fundamental questions have not been addressed in the literature: "How many models/atlases are needed for optimally encoding prior information to address the differing body habitus factor in that population?" and "Images of how many subjects in the given population are needed to optimally harness prior information?" We propose a method to seek answers to these questions. We assume that there is a well-defined body region of interest and a subject population under consideration, and that we are given a set of representative images of the body region for the population. After images are trimmed to the exact body region, a hierarchical agglomerative clustering algorithm partitions the set of images into a specified number of groups by using pairwise image (dis)similarity as a cost function. Optionally the images may be pre-registered among themselves prior to this partitioning operation. We define a measure called Residual Dissimilarity (RD) to determine the goodness of each partition. We then ascertain how RD varies as a function of the number of elements in the partition for finding the optimum number(s) of groups. Breakpoints in this function are taken as the recommended number of groups/models/atlases. Our results from analysis of sizeable CT data sets of adult patients from two body regions - thorax (346) and head and neck (298) - can be summarized as follows. (1) A minimum of 5 to 8 groups (or models/atlases) seems essential to properly capture information about differing anatomic forms and body habitus. (2) A minimum of 150 images from different subjects in a population seems essential to cover the anatomical variations for a given body region. (3) In grouping, body habitus variations seem to override differences due to other factors such as gender, with/without contrast enhancement in image acquisition, and presence of moderate pathology. This method may be helpful for constructing high quality models/atlases from a sufficiently large population of images and in optimally selecting the training image sets needed in deep learning strategies.

An atlas-based method to predict three-dimensional dose distributions for cancer patients who receive radiotherapy

Due to the complexity of advanced radiotherapy techniques, treatment planning process is usually time consuming and plan quality can vary considerably among planners and institutions. It is also impractical to generate all possible treatment plans based on available radiotherapy techniques and select the best option for a specific patient. Automatic dose prediction will be very helpful in these situations, while there were a few studies of three-dimensional (3D) dose prediction for patients who received radiotherapy. The purpose of this work was to develop a novel atlas-based method to predict 3D dose prediction and to evaluate its performance. Previously treated nineteen left-sided post-mastectomy breast cancer patients and sixteen prostate cancer patients were included in this study. One patient was arbitrarily chosen as the reference for each type of cancer and all the remaining patients' computed tomography (CT) images and contours were aligned to it using deformable image registration (DIR). Deformable vector field (DVF) for each patient i (DVF _i-ref) was used to deform the original 3D dose matrix of that patient. CT scan of a test patient was also registered with the same reference patient using DIR and both direct DVF (DVF_test-ref) and inverse DVF ([Formula: see text]) were derived. Similarity of atlas patients to the test patient was determined based on the similarity of DVF_test-ref to atlas DVFs (DVF _i-ref) and appropriate weighting factors were calculated. Patients' doses in the atlas were deformed again using [Formula: see text] to transform them from the reference patient's coordinates to the test patient's coordinates and the final 3D dose distribution for the test patient was predicted by summing the weighted individual 3D dose distributions. Performance of our method was evaluated and the results revealed that the proposed method was able to predict the 3D dose distributions accurately. The mean dose difference between clinical and predicted 3D dose distributions were 0.9  ±  1.1 Gy and 1.9  ±  1.2 Gy for breast and prostate plans. The proposed dose prediction method can be used to improve planning quality and facilitate plan comparisons.

Can Atlas-Based Auto-Segmentation Ever Be Perfect? Insights From Extreme Value Theory

Atlas-based segmentation is used in radiotherapy planning to accelerate the delineation of organs at risk (OARs). Atlas selection has been proposed to improve the performance of segmentation, assuming that the more similar the atlas is to the patient, the better the result. It follows that the larger the database of atlases from which to select, the better the results should be. This paper seeks to estimate a clinically achievable expected performance under this assumption. Assuming a perfect atlas selection, an extreme value theory has been applied to estimate the accuracy of single-atlas and multi-atlas segmentation given a large database of atlases. For this purpose, clinical contours of most common OARs on computed tomography of the head and neck ( N=316 ) and thoracic ( N=280 ) cases were used. This paper found that while for most organs, perfect segmentation cannot be reasonably expected, auto-contouring performance of a level corresponding to clinical quality could be consistently expected given a database of 5000 atlases under the assumption of perfect atlas selection.

Multi-Atlas Segmentation of MR Tumor Brain Images Using Low-Rank Based Image Recovery

We introduce a new multi-atlas segmentation (MAS) framework for MR tumor brain images. The basic idea of MAS is to register and fuse label information from multiple normal brain atlases to a new brain image for segmentation. Many MAS methods have been proposed with success. However, most of them are developed for normal brain images, and tumor brain images usually pose a great challenge for them. This is because tumors cause difficulties in registration of normal brain atlases to the tumor brain image. To address this challenge, in the first step of our MAS framework, a new low-rank method is used to get the recovered image of normal-looking brain from the MR tumor brain image based on the information of normal brain atlases. Different from conventional low-rank methods that produce the recovered image with distorted normal brain regions, our low-rank method harnesses a spatial constraint to get the recovered image with preserved normal brain regions. Then in the second step, normal brain atlases can be registered to the recovered image without influence from tumors. These two steps are iteratively proceeded until convergence, for obtaining the final segmentation of the tumor brain image. During the iteration, both the recovered image and the registration of normal brain atlases to the recovered image are gradually refined. We have compared our proposed method with state-of-the-art methods by using both synthetic and real MR tumor brain images. Experimental results show that our proposed method can get effectively recovered images and also improves segmentation accuracy.

Case-control comparison brain lesion segmentation for early infarct detection

Computed Tomography (CT) images are widely used for the identification of abnormal brain tissues following infarct and hemorrhage of a stroke. The treatment of this medical condition mainly depends on doctors' experience. While manual lesion delineation by medical doctors is currently considered as the standard approach, it is time-consuming and dependent on each doctor's expertise and experience. In this study, a case-control comparison brain lesion segmentation (CCBLS) method is proposed to segment the region pertaining to brain injury by comparing the voxel intensity of CT images between control subjects and stroke patients. The method is able to segment the brain lesion from the stacked CT images automatically without prior knowledge of the location or the presence of the lesion. The aim is to reduce medical doctors' burden and assist them in making an accurate diagnosis. A case study with 300 sets of CT images from control subjects and stroke patients is conducted. Comparing with other existing methods, the outcome ascertains the effectiveness of the proposed method in detecting brain infarct of stroke patients.

Atlas pre-selection strategies to enhance the efficiency and accuracy of multi-atlas brain segmentation tools

Multi-atlas brain segmentation of human brain MR images allows quantification research in structural neuroimaging. To achieve high accuracy and computational efficiency of segmentation relies on a custom subset of atlases for each target subject. However, the criterion for atlas pre-selection remains an open question. In this study, two atlas pre-selection approaches based on location-based feature matching were proposed and compared to random and mutual information-based methods using a database of 47 atlases. A varying number of atlases ranked top with hierarchical structural granularity were compared using Dice overlap. The results indicated that the proposed 4L approach consistently led to the highest level of accuracy at a given number of employed atlases in both adult and geriatric populations. In addition, the proposed two methods (4L and LV) can reduce 20 times computational time compared with the stereotypical mutual information-based method. Our pre-selection strategy would provide better segmentation performance in terms of both accuracy and efficiency. The proposed atlas pre-selection will be further implemented into our online automatic brain image segmentation system (www.mricloud.org).

Voxel Deconvolutional Networks for 3D Brain Image Labeling

Deep learning methods have shown great success in pixel-wise prediction tasks. One of the most popular methods employs an encoder-decoder network in which deconvolutional layers are used for up-sampling feature maps. However, a key limitation of the deconvolutional layer is that it suffers from the checkerboard artifact problem, which harms the prediction accuracy. This is caused by the independency among adjacent pixels on the output feature maps. Previous work only solved the checkerboard artifact issue of deconvolutional layers in the 2D space. Since the number of intermediate feature maps needed to generate a deconvolutional layer grows exponentially with dimensionality, it is more challenging to solve this issue in higher dimensions. In this work, we propose the voxel deconvolutional layer (VoxelDCL) to solve the checkerboard artifact problem of deconvolutional layers in 3D space. We also provide an efficient approach to implement VoxelDCL. To demonstrate the effectiveness of VoxelDCL, we build four variations of voxel deconvolutional networks (VoxelDCN) based on the U-Net architecture with VoxelDCL. We apply our networks to address volumetric brain images labeling tasks using the ADNI and LONI LPBA40 datasets. The experimental results show that the proposed iVoxelDCNa achieves improved performance in all experiments. It reaches 83.34% in terms of dice ratio on the ADNI dataset and 79.12% on the LONI LPBA40 dataset, which increases 1.39% and 2.21% respectively compared with the baseline. In addition, all the variations of VoxelDCN we proposed outperform the baseline methods on the above datasets, which demonstrates the effectiveness of our methods.

Multi atlas based segmentation: should we prefer the best atlas group over the group of best atlases?

Multi atlas based segmentation (MABS) uses a database of atlas images, and an atlas selection process is used to choose an atlas subset for registration and voting. In the current state of the art, atlases are chosen according to a similarity criterion between the target subject and each atlas in the database. In this paper, we propose a new concept for atlas selection that relies on selecting the best performing group of atlases rather than the group of highest scoring individual atlases. Experiments were performed using CT images of 50 patients, with contours of brainstem and parotid glands. The dataset was randomly split into two groups: 20 volumes were used as an atlas database and 30 served as target subjects for testing. Classic oracle selection, where atlases are chosen by the highest dice similarity coefficient (DSC) with the target, was performed. This was compared to oracle group selection, where all the combinations of atlas subgroups were considered and scored by computing DSC with the target subject. Subsequently, convolutional neural networks were designed to predict the best group of atlases. The results were also compared with the selection strategy based on normalized mutual information (NMI). Oracle group was proven to be significantly better than classic oracle selection (p  <  10^-5). Atlas group selection led to a median  ±  interquartile DSC of 0.740  ±  0.084, 0.718  ±  0.086 and 0.670  ±  0.097 for brainstem and left/right parotid glands respectively, outperforming NMI selection 0.676  ±  0.113, 0.632  ±  0.104 and 0.606  ±  0.118 (p  <  0.001) as well as classic oracle selection. The implemented methodology is a proof of principle that selecting the atlases by considering the performance of the entire group of atlases instead of each single atlas leads to higher segmentation accuracy, being even better then current oracle strategy. This finding opens a new discussion about the most appropriate atlas selection criterion for MABS.

Fast anatomy segmentation by combining coarse scale multi-atlas label fusion with fine scale corrective learning

Deformable registration based multi-atlas segmentation has been successfully applied in a broad range of anatomy segmentation applications. However, the excellent performance comes with a high computational burden due to the requirement for deformable image registration and voxel-wise label fusion. To address this problem, we investigate the role of corrective learning (Wang et al., 2011) in speeding up multi-atlas segmentation. We propose to combine multi-atlas segmentation with corrective learning in a multi-scale analysis fashion for faster speeds. First, multi-atlas segmentation is applied in a low spatial resolution. After resampling the segmentation result back to the native image space, learning-based error correction is applied to correct systematic errors due to performing multi-atlas segmentation in a low spatial resolution. In cardiac CT and brain MR segmentation experiments, we show that applying multi-atlas segmentation in a coarse scale followed by learning-based error correction in the native space can substantially reduce the overall computational cost, with only modest or no sacrificing segmentation accuracy.

Interleaved 3D-CNNs for joint segmentation of small-volume structures in head and neck CT images

PURPOSE

Accurate 3D image segmentation is a crucial step in radiation therapy planning of head and neck tumors. These segmentation results are currently obtained by manual outlining of tissues, which is a tedious and time-consuming procedure. Automatic segmentation provides an alternative solution, which, however, is often difficult for small tissues (i.e., chiasm and optic nerves in head and neck CT images) because of their small volumes and highly diverse appearance/shape information. In this work, we propose to interleave multiple 3D Convolutional Neural Networks (3D-CNNs) to attain automatic segmentation of small tissues in head and neck CT images.

METHOD

A 3D-CNN was designed to segment each structure of interest. To make full use of the image appearance information, multiscale patches are extracted to describe the center voxel under consideration and then input to the CNN architecture. Next, as neighboring tissues are often highly related in the physiological and anatomical perspectives, we interleave the CNNs designated for the individual tissues. In this way, the tentative segmentation result of a specific tissue can contribute to refine the segmentations of other neighboring tissues. Finally, as more CNNs are interleaved and cascaded, a complex network of CNNs can be derived, such that all tissues can be jointly segmented and iteratively refined.

RESULT

Our method was validated on a set of 48 CT images, obtained from the Medical Image Computing and Computer Assisted Intervention (MICCAI) Challenge 2015. The Dice coefficient (DC) and the 95% Hausdorff Distance (95HD) are computed to measure the accuracy of the segmentation results. The proposed method achieves higher segmentation accuracy (with the average DC: 0.58 ± 0.17 for optic chiasm, and 0.71 ± 0.08 for optic nerve; 95HD: 2.81 ± 1.56 mm for optic chiasm, and 2.23 ± 0.90 mm for optic nerve) than the MICCAI challenge winner (with the average DC: 0.38 for optic chiasm, and 0.68 for optic nerve; 95HD: 3.48 for optic chiasm, and 2.48 for optic nerve).

CONCLUSION

An accurate and automatic segmentation method has been proposed for small tissues in head and neck CT images, which is important for the planning of radiotherapy.

Learning non-linear patch embeddings with neural networks for label fusion

In brain structural segmentation, multi-atlas strategies are increasingly being used over single-atlas strategies because of their ability to fit a wider anatomical variability. Patch-based label fusion (PBLF) is a type of such multi-atlas approaches that labels each target point as a weighted combination of neighboring atlas labels, where atlas points with higher local similarity to the target contribute more strongly to label fusion. PBLF can be potentially improved by increasing the discriminative capabilities of the local image similarity measurements. We propose a framework to compute patch embeddings using neural networks so as to increase discriminative abilities of similarity-based weighted voting in PBLF. As particular cases, our framework includes embeddings with different complexities, namely, a simple scaling, an affine transformation, and non-linear transformations. We compare our method with state-of-the-art alternatives in whole hippocampus and hippocampal subfields segmentation experiments using publicly available datasets. Results show that even the simplest versions of our method outperform standard PBLF, thus evidencing the benefits of discriminative learning. More complex transformation models tended to achieve better results than simpler ones, obtaining a considerable increase in average Dice score compared to standard PBLF.

Computational and mathematical methods in brain atlasing

Brain atlases have a wide range of use from education to research to clinical applications. Mathematical methods as well as computational methods and tools play a major role in the process of brain atlas building and developing atlas-based applications. Computational methods and tools cover three areas: dedicated editors for brain model creation, brain navigators supporting multiple platforms, and atlas-assisted specific applications. Mathematical methods in atlas building and developing atlas-aided applications deal with problems in image segmentation, geometric body modelling, physical modelling, atlas-to-scan registration, visualisation, interaction and virtual reality. Here I overview computational and mathematical methods in atlas building and developing atlas-assisted applications, and share my contribution to and experience in this field.

Discriminative confidence estimation for probabilistic multi-atlas label fusion

Quantitative neuroimaging analyses often rely on the accurate segmentation of anatomical brain structures. In contrast to manual segmentation, automatic methods offer reproducible outputs and provide scalability to study large databases. Among existing approaches, multi-atlas segmentation has recently shown to yield state-of-the-art performance in automatic segmentation of brain images. It consists in propagating the labelmaps from a set of atlases to the anatomy of a target image using image registration, and then fusing these multiple warped labelmaps into a consensus segmentation on the target image. Accurately estimating the contribution of each atlas labelmap to the final segmentation is a critical step for the success of multi-atlas segmentation. Common approaches to label fusion either rely on local patch similarity, probabilistic statistical frameworks or a combination of both. In this work, we propose a probabilistic label fusion framework based on atlas label confidences computed at each voxel of the structure of interest. Maximum likelihood atlas confidences are estimated using a supervised approach, explicitly modeling the relationship between local image appearances and segmentation errors produced by each of the atlases. We evaluate different spatial pooling strategies for modeling local segmentation errors. We also present a novel type of label-dependent appearance features based on atlas labelmaps that are used during confidence estimation to increase the accuracy of our label fusion. Our approach is evaluated on the segmentation of seven subcortical brain structures from the MICCAI 2013 SATA Challenge dataset and the hippocampi from the ADNI dataset. Overall, our results indicate that the proposed label fusion framework achieves superior performance to state-of-the-art approaches in the majority of the evaluated brain structures and shows more robustness to registration errors.

Nonlocal atlas-guided multi-channel forest learning for human brain labeling

PURPOSE

It is important for many quantitative brain studies to label meaningful anatomical regions in MR brain images. However, due to high complexity of brain structures and ambiguous boundaries between different anatomical regions, the anatomical labeling of MR brain images is still quite a challenging task. In many existing label fusion methods, appearance information is widely used. However, since local anatomy in the human brain is often complex, the appearance information alone is limited in characterizing each image point, especially for identifying the same anatomical structure across different subjects. Recent progress in computer vision suggests that the context features can be very useful in identifying an object from a complex scene. In light of this, the authors propose a novel learning-based label fusion method by using both low-level appearance features (computed from the target image) and high-level context features (computed from warped atlases or tentative labeling maps of the target image).

METHODS

In particular, the authors employ a multi-channel random forest to learn the nonlinear relationship between these hybrid features and target labels (i.e., corresponding to certain anatomical structures). Specifically, at each of the iterations, the random forest will output tentative labeling maps of the target image, from which the authors compute spatial label context features and then use in combination with original appearance features of the target image to refine the labeling. Moreover, to accommodate the high inter-subject variations, the authors further extend their learning-based label fusion to a multi-atlas scenario, i.e., they train a random forest for each atlas and then obtain the final labeling result according to the consensus of results from all atlases.

RESULTS

The authors have comprehensively evaluated their method on both public LONI_LBPA40 and IXI datasets. To quantitatively evaluate the labeling accuracy, the authors use the dice similarity coefficient to measure the overlap degree. Their method achieves average overlaps of 82.56% on 54 regions of interest (ROIs) and 79.78% on 80 ROIs, respectively, which significantly outperform the baseline method (random forests), with the average overlaps of 72.48% on 54 ROIs and 72.09% on 80 ROIs, respectively.

CONCLUSIONS

The proposed methods have achieved the highest labeling accuracy, compared to several state-of-the-art methods in the literature.

Learning image based surrogate relevance criterion for atlas selection in segmentation

Picking geometrically relevant atlases from the whole training set is crucial to multi-atlas based image segmentation, especially with extensive data of heterogeneous quality in the Big Data era. Unfortunately, there is very limited understanding of how currently used image similarity criteria reveal geometric relevance, let alone the optimization of them. This paper aims to develop a good image based surrogate relevance criterion to best reflect the underlying inaccessible geometric relevance in a learning context. We cast this surrogate learning problem into an optimization framework, by encouraging the image based surrogate to behave consistently with geometric relevance during training. In particular, we desire a criterion to be small for image pairs with similar geometry and large for those with significantly different segmentation geometry. Validation experiments on corpus callosum segmentation demonstrate the improved quality of the learned surrogate compared to benchmark surrogate candidates.

Contextual Atlas Regression Forests: Multiple-Atlas-Based Automated Dose Prediction in Radiation Therapy

Radiation therapy is an integral part of cancer treatment, but to date it remains highly manual. Plans are created through optimization of dose volume objectives that specify intent to minimize, maximize, or achieve a prescribed dose level to clinical targets and organs. Optimization is NP-hard, requiring highly iterative and manual initialization procedures. We present a proof-of-concept for a method to automatically infer the radiation dose directly from the patient's treatment planning image based on a database of previous patients with corresponding clinical treatment plans. Our method uses regression forests augmented with density estimation over the most informative features to learn an automatic atlas-selection metric that is tailored to dose prediction. We validate our approach on 276 patients from 3 clinical treatment plan sites (whole breast, breast cavity, and prostate), with an overall dose prediction accuracies of 78.68%, 64.76%, 86.83% under the Gamma metric.

MUSE: MUlti-atlas region Segmentation utilizing Ensembles of registration algorithms and parameters, and locally optimal atlas selection

Atlas-based automated anatomical labeling is a fundamental tool in medical image segmentation, as it defines regions of interest for subsequent analysis of structural and functional image data. The extensive investigation of multi-atlas warping and fusion techniques over the past 5 or more years has clearly demonstrated the advantages of consensus-based segmentation. However, the common approach is to use multiple atlases with a single registration method and parameter set, which is not necessarily optimal for every individual scan, anatomical region, and problem/data-type. Different registration criteria and parameter sets yield different solutions, each providing complementary information. Herein, we present a consensus labeling framework that generates a broad ensemble of labeled atlases in target image space via the use of several warping algorithms, regularization parameters, and atlases. The label fusion integrates two complementary sources of information: a local similarity ranking to select locally optimal atlases and a boundary modulation term to refine the segmentation consistently with the target image's intensity profile. The ensemble approach consistently outperforms segmentations using individual warping methods alone, achieving high accuracy on several benchmark datasets. The MUSE methodology has been used for processing thousands of scans from various datasets, producing robust and consistent results. MUSE is publicly available both as a downloadable software package, and as an application that can be run on the CBICA Image Processing Portal (https://ipp.cbica.upenn.edu), a web based platform for remote processing of medical images.

A Multi-Atlas Labeling Approach for Identifying Subject-Specific Functional Regions of Interest

The functional region of interest (fROI) approach has increasingly become a favored methodology in functional magnetic resonance imaging (fMRI) because it can circumvent inter-subject anatomical and functional variability, and thus increase the sensitivity and functional resolution of fMRI analyses. The standard fROI method requires human experts to meticulously examine and identify subject-specific fROIs within activation clusters. This process is time-consuming and heavily dependent on experts’ knowledge. Several algorithmic approaches have been proposed for identifying subject-specific fROIs; however, these approaches cannot easily incorporate prior knowledge of inter-subject variability. In the present study, we improved the multi-atlas labeling approach for defining subject-specific fROIs. In particular, we used a classifier-based atlas-encoding scheme and an atlas selection procedure to account for the large spatial variability across subjects. Using a functional atlas database for face recognition, we showed that with these two features, our approach efficiently circumvented inter-subject anatomical and functional variability and thus improved labeling accuracy. Moreover, in comparison with a single-atlas approach, our multi-atlas labeling approach showed better performance in identifying subject-specific fROIs.

Brain extraction in pediatric ADC maps, toward characterizing neuro-development in multi-platform and multi-institution clinical images

Apparent Diffusion Coefficient (ADC) maps can be used to characterize myelination and to detect abnormalities in the developing brain. However, given the normal variation in regional ADC with myelination, detection of abnormalities is difficult when based on visual assessment. Quantitative and automated analysis of pediatric ADC maps is thus desired but requires accurate brain extraction as the first step. Currently, most existing brain extraction methods are optimized for structural T1-weighted MR images of fully myelinated brains. Due to differences in age and image contrast, these approaches do not translate well to pediatric ADC maps. To address this problem, we present a multi-atlas brain extraction framework that has 1) specificity: designed and optimized specifically for pediatric ADC maps; 2) generality: applicable to multi-platform and multi-institution data, and to subjects at various neuro-developmental stages across the first 6 years of life; 3) accuracy: highly accurate compared to expert annotations; and 4) consistency: consistently accurate regardless of sources of data and ages of subjects. We show how we achieve these goals, via optimizing major components in a multi-atlas brain extraction framework, and via developing and evaluating new criteria for its atlas ranking component. Moreover, we demonstrate that these goals can be achieved with a fixed set of atlases and a fixed set of parameters, which opens doors for our optimized framework to be used in large-scale and multi-institution neuro-developmental and clinical studies. In a pilot study, we use this framework in a dataset containing scanner-generated ADC maps from 308 pediatric patients collected during the course of routine clinical care. Our framework leads to successful quantifications of the changes in whole-brain volumes and mean ADC values across the first 6 years of life.

Multi-view Classification for Identification of Alzheimer's Disease

In this paper, we propose a multi-view learning method using Magnetic Resonance Imaging (MRI) data for Alzheimer's Disease (AD) diagnosis. Specifically, we extract both Region-Of-Interest (ROI) features and Histograms of Oriented Gradient (HOG) features from each MRI image, and then propose mapping HOG features onto the space of ROI features to make them comparable and to impose high intra-class similarity with low inter-class similarity. Finally, both mapped HOG features and original ROI features are input to the support vector machine for AD diagnosis. The purpose of mapping HOG features onto the space of ROI features is to provide complementary information so that features from different views can not only be comparable (i.e., homogeneous) but also be interpretable. For example, ROI features are robust to noise, but lack of reflecting small or subtle changes, while HOG features are diverse but less robust to noise. The proposed multi-view learning method is designed to learn the transformation between two spaces and to separate the classes under the supervision of class labels. The experimental results on the MRI images from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset show that the proposed multi-view method helps enhance disease status identification performance, outperforming both baseline methods and state-of-the-art methods.

A transversal approach for patch-based label fusion via matrix completion

Recently, multi-atlas patch-based label fusion has received an increasing interest in the medical image segmentation field. After warping the anatomical labels from the atlas images to the target image by registration, label fusion is the key step to determine the latent label for each target image point. Two popular types of patch-based label fusion approaches are (1) reconstruction-based approaches that compute the target labels as a weighted average of atlas labels, where the weights are derived by reconstructing the target image patch using the atlas image patches; and (2) classification-based approaches that determine the target label as a mapping of the target image patch, where the mapping function is often learned using the atlas image patches and their corresponding labels. Both approaches have their advantages and limitations. In this paper, we propose a novel patch-based label fusion method to combine the above two types of approaches via matrix completion (and hence, we call it transversal). As we will show, our method overcomes the individual limitations of both reconstruction-based and classification-based approaches. Since the labeling confidences may vary across the target image points, we further propose a sequential labeling framework that first labels the highly confident points and then gradually labels more challenging points in an iterative manner, guided by the label information determined in the previous iterations. We demonstrate the performance of our novel label fusion method in segmenting the hippocampus in the ADNI dataset, subcortical and limbic structures in the LONI dataset, and mid-brain structures in the SATA dataset. We achieve more accurate segmentation results than both reconstruction-based and classification-based approaches. Our label fusion method is also ranked 1st in the online SATA Multi-Atlas Segmentation Challenge.

Multi-atlas segmentation of biomedical images: A survey

Multi-atlas segmentation (MAS), first introduced and popularized by the pioneering work of Rohlfing, et al. (2004), Klein, et al. (2005), and Heckemann, et al. (2006), is becoming one of the most widely-used and successful image segmentation techniques in biomedical applications. By manipulating and utilizing the entire dataset of "atlases" (training images that have been previously labeled, e.g., manually by an expert), rather than some model-based average representation, MAS has the flexibility to better capture anatomical variation, thus offering superior segmentation accuracy. This benefit, however, typically comes at a high computational cost. Recent advancements in computer hardware and image processing software have been instrumental in addressing this challenge and facilitated the wide adoption of MAS. Today, MAS has come a long way and the approach includes a wide array of sophisticated algorithms that employ ideas from machine learning, probabilistic modeling, optimization, and computer vision, among other fields. This paper presents a survey of published MAS algorithms and studies that have applied these methods to various biomedical problems. In writing this survey, we have three distinct aims. Our primary goal is to document how MAS was originally conceived, later evolved, and now relates to alternative methods. Second, this paper is intended to be a detailed reference of past research activity in MAS, which now spans over a decade (2003-2014) and entails novel methodological developments and application-specific solutions. Finally, our goal is to also present a perspective on the future of MAS, which, we believe, will be one of the dominant approaches in biomedical image segmentation.