A logarithmic opinion pool based STAPLE algorithm for the fusion of segmentations with associated reliability weights

Advances in Fetal Brain Imaging

Over the last 20 years, there have been remarkable developments in fetal brain MR imaging analysis methods. This article delves into the specifics of structural imaging, diffusion imaging, functional MR imaging, and spectroscopy, highlighting the latest advancements in motion correction, fetal brain development atlases, and the challenges and innovations. Furthermore, this article explores the clinical applications of these advanced imaging techniques in comprehending and diagnosing fetal brain development and abnormalities.

Development of an enhanced scoring system to predict ICU readmission or in-hospital death within 24 hours using routine patient data from two NHS Foundation Trusts

RATIONALE

Intensive care units (ICUs) admit the most severely ill patients. Once these patients are discharged from the ICU to a step-down ward, they continue to have their vital signs monitored by nursing staff, with Early Warning Score (EWS) systems being used to identify those at risk of deterioration.

OBJECTIVES

We report the development and validation of an enhanced continuous scoring system for predicting adverse events, which combines vital signs measured routinely on acute care wards (as used by most EWS systems) with a risk score of a future adverse event calculated on discharge from the ICU.

DESIGN

A modified Delphi process identified candidate variables commonly available in electronic records as the basis for a 'static' score of the patient's condition immediately after discharge from the ICU. L1-regularised logistic regression was used to estimate the in-hospital risk of future adverse event. We then constructed a model of physiological normality using vital sign data from the day of hospital discharge. This is combined with the static score and used continuously to quantify and update the patient's risk of deterioration throughout their hospital stay.

SETTING

Data from two National Health Service Foundation Trusts (UK) were used to develop and (externally) validate the model.

PARTICIPANTS

A total of 12 394 vital sign measurements were acquired from 273 patients after ICU discharge for the development set, and 4831 from 136 patients in the validation cohort.

RESULTS

Outcome validation of our model yielded an area under the receiver operating characteristic curve of 0.724 for predicting ICU readmission or in-hospital death within 24 hours. It showed an improved performance with respect to other competitive risk scoring systems, including the National EWS (0.653).

CONCLUSIONS

We showed that a scoring system incorporating data from a patient's stay in the ICU has better performance than commonly used EWS systems based on vital signs alone.

TRIAL REGISTRATION NUMBER

ISRCTN32008295.

Assessing robustness and generalization of a deep neural network for brain MS lesion segmentation on real-world data

OBJECTIVES

Evaluate the performance of a deep learning (DL)-based model for multiple sclerosis (MS) lesion segmentation and compare it to other DL and non-DL algorithms.

METHODS

This ambispective, multicenter study assessed the performance of a DL-based model for MS lesion segmentation and compared it to alternative DL- and non-DL-based methods. Models were tested on internal (n = 20) and external (n = 18) datasets from Latin America, and on an external dataset from Europe (n = 49). We also examined robustness by rescanning six patients (n = 6) from our MS clinical cohort. Moreover, we studied inter-human annotator agreement and discussed our findings in light of these results. Performance and robustness were assessed using intraclass correlation coefficient (ICC), Dice coefficient (DC), and coefficient of variation (CV).

RESULTS

Inter-human ICC ranged from 0.89 to 0.95, while spatial agreement among annotators showed a median DC of 0.63. Using expert manual segmentations as ground truth, our DL model achieved a median DC of 0.73 on the internal, 0.66 on the external, and 0.70 on the challenge datasets. The performance of our DL model exceeded that of the alternative algorithms on all datasets. In the robustness experiment, our DL model also achieved higher DC (ranging from 0.82 to 0.90) and lower CV (ranging from 0.7 to 7.9%) when compared to the alternative methods.

CONCLUSION

Our DL-based model outperformed alternative methods for brain MS lesion segmentation. The model also proved to generalize well on unseen data and has a robust performance and low processing times both on real-world and challenge-based data.

CLINICAL RELEVANCE STATEMENT

Our DL-based model demonstrated superior performance in accurately segmenting brain MS lesions compared to alternative methods, indicating its potential for clinical application with improved accuracy, robustness, and efficiency.

KEY POINTS

• Automated lesion load quantification in MS patients is valuable; however, more accurate methods are still necessary. • A novel deep learning model outperformed alternative MS lesion segmentation methods on multisite datasets. • Deep learning models are particularly suitable for MS lesion segmentation in clinical scenarios.

Investigation of an efficient multi-modal convolutional neural network for multiple sclerosis lesion detection

In this study, an automated 2D machine learning approach for fast and precise segmentation of MS lesions from multi-modal magnetic resonance images (mmMRI) is presented. The method is based on an U-Net like convolutional neural network (CNN) for automated 2D slice-based-segmentation of brain MRI volumes. The individual modalities are encoded in separate downsampling branches without weight sharing, to leverage the specific features. Skip connections input feature maps to multi-scale feature fusion (MSFF) blocks at every decoder stage of the network. Those are followed by multi-scale feature upsampling (MSFU) blocks which use the information about lesion shape and location. The CNN is evaluated on two publicly available datasets: The ISBI 2015 longitudinal MS lesion segmentation challenge dataset containing 19 subjects and the MICCAI 2016 MSSEG challenge dataset containing 15 subjects from various scanners. The proposed multi-input 2D architecture is among the top performing approaches in the ISBI challenge, to which open-access papers are available, is able to outperform state-of-the-art 3D approaches without additional post-processing, can be adapted to other scanners quickly, is robust against scanner variability and can be deployed for inference even on a standard laptop without a dedicated GPU.

Learning from multiple annotators for medical image segmentation

Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the "actual" segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator's "unreliable behavior" (we call it "maximally unreliable") while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method's efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators' mistakes.

Multi-arm U-Net with dense input and skip connectivity for T2 lesion segmentation in clinical trials of multiple sclerosis

T2 lesion quantification plays a crucial role in monitoring disease progression and evaluating treatment response in multiple sclerosis (MS). We developed a 3D, multi-arm U-Net for T2 lesion segmentation, which was trained on a large, multicenter clinical trial dataset of relapsing MS. We investigated its generalization to other relapsing and primary progressive MS clinical trial datasets, and to an external dataset from the MICCAI 2016 MS lesion segmentation challenge. Additionally, we assessed the model's ability to reproduce the separation of T2 lesion volumes between treatment and control arms; and the association of baseline T2 lesion volumes with clinical disability scores compared with manual lesion annotations. The trained model achieved a mean dice coefficient of ≥ 0.66 and a lesion detection sensitivity of ≥ 0.72 across the internal test datasets. On the external test dataset, the model achieved a mean dice coefficient of 0.62, which is comparable to 0.59 from the best model in the challenge, and a lesion detection sensitivity of 0.68. Lesion detection performance was reduced for smaller lesions (≤ 30 μL, 3-10 voxels). The model successfully maintained the separation of the longitudinal changes in T2 lesion volumes between the treatment and control arms. Such tools could facilitate semi-automated MS lesion quantification; and reduce rater burden in clinical trials.

Improved accuracy of auto-segmentation of organs at risk in radiotherapy planning for nasopharyngeal carcinoma based on fully convolutional neural network deep learning

OBJECTIVE

We examined a modified encoder-decoder architecture-based fully convolutional neural network, OrganNet, for simultaneous auto-segmentation of 24 organs at risk (OARs) in the head and neck, followed by validation tests and evaluation of clinical application.

MATERIALS AND METHODS

Computed tomography (CT) images from 310 radiotherapy plans were used as the experimental data set, of which 260 and 50 were used as the training and test sets, respectively. An improved U-Net architecture was established by introducing a batch normalization layer, residual squeeze-and-excitation layer, and unique organ-specific loss function for deep learning training. The performance of the trained network model was evaluated by comparing the manual-delineation and the STAPLE contour of 10 physicians from different centers.

RESULTS

Our model achieved good segmentation in all 24 OARs in nasopharyngeal cancer radiotherapy plan CT images, with an average Dice similarity coefficient of 83.75%. Specifically, the mean Dice coefficients in large-volume organs (brainstem, spinal cord, left/right parotid glands, left/right temporal lobes, and left/right mandibles) were 84.97% - 95.00%, and in small-volume organs (pituitary, lens, optic nerve, and optic chiasma) were 55.46% - 91.56%. respectively. Using the STAPLE contours as standard contour, the OrganNet achieved comparable or better DICE in organ segmentation then that of the manual-delineation as well.

CONCLUSION

The established OrganNet enables simultaneous automatic segmentation of multiple targets on CT images of the head and neck radiotherapy plans, effectively improves the accuracy of U-Net based segmentation for OARs, especially for small-volume organs.

Leveraging non-expert crowdsourcing to segment the optic cup and disc of multicolor fundus images

Multicolor scanning laser imaging (MCI) images have broad application potential in the diagnosis of fundus diseases such as glaucoma. However, the performance level of automatic aided diagnosis systems based on MCI images is limited by the lack of high-quality annotations of numerous images. Producing annotations for vast amounts of MCI images will be a prolonged process if we only employ experts. Therefore, we consider non-expert crowdsourcing, which is an alternative approach to produce useful annotations efficiently and low cost. In this work, we aim to explore the effectiveness of non-expert crowdsourcing on the segmentation of the optic cup (OC) and optic disc (OD), which is an upstream task for glaucoma diagnosis, using MCI images. To this end, desensitized MCI images are independently annotated by four non-expert annotators, constructing a crowdsourcing dataset. To profit from crowdsourcing, we propose a model consisting of coupled regularization network and segmentation network. The regularization network generates learnable pixel-wise confusion matrices (CMs) that reflects preferences of each annotator. During training, the CMs and segmentation network are simultaneously optimized to enable dynamic trade-offs for non-expert annotations and generate reliable predictions. Crowdsourcing learning using our method have an average Mean Intersection Over Union ( M ) of 91.34%, while the average M of model trained by expert annotations is 91.72%. In addition, comparative experiments show that in our segmentation task non-expert crowdsourcing can be on a par with the expert who annotates 90% of data. Our work suggests that crowdsourcing in the segmentation of OC and OD using MCI images has the potential to be a substitute to expert annotation, which will accelerate the construction of large datasets to facilitate the application of deep learning in clinical diagnosis using MCI images.

Automatic segmentation of white matter hyperintensities: validation and comparison with state-of-the-art methods on both Multiple Sclerosis and elderly subjects

•

Automatic segmentation of MS lesions and age-related WMH from 3D T1 and T2-FLAIR.

•

Comparison to consensus show improved performance of WHASA-3D compared to WHASA.

•

WHASA-3D outperforms available state-of-the-art methods with their default settings.

•

WHASA-3D could be a useful tool for clinical practice and clinical trials.

Different types of white matter hyperintensities (WMH) can be observed through MRI in the brain and spinal cord, especially Multiple Sclerosis (MS) lesions for patients suffering from MS and age-related WMH for subjects with cognitive disorders and/or elderly people. To better diagnose and monitor the disease progression, the quantitative evaluation of WMH load has proven to be useful for clinical routine and trials. Since manual delineation for WMH segmentation is highly time-consuming and suffers from intra and inter observer variability, several methods have been proposed to automatically segment either MS lesions or age-related WMH, but none is validated on both WMH types. Here, we aim at proposing the White matter Hyperintensities Automatic Segmentation Algorithm adapted to 3D T2-FLAIR datasets (WHASA-3D), a fast and robust automatic segmentation tool designed to be implemented in clinical practice for the detection of both MS lesions and age-related WMH in the brain, using both 3D T1-weighted and T2-FLAIR images. In order to increase its robustness for MS lesions, WHASA-3D expands the original WHASA method, which relies on the coupling of non-linear diffusion framework and watershed parcellation, where regions considered as WMH are selected based on intensity and location characteristics, and finally refined with geodesic dilation. The previous validation was performed on 2D T2-FLAIR and subjects with cognitive disorders and elderly subjects. 60 subjects from a heterogeneous database of dementia patients, multiple sclerosis patients and elderly subjects with multiple MRI scanners and a wide range of lesion loads were used to evaluate WHASA and WHASA-3D through volume and spatial agreement in comparison with consensus reference segmentations. In addition, a direct comparison on the MS database with six available supervised and unsupervised state-of-the-art WMH segmentation methods (LST-LGA and LPA, Lesion-TOADS, lesionBrain, BIANCA and nicMSlesions) with default and optimised settings (when feasible) was conducted. WHASA-3D confirmed an improved performance with respect to WHASA, achieving a better spatial overlap (Dice) (0.67 vs 0.63), a reduced absolute volume error (AVE) (3.11 vs 6.2 mL) and an increased volume agreement (intraclass correlation coefficient, ICC) (0.96 vs 0.78). Compared to available state-of-the-art algorithms on the MS database, WHASA-3D outperformed both unsupervised and supervised methods when used with their default settings, showing the highest volume agreement (ICC = 0.95) as well as the highest average Dice (0.58). Optimising and/or retraining LST-LGA, BIANCA and nicMSlesions, using a subset of the MS database as training set, resulted in improved performances on the remaining testing set (average Dice: LST-LGA default/optimized = 0.41/0.51, BIANCA default/optimized = 0.22/0.39, nicMSlesions default/optimized = 0.17/0.63, WHASA-3D = 0.58). Evaluation and comparison results suggest that WHASA-3D is a reliable and easy-to-use method for the automated segmentation of white matter hyperintensities, for both MS lesions and age-related WMH. Further validation on larger datasets would be useful to confirm these first findings.

Robust Bayesian fusion of continuous segmentation maps

The fusion of probability maps is required when trying to analyse a collection of image labels or probability maps produced by several segmentation algorithms or human raters. The challenge is to weight the combination of maps correctly, in order to reflect the agreement among raters, the presence of outliers and the spatial uncertainty in the consensus. In this paper, we address several shortcomings of prior work in continuous label fusion. We introduce a novel approach to jointly estimate a reliable consensus map and to assess the presence of outliers and the confidence in each rater. Our robust approach is based on heavy-tailed distributions allowing local estimates of raters performances. In particular, we investigate the Laplace, the Student's t and the generalized double Pareto distributions, and compare them with respect to the classical Gaussian likelihood used in prior works. We unify these distributions into a common tractable inference scheme based on variational calculus and scale mixture representations. Moreover, the introduction of bias and spatial priors leads to proper rater bias estimates and control over the smoothness of the consensus map. Finally, we propose an approach that clusters raters based on variational boosting, and thus may produce several alternative consensus maps. Our approach was successfully tested on MR prostate delineations and on lung nodule segmentations from the LIDC-IDRI dataset.

Three-dimensional self super-resolution for pelvic floor MRI using a convolutional neural network with multi-orientation data training

PURPOSE

High-resolution pelvic magnetic resonance (MR) imaging is important for the high-resolution and high-precision evaluation of pelvic floor disorders (PFDs), but the data acquisition time is long. Because high-resolution three-dimensional (3D) MR data of the pelvic floor are difficult to obtain, MR images are usually obtained in three orthogonal planes: axial, sagittal, and coronal. The in-plane resolution of the MR data in each plane is high, but the through-plane resolution is low. Thus, we aimed to achieve 3D super-resolution using a convolutional neural network (CNN) approach to capture the intrinsic similarity of low-resolution 3D MR data from three orientations.

METHODS

We used a two-dimensional (2D) super-resolution CNN model to solve the 3D super-resolution problem. The residual-in-residual dense block network (RRDBNet) was used as our CNN backbone. For a given set of low through-plane resolution pelvic floor MR data in the axial or coronal or sagittal scan plane, we applied the RRDBNet sequentially to perform super-resolution on its two projected low-resolution views. Three datasets were used in the experiments, including two private datasets and one public dataset. In the first dataset (dataset 1), MR data acquired from 34 subjects in three planes were used to train our super-resolution model, and low-resolution MR data from nine subjects were used for testing. The second dataset (dataset 2) included a sequence of relatively high-resolution MR data acquired in the coronal plane. The public MR dataset (dataset 3) was used to demonstrate the generalization ability of our model. To show the effectiveness of RRDBNet, we used datasets 1 and 2 to compare RRDBNet with interpolation and enhanced deep super-resolution (EDSR) methods in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index. As 3D MR data from one view have two projected low-resolution views, different super-resolution orders were compared in terms of PSNR and SSIM. Finally, to demonstrate the impact of super-resolution on the image analysis task, we used datasets 2 and 3 to compare the performance of our method with interpolation on the 3D geometric model reconstruction of the urinary bladder.

RESULTS

A CNN-based method was used to learn the intrinsic similarity among MR acquisitions from different scan planes. Through-plane super-resolution for pelvic MR images was achieved without using high-resolution 3D data, which is useful for the analysis of PFDs.

Bayesian Spatial Binary Regression for Label Fusion in Structural Neuroimaging

Alzheimer's disease is a neurodegenerative condition that accelerates cognitive decline relative to normal aging. It is of critical scientific importance to gain a better understanding of early disease mechanisms in the brain to facilitate effective, targeted therapies. The volume of the hippocampus is often used in diagnosis and monitoring of the disease. Measuring this volume via neuroimaging is difficult since each hippocampus must either be manually identified or automatically delineated, a task referred to as segmentation. Automatic hippocampal segmentation often involves mapping a previously manually segmented image to a new brain image and propagating the labels to obtain an estimate of where each hippocampus is located in the new image. A more recent approach to this problem is to propagate labels from multiple manually segmented atlases and combine the results using a process known as label fusion. To date, most label fusion algorithms employ voting procedures with voting weights assigned directly or estimated via optimization. We propose using a fully Bayesian spatial regression model for label fusion that facilitates direct incorporation of covariate information while making accessible the entire posterior distribution. Our results suggest that incorporating tissue classification (e.g, gray matter) into the label fusion procedure can greatly improve segmentation when relatively homogeneous, healthy brains are used as atlases for diseased brains. The fully Bayesian approach also produces meaningful uncertainty measures about hippocampal volumes, information which can be leveraged to detect significant, scientifically meaningful differences between healthy and diseased populations, improving the potential for early detection and tracking of the disease.

Decision fusion in healthcare and medicine: a narrative review

OBJECTIVE

To provide an overview of the decision fusion (DF) technique and describe the applications of the technique in healthcare and medicine at prevention, diagnosis, treatment and administrative levels.

BACKGROUND

The rapid development of technology over the past 20 years has led to an explosion in data growth in various industries, like healthcare. Big data analysis within the healthcare systems is essential for arriving to a value-based decision over a period of time. Diversity and uncertainty in big data analytics have made it impossible to analyze data by using conventional data mining techniques and thus alternative solutions are required. DF is a form of data fusion techniques that could increase the accuracy of diagnosis and facilitate interpretation, summarization and sharing of information.

METHODS

We conducted a review of articles published between January 1980 and December 2020 from various databases such as Google Scholar, IEEE, PubMed, Science Direct, Scopus and web of science using the keywords decision fusion (DF), information fusion, healthcare, medicine and big data. A total of 141 articles were included in this narrative review.

CONCLUSIONS

Given the importance of big data analysis in reducing costs and improving the quality of healthcare; along with the potential role of DF in big data analysis, it is recommended to know the full potential of this technique including the advantages, challenges and applications of the technique before its use. Future studies should focus on describing the methodology and types of data used for its applications within the healthcare sector.

Multiple sclerosis lesions segmentation from multiple experts: The MICCAI 2016 challenge dataset

MRI plays a crucial role in multiple sclerosis diagnostic and patient follow-up. In particular, the delineation of T2-FLAIR hyperintense lesions is crucial although mostly performed manually - a tedious task. Many methods have thus been proposed to automate this task. However, sufficiently large datasets with a thorough expert manual segmentation are still lacking to evaluate these methods. We present a unique dataset for MS lesions segmentation evaluation. It consists of 53 patients acquired on 4 different scanners with a harmonized protocol. Hyperintense lesions on FLAIR were manually delineated on each patient by 7 experts with control on T2 sequence, and gathered in a consensus segmentation for evaluation. We provide raw and preprocessed data and a split of the dataset into training and testing data, the latter including data from a scanner not present in the training dataset. We strongly believe that this dataset will become a reference in MS lesions segmentation evaluation, allowing to evaluate many aspects: evaluation of performance on unseen scanner, comparison to individual experts performance, comparison to other challengers who already used this dataset, etc.

SoftSeg: Advantages of soft versus binary training for image segmentation

Most image segmentation algorithms are trained on binary masks formulated as a classification task per pixel. However, in applications such as medical imaging, this "black-and-white" approach is too constraining because the contrast between two tissues is often ill-defined, i.e., the voxels located on objects' edges contain a mixture of tissues (a partial volume effect). Consequently, assigning a single "hard" label can result in a detrimental approximation. Instead, a soft prediction containing non-binary values would overcome that limitation. In this study, we introduce SoftSeg, a deep learning training approach that takes advantage of soft ground truth labels, and is not bound to binary predictions. SoftSeg aims at solving a regression instead of a classification problem. This is achieved by using (i) no binarization after preprocessing and data augmentation, (ii) a normalized ReLU final activation layer (instead of sigmoid), and (iii) a regression loss function (instead of the traditional Dice loss). We assess the impact of these three features on three open-source MRI segmentation datasets from the spinal cord gray matter, the multiple sclerosis brain lesion, and the multimodal brain tumor segmentation challenges. Across multiple random dataset splittings, SoftSeg outperformed the conventional approach, leading to an increase in Dice score of 2.0% on the gray matter dataset (p=0.001), 3.3% for the brain lesions, and 6.5% for the brain tumors. SoftSeg produces consistent soft predictions at tissues' interfaces and shows an increased sensitivity for small objects (e.g., multiple sclerosis lesions). The richness of soft labels could represent the inter-expert variability, the partial volume effect, and complement the model uncertainty estimation, which is typically unclear with binary predictions. The developed training pipeline can easily be incorporated into most of the existing deep learning architectures. SoftSeg is implemented in the freely-available deep learning toolbox ivadomed (https://ivadomed.org).

3D Ultrasound Strain Imaging of Puborectalis Muscle

The female pelvic floor (PF) muscles provide support to the pelvic organs. During delivery, some of these muscles have to stretch up to three times their original length to allow passage of the baby, leading frequently to damage and consequently later-life PF dysfunction (PFD). Three-dimensional (3D) ultrasound (US) imaging can be used to image these muscles and to diagnose the damage by assessing quantitative, geometric and functional information of the muscles through strain imaging. In this study we developed 3D US strain imaging of the PF muscles and explored its application to the puborectalis muscle (PRM), which is one of the major PF muscles.

Simultaneous lesion and brain segmentation in multiple sclerosis using deep neural networks

Segmentation of white matter lesions and deep grey matter structures is an important task in the quantification of magnetic resonance imaging in multiple sclerosis. In this paper we explore segmentation solutions based on convolutional neural networks (CNNs) for providing fast, reliable segmentations of lesions and grey-matter structures in multi-modal MR imaging, and the performance of these methods when applied to out-of-centre data. We trained two state-of-the-art fully convolutional CNN architectures on the 2016 MSSEG training dataset, which was annotated by seven independent human raters: a reference implementation of a 3D Unet, and a more recently proposed 3D-to-2D architecture (DeepSCAN). We then retrained those methods on a larger dataset from a single centre, with and without labels for other brain structures. We quantified changes in performance owing to dataset shift, and changes in performance by adding the additional brain-structure labels. We also compared performance with freely available reference methods. Both fully-convolutional CNN methods substantially outperform other approaches in the literature when trained and evaluated in cross-validation on the MSSEG dataset, showing agreement with human raters in the range of human inter-rater variability. Both architectures showed drops in performance when trained on single-centre data and tested on the MSSEG dataset. When trained with the addition of weak anatomical labels derived from Freesurfer, the performance of the 3D Unet degraded, while the performance of the DeepSCAN net improved. Overall, the DeepSCAN network predicting both lesion and anatomical labels was the best-performing network examined.

A fully automated pipeline for brain structure segmentation in multiple sclerosis

Accurate volume measurements of the brain structures are important for treatment evaluation and disease follow-up in multiple sclerosis (MS) patients. With the aim of obtaining reproducible measurements and avoiding the intra-/inter-rater variability that manual delineations introduce, several automated brain structure segmentation strategies have been proposed in recent years. However, most of these strategies tend to be affected by the abnormal MS lesion intensities, which corrupt the structure segmentation result. To address this problem, we recently reformulated two label fusion strategies of the state of the art, improving their segmentation performance on the lesion areas. Here, we integrate these reformulated strategies in a completely automated pipeline that includes pre-processing (inhomogeneity correction and intensity normalization), atlas selection, masked registration and label fusion, and combine them with an automated lesion segmentation method of the state of the art. We study the effect of automating the lesion mask acquisition on the structure segmentation result, analyzing the output of the proposed pipeline when used in combination with manually and automatically segmented lesion masks. We further analyze the effect of those masks on the segmentation result of the original label fusion strategies when combined with the well-established pre-processing step of lesion filling. The experiments performed show that, when the original methods are used to segment the lesion-filled images, significant structure volume differences are observed in a comparison between manually and automatically segmented lesion masks. The results indicate a mean volume decrease of 1.13%±1.93 in the cerebrospinal fluid, and a mean volume increase of 0.13%±0.14 and 0.05%±0.08 in the cerebral white matter and cerebellar gray matter, respectively. On the other hand, no significant volume differences were found when the proposed automated pipeline was used for segmentation, which demonstrates its robustness against variations in the lesion mask used.

Unsupervised Domain Adaptation With Optimal Transport in Multi-Site Segmentation of Multiple Sclerosis Lesions From MRI Data

Automatic segmentation of Multiple Sclerosis (MS) lesions from Magnetic Resonance Imaging (MRI) images is essential for clinical assessment and treatment planning of MS. Recent years have seen an increasing use of Convolutional Neural Networks (CNNs) for this task. Although these methods provide accurate segmentation, their applicability in clinical settings remains limited due to a reproducibility issue across different image domains. MS images can have highly variable characteristics across patients, MRI scanners and imaging protocols; retraining a supervised model with data from each new domain is not a feasible solution because it requires manual annotation from expert radiologists. In this work, we explore an unsupervised solution to the problem of domain shift. We present a framework, Seg-JDOT, which adapts a deep model so that samples from a source domain and samples from a target domain sharing similar representations will be similarly segmented. We evaluated the framework on a multi-site dataset, MICCAI 2016, and showed that the adaptation toward a target site can bring remarkable improvements in a model performance over standard training.

White matter extension of the Melbourne Children's Regional Infant Brain atlas: M-CRIB-WM

Brain atlases providing standardised identification of neonatal brain regions are key in investigating neurological disorders of early childhood. Our previously developed Melbourne Children's Regional Infant Brain (M-CRIB) and M-CRIB 2.0 neonatal brain atlases provide standardised parcellation of 100 brain regions including cortical, subcortical, and cerebellar regions. The aim of this study was to extend M-CRIB atlas coverage to include 54 white matter (WM) regions. Participants were 10 healthy term-born neonates that were used to create the initial M-CRIB atlas. WM regions were manually segmented based on T₂ images and co-registered diffusion tensor imaging-based, direction-encoded colour maps. Our labelled regions imitate the Johns Hopkins University neonatal atlas, with minor anatomical modifications. All segmentations were reviewed and approved by a paediatric radiologist and a neurosurgery research fellow for anatomical accuracy. The resulting neonatal WM atlas comprises 54 WM regions: 24 paired regions, and six unpaired regions comprising five corpus callosum subdivisions, and one pontine crossing tract. Detailed protocols for manual WM parcellations are provided, and the M-CRIB-WM atlas is presented together with the existing M-CRIB cortical, subcortical, and cerebellar parcellations in 10 individual neonatal MRI data sets. The novel M-CRIB-WM atlas, along with the M-CRIB cortical and subcortical atlases, provide neonatal whole brain MRI coverage in the first multi-subject manually parcellated neonatal atlas compatible with atlases commonly used at older time points. The M-CRIB-WM atlas is publicly available, providing a valuable tool that will help facilitate neuroimaging research into neonatal brain development in both healthy and diseased states.

Biomechanical modeling and computer simulation of the brain during neurosurgery

Computational biomechanics of the brain for neurosurgery is an emerging area of research recently gaining in importance and practical applications. This review paper presents the contributions of the Intelligent Systems for Medicine Laboratory and its collaborators to this field, discussing the modeling approaches adopted and the methods developed for obtaining the numerical solutions. We adopt a physics-based modeling approach and describe the brain deformation in mechanical terms (such as displacements, strains, and stresses), which can be computed using a biomechanical model, by solving a continuum mechanics problem. We present our modeling approaches related to geometry creation, boundary conditions, loading, and material properties. From the point of view of solution methods, we advocate the use of fully nonlinear modeling approaches, capable of capturing very large deformations and nonlinear material behavior. We discuss finite element and meshless domain discretization, the use of the total Lagrangian formulation of continuum mechanics, and explicit time integration for solving both time-accurate and steady-state problems. We present the methods developed for handling contacts and for warping 3D medical images using the results of our simulations. We present two examples to showcase these methods: brain shift estimation for image registration and brain deformation computation for neuronavigation in epilepsy treatment.

Desikan-Killiany-Tourville Atlas Compatible Version of M-CRIB Neonatal Parcellated Whole Brain Atlas: The M-CRIB 2.0

Our recently published M-CRIB atlas comprises 100 neonatal brain regions including 68 compatible with the widely-used Desikan-Killiany adult cortical atlas. A successor to the Desikan-Killiany atlas is the Desikan-Killiany-Tourville atlas, in which some regions with unclear boundaries were removed, and many existing boundaries were revised to conform to clearer landmarks in sulcal fundi. Our first aim here was to modify cortical M-CRIB regions to comply with the Desikan-Killiany-Tourville protocol, in order to offer: (a) compatibility with this adult cortical atlas, (b) greater labeling accuracy due to clearer landmarks, and (c) optimisation of cortical regions for integration with surface-based infant parcellation pipelines. Secondly, we aimed to update subcortical regions in order to offer greater compatibility with subcortical segmentations produced in FreeSurfer. Data utilized were the T2-weighted MRI scans in our M-CRIB atlas, for 10 healthy neonates (post-menstrual age at MRI 40–43 weeks, four female), and corresponding parcellated images. Edits were performed on the parcellated images in volume space using ITK-SNAP. Cortical updates included deletion of frontal and temporal poles and ‘Banks STS,’ and modification of boundaries of many other regions. Changes to subcortical regions included the addition of ‘ventral diencephalon,’ and deletion of ‘subcortical matter’ labels. A detailed updated parcellation protocol was produced. The resulting whole-brain M-CRIB 2.0 atlas comprises 94 regions altogether. This atlas provides comparability with adult Desikan-Killiany-Tourville-labeled cortical data and FreeSurfer-labeed subcortical data, and is more readily adaptable for incorporation into surface-based neonatal parcellation pipelines. As such, it offers the ability to help facilitate a broad range of investigations into brain structure and function both at the neonatal time point and developmentally across the lifespan.

Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure

We present a study of multiple sclerosis segmentation algorithms conducted at the international MICCAI 2016 challenge. This challenge was operated using a new open-science computing infrastructure. This allowed for the automatic and independent evaluation of a large range of algorithms in a fair and completely automatic manner. This computing infrastructure was used to evaluate thirteen methods of MS lesions segmentation, exploring a broad range of state-of-theart algorithms, against a high-quality database of 53 MS cases coming from four centers following a common definition of the acquisition protocol. Each case was annotated manually by an unprecedented number of seven different experts. Results of the challenge highlighted that automatic algorithms, including the recent machine learning methods (random forests, deep learning, …), are still trailing human expertise on both detection and delineation criteria. In addition, we demonstrate that computing a statistically robust consensus of the algorithms performs closer to human expertise on one score (segmentation) although still trailing on detection scores.

Voxel-Wise Logistic Regression and Leave-One-Source-Out Cross Validation for white matter hyperintensity segmentation

Many algorithms have been proposed for automated segmentation of white matter hyperintensities (WMH) in brain MRI. Yet, broad uptake of any particular algorithm has not been observed. In this work, we argue that this may be due to variable and suboptimal validation data and frameworks, precluding direct comparison of methods on heterogeneous data. As a solution, we present Leave-One-Source-Out Cross Validation (LOSO-CV), which leverages all available data for performance estimation, and show that this gives more realistic (lower) estimates of segmentation algorithm performance on data from different scanners. We also develop a FLAIR-only WMH segmentation algorithm: Voxel-Wise Logistic Regression (VLR), inspired by the open-source Lesion Prediction Algorithm (LPA). Our variant facilitates more accurate parameter estimation, and permits intuitive interpretation of model parameters. We illustrate the performance of the VLR algorithm using the LOSO-CV framework with a dataset comprising freely available data from several recent competitions (96 images from 7 scanners). The performance of the VLR algorithm (median Similarity Index of 0.69) is compared to its LPA predecessor (0.58), and the results of the VLR algorithm in the 2017 WMH Segmentation Competition are also presented.

Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions

Quantitative analysis of brain MRI is routine for many neurological diseases and conditions and relies on accurate segmentation of structures of interest. Deep learning-based segmentation approaches for brain MRI are gaining interest due to their self-learning and generalization ability over large amounts of data. As the deep learning architectures are becoming more mature, they gradually outperform previous state-of-the-art classical machine learning algorithms. This review aims to provide an overview of current deep learning-based segmentation approaches for quantitative brain MRI. First we review the current deep learning architectures used for segmentation of anatomical brain structures and brain lesions. Next, the performance, speed, and properties of deep learning approaches are summarized and discussed. Finally, we provide a critical assessment of the current state and identify likely future developments and trends.

Active delineation of Meyer's loop using oriented priors through MAGNEtic tractography (MAGNET)

Streamline tractography algorithms infer connectivity from diffusion MRI (dMRI) by following diffusion directions which are similarly aligned between neighboring voxels. However, not all white matter (WM) fascicles are organized in this manner. For example, Meyer's loop is a highly curved portion of the optic radiation (OR) that exhibits a narrow turn, kissing and crossing pathways, and changes in fascicle dispersion. From a neurosurgical perspective, damage to Meyer's loop carries a potential risk of inducing vision deficits to the patient, especially during temporal lobe resection surgery. To prevent such impairment, achieving an accurate delineation of Meyer's loop with tractography is thus of utmost importance. However, current algorithms tend to under-estimate the full extent of Meyer's loop, mainly attributed to the aforementioned rule for connectivity which requires a direction to be chosen across a field of orientations. In this article, it was demonstrated that MAGNEtic Tractography (MAGNET) can benefit Meyer's loop delineation by incorporating anatomical knowledge of the expected fiber orientation to overcome local ambiguities. A new ROI-mechanism was proposed which supplies additional information to streamline reconstruction algorithms by the means of oriented priors. Their results showed that MAGNET can accurately generate Meyer's loop in all of our 15 child subjects (8 males; mean age 10.2 years ± 3.1). It effectively improved streamline coverage when compared with deterministic tractography, and significantly reduced the distance between the anterior-most portion of Meyer's loop and the temporal pole by 16.7 mm on average, a crucial landmark used for preoperative planning of temporal lobe surgery. Hum Brain Mapp 38:509-527, 2017. © 2016 Wiley Periodicals, Inc.

Simultaneous total intracranial volume and posterior fossa volume estimation using multi-atlas label fusion

Total intracranial volume (TICV) is an essential covariate in brain volumetric analyses. The prevalent brain imaging software packages provide automatic TICV estimates. FreeSurfer and FSL estimate TICV using a scaling factor while SPM12 accumulates probabilities of brain tissues. None of the three provide explicit skull/CSF boundary (SCB) since it is challenging to distinguish these dark structures in a T1-weighted image. However, explicit SCB not only leads to a natural way of obtaining TICV (i.e., counting voxels inside the skull) but also allows sub-definition of TICV, for example, the posterior fossa volume (PFV). In this article, they proposed to use multi-atlas label fusion to obtain TICV and PFV simultaneously. The main contributions are: (1) TICV and PFV are simultaneously obtained with explicit SCB from a single T1-weighted image. (2) TICV and PFV labels are added to the widely used BrainCOLOR atlases. (3) Detailed mathematical derivation of non-local spatial STAPLE (NLSS) label fusion is presented. As the skull is clearly distinguished in CT images, we use a semi-manual procedure to obtain atlases with TICV and PFV labels using 20 subjects who both have a MR and CT scan. The proposed method provides simultaneous TICV and PFV estimation while achieving more accurate TICV estimation compared with FreeSurfer, FSL, SPM12, and the previously proposed STAPLE based approach. The newly developed TICV and PFV labels for the OASIS BrainCOLOR atlases provide acceptable performance, which enables simultaneous TICV and PFV estimation during whole brain segmentation. The NLSS method and the new atlases have been made freely available. Hum Brain Mapp 38:599-616, 2017. © 2016 Wiley Periodicals, Inc.

Neurophysiological evidence of preserved connectivity in tuber tissue

We present a case of preserved corticospinal connectivity in a cortical tuber, in a 10 year-old boy with intractable epilepsy and tuberous sclerosis complex (TSC). The patient had multiple subcortical tubers, one of which was located in the right central sulcus. In preparation for epilepsy surgery, motor mapping, by neuronavigated transcranial magnetic stimulation (nTMS) coupled with surface electromyography (EMG) was performed to locate the primary motor cortical areas. The resulting functional motor map revealed expected corticospinal connectivity in the left precentral gyrus. Surprisingly, robust contralateral deltoid and tibialis anterior motor evoked potentials (MEPs) were also elicited with direct stimulation of the cortical tuber in the right central sulcus. MRI with diffusion tensor imaging (DTI) tractography confirmed corticospinal fibers originating in the tuber. As there are no current reports of preserved connectivity between a cortical tuber and the corticospinal tract, this case serves to highlight the functional interdigitation of tuber and eloquent cortex. Our case also illustrates the widening spectrum of neuropathological abnormality in TSC that is becoming apparent with modern MRI methodology. Finally, our finding underscores the need for further study of preserved function in tuber tissue during presurgical workup in patients with TSC.

Tubers are neither static nor discrete: Evidence from serial diffusion tensor imaging

OBJECTIVE

To assess the extent and evolution of tissue abnormality of tubers, perituber tissue, and normal-appearing white matter (NAWM) in patients with tuberous sclerosis complex using serial diffusion tensor imaging.

METHODS

We applied automatic segmentation based on a combined global-local intensity mixture model of 3T structural and 35 direction diffusion tensor MRIs (diffusion tensor imaging) to define 3 regions: tuber tissue, an equal volume perituber rim, and the remaining NAWM. For each patient, scan, lobe, and tissue type, we analyzed the averages of mean diffusivity (MD) and fractional anisotropy (FA) in a generalized additive mixed model.

RESULTS

Twenty-five patients (mean age 5.9 years; range 0.5-24.5 years) underwent 2 to 6 scans each, totaling 70 scans. Average time between scans was 1.2 years (range 0.4-2.9). Patient scans were compared with those of 73 healthy controls. FA values were lowest, and MD values were highest in tubers, next in perituber tissue, then in NAWM. Longitudinal analysis showed a positive (FA) and negative (MD) correlation with age in tubers, perituber tissue, and NAWM. All 3 tissue types followed a biexponential developmental trajectory, similar to the white matter of controls. An additional qualitative analysis showed a gradual transition of diffusion values across the tissue type boundaries.

CONCLUSIONS

Similar to NAWM, tuber and perituber tissues in tuberous sclerosis complex undergo microstructural evolution with age. The extent of diffusion abnormality decreases with distance to the tuber, in line with known extension of histologic, immunohistochemical, and molecular abnormalities beyond tuber pathology.

Multi-atlas segmentation of biomedical images: A survey

Multi-atlas segmentation (MAS), first introduced and popularized by the pioneering work of Rohlfing, et al. (2004), Klein, et al. (2005), and Heckemann, et al. (2006), is becoming one of the most widely-used and successful image segmentation techniques in biomedical applications. By manipulating and utilizing the entire dataset of "atlases" (training images that have been previously labeled, e.g., manually by an expert), rather than some model-based average representation, MAS has the flexibility to better capture anatomical variation, thus offering superior segmentation accuracy. This benefit, however, typically comes at a high computational cost. Recent advancements in computer hardware and image processing software have been instrumental in addressing this challenge and facilitated the wide adoption of MAS. Today, MAS has come a long way and the approach includes a wide array of sophisticated algorithms that employ ideas from machine learning, probabilistic modeling, optimization, and computer vision, among other fields. This paper presents a survey of published MAS algorithms and studies that have applied these methods to various biomedical problems. In writing this survey, we have three distinct aims. Our primary goal is to document how MAS was originally conceived, later evolved, and now relates to alternative methods. Second, this paper is intended to be a detailed reference of past research activity in MAS, which now spans over a decade (2003-2014) and entails novel methodological developments and application-specific solutions. Finally, our goal is to also present a perspective on the future of MAS, which, we believe, will be one of the dominant approaches in biomedical image segmentation.

Optimal MAP Parameters Estimation in STAPLE Using Local Intensity Similarity Information

In recent years, fusing segmentation results obtained based on multiple template images has become a standard practice in many medical imaging applications. Such multiple-templates-based methods are found to provide more reliable and accurate segmentations than the single-template-based methods. In this paper, we present a new approach for learning prior knowledge about the performance parameters of template images using the local intensity similarity information; we also propose a methodology to incorporate that prior knowledge through the estimation of the optimal MAP parameters. The proposed method is evaluated in the context of segmentation of structures in the brain magnetic resonance images by comparing our results with some of the state-of-the-art segmentation methods. These experiments have clearly demonstrated the advantages of learning and incorporating prior knowledge about the performance parameters using the proposed method.

A multi-atlas-based segmentation framework for prostate brachytherapy

Low-dose-rate brachytherapy is a radiation treatment method for localized prostate cancer. The standard of care for this treatment procedure is to acquire transrectal ultrasound images of the prostate in order to devise a plan to deliver sufficient radiation dose to the cancerous tissue. Brachytherapy planning involves delineation of contours in these images, which closely follow the prostate boundary, i.e., clinical target volume. This process is currently performed either manually or semi-automatically, which requires user interaction for landmark initialization. In this paper, we propose a multi-atlas fusion framework to automatically delineate the clinical target volume in ultrasound images. A dataset of a priori segmented ultrasound images, i.e., atlases, is registered to a target image. We introduce a pairwise atlas agreement factor that combines an image-similarity metric and similarity between a priori segmented contours. This factor is used in an atlas selection algorithm to prune the dataset before combining the atlas contours to produce a consensus segmentation. We evaluate the proposed segmentation approach on a set of 280 transrectal prostate volume studies. The proposed method produces segmentation results that are within the range of observer variability when compared to a semi-automatic segmentation technique that is routinely used in our cancer clinic.