CEREBRUM: a fast and fully-volumetric Convolutional Encoder-decodeR for weakly-supervised sEgmentation of BRain strUctures from out-of-the-scanner MRI

Fighting the scanner effect in brain MRI segmentation with a progressive level-of-detail network trained on multi-site data

Many clinical and research studies of the human brain require accurate structural MRI segmentation. While traditional atlas-based methods can be applied to volumes from any acquisition site, recent deep learning algorithms ensure high accuracy only when tested on data from the same sites exploited in training (i.e., internal data). Performance degradation experienced on external data (i.e., unseen volumes from unseen sites) is due to the inter-site variability in intensity distributions, and to unique artefacts caused by different MR scanner models and acquisition parameters. To mitigate this site-dependency, often referred to as the scanner effect, we propose LOD-Brain, a 3D convolutional neural network with progressive levels-of-detail (LOD), able to segment brain data from any site. Coarser network levels are responsible for learning a robust anatomical prior helpful in identifying brain structures and their locations, while finer levels refine the model to handle site-specific intensity distributions and anatomical variations. We ensure robustness across sites by training the model on an unprecedentedly rich dataset aggregating data from open repositories: almost 27,000 T1w volumes from around 160 acquisition sites, at 1.5 - 3T, from a population spanning from 8 to 90 years old. Extensive tests demonstrate that LOD-Brain produces state-of-the-art results, with no significant difference in performance between internal and external sites, and robust to challenging anatomical variations. Its portability paves the way for large-scale applications across different healthcare institutions, patient populations, and imaging technology manufacturers. Code, model, and demo are available on the project website.

Weakly Supervised Object Localization and Detection: A Survey

As an emerging and challenging problem in the computer vision community, weakly supervised object localization and detection plays an important role for developing new generation computer vision systems and has received significant attention in the past decade. As methods have been proposed, a comprehensive survey of these topics is of great importance. In this work, we review (1) classic models, (2) approaches with feature representations from off-the-shelf deep networks, (3) approaches solely based on deep learning, and (4) publicly available datasets and standard evaluation metrics that are widely used in this field. We also discuss the key challenges in this field, development history of this field, advantages/disadvantages of the methods in each category, the relationships between methods in different categories, applications of the weakly supervised object localization and detection methods, and potential future directions to further promote the development of this research field.

Weakly Supervised Skull Stripping of Magnetic Resonance Imaging of Brain Tumor Patients

Automatic brain tumor segmentation is particularly challenging on magnetic resonance imaging (MRI) with marked pathologies, such as brain tumors, which usually cause large displacement, abnormal appearance, and deformation of brain tissue. Despite an abundance of previous literature on learning-based methodologies for MRI segmentation, few works have focused on tackling MRI skull stripping of brain tumor patient data. This gap in literature can be associated with the lack of publicly available data (due to concerns about patient identification) and the labor-intensive nature of generating ground truth labels for model training. In this retrospective study, we assessed the performance of Dense-Vnet in skull stripping brain tumor patient MRI trained on our large multi-institutional brain tumor patient dataset. Our data included pretreatment MRI of 668 patients from our in-house institutional review board-approved multi-institutional brain tumor repository. Because of the absence of ground truth, we used imperfect automatically generated training labels using SPM12 software. We trained the network using common MRI sequences in oncology: T1-weighted with gadolinium contrast, T2-weighted fluid-attenuated inversion recovery, or both. We measured model performance against 30 independent brain tumor test cases with available manual brain masks. All images were harmonized for voxel spacing and volumetric dimensions before model training. Model training was performed using the modularly structured deep learning platform NiftyNet that is tailored toward simplifying medical image analysis. Our proposed approach showed the success of a weakly supervised deep learning approach in MRI brain extraction even in the presence of pathology. Our best model achieved an average Dice score, sensitivity, and specificity of, respectively, 94.5, 96.4, and 98.5% on the multi-institutional independent brain tumor test set. To further contextualize our results within existing literature on healthy brain segmentation, we tested the model against healthy subjects from the benchmark LBPA40 dataset. For this dataset, the model achieved an average Dice score, sensitivity, and specificity of 96.2, 96.6, and 99.2%, which are, although comparable to other publications, slightly lower than the performance of models trained on healthy patients. We associate this drop in performance with the use of brain tumor data for model training and its influence on brain appearance.

Quantitative magnetic resonance imaging for segmentation and white matter extraction of the hypothalamus

Since the hypothalamus is involved in many neuroendocrine, metabolic, and affective disorders, detailed hypothalamic imaging has become of major interest to better characterize disease-induced tissue damages and abnormalities. Still, image contrast of conventional anatomical magnetic resonance imaging lacks morphological detail, thus complicating complete and precise segmentation of the hypothalamus. The hypothalamus' position lateral to the third ventricle and close proximity to white matter tracts including the optic tract, fornix, and mammillothalamic tract display one of the remaining shortcomings of hypothalamic segmentation, as reliable exclusion of white matter is not yet possible. Recent studies found that quantitative magnetic resonance imaging (qMRI), a method to create maps of different standardized tissue contents, improved segmentation of cortical and subcortical brain regions. So far, this has not been tested for the hypothalamus. Therefore, in this study, we investigated the usability of qMRI and diffusion MRI for the purpose of detailed and reproducible manual segmentation of the hypothalamus and data-driven white matter extraction and compared our results to recent state-of-the-art segmentations. Our results show that qMRI presents good contrast for delineation of the hypothalamus and white matter, and that the properties of these images differ between subunits, such that they can be used to reliably exclude white matter from hypothalamic tissue. We propose that qMRI poses a useful addition to detailed hypothalamic segmentation and volumetry.

CEREBRUM-7T: Fast and Fully Volumetric Brain Segmentation of 7 Tesla MR Volumes

Ultra-high-field magnetic resonance imaging (MRI) enables sub-millimetre resolution imaging of the human brain, allowing the study of functional circuits of cortical layers at the meso-scale. An essential step in many functional and structural neuroimaging studies is segmentation, the operation of partitioning the MR images in anatomical structures. Despite recent efforts in brain imaging analysis, the literature lacks in accurate and fast methods for segmenting 7-tesla (7T) brain MRI. We here present CEREBRUM-7T, an optimised end-to-end convolutional neural network, which allows fully automatic segmentation of a whole 7T T1_w MRI brain volume at once, without partitioning the volume, pre-processing, nor aligning it to an atlas. The trained model is able to produce accurate multi-structure segmentation masks on six different classes plus background in only a few seconds. The experimental part, a combination of objective numerical evaluations and subjective analysis, confirms that the proposed solution outperforms the training labels it was trained on and is suitable for neuroimaging studies, such as layer functional MRI studies. Taking advantage of a fine-tuning operation on a reduced set of volumes, we also show how it is possible to effectively apply CEREBRUM-7T to different sites data. Furthermore, we release the code, 7T data, and other materials, including the training labels and the Turing test.

Predicting the retinotopic organization of human visual cortex from anatomy using geometric deep learning

Whether it be in a single neuron or a more complex biological system like the human brain, form and function are often directly related. The functional organization of human visual cortex, for instance, is tightly coupled with the underlying anatomy with cortical shape having been shown to be a useful predictor of the retinotopic organization in early visual cortex. Although the current state-of-the-art in predicting retinotopic maps is able to account for gross individual differences, such models are unable to account for any idiosyncratic differences in the structure-function relationship from anatomical information alone due to their initial assumption of a template. Here we developed a geometric deep learning model capable of exploiting the actual structure of the cortex to learn the complex relationship between brain function and anatomy in human visual cortex such that more realistic and idiosyncratic maps could be predicted. We show that our neural network was not only able to predict the functional organization throughout the visual cortical hierarchy, but that it was also able to predict nuanced variations across individuals. Although we demonstrate its utility for modeling the relationship between structure and function in human visual cortex, our approach is flexible and well-suited for a range of other applications involving data structured in non-Euclidean spaces.

Thalamus Atrophy in the Peri-Pregnancy Period in Clinically Stable Multiple Sclerosis Patients: Preliminary Results

Radiological activity in the post-partum period in MS patients is a well-known phenomenon, but there is no data concerning the influence of pregnancy on regional brain atrophy. The aim of this article was to investigate local brain atrophy in the peri-pregnancy period (PPP) in patients with MS. Thalamic volume (TV); corpus callosum volume (CCV) and classical MRI activity (new gadolinium enhancing lesions (Gd+), new T2 lesions, T1 lesions volume (T1LV) and T2 lesions volume (T2LV)) were analyzed in 12 clinically stable women with relapsing–remitting MS and with MRI performed in the PPP. We showed that there was a significant decrease in TV (p = 0.021) in the PPP. We also observed a significant increase in the T1 lesion volume (p = 0.028), new gadolinium-enhanced and new T2 lesions (in 46% and 77% of the scans, respectively) in the post-partum period. Our results suggest that the PPP in MS may be associated not only with classical MRI activity but, also, with regional brain atrophy.

Automated multiclass tissue segmentation of clinical brain MRIs with lesions

•

A U-Net incorporating spatial prior information can successfully segment 6 brain tissue types.

•

The U-Net was able to segment gray and white matter in the presence of lesions.

•

The U-Net surpassed the performance of its source algorithm in an external dataset.

•

Segmentations were produced in a hundredth of the time of its predecessor algorithm.

Delineation and quantification of normal and abnormal brain tissues on Magnetic Resonance Images is fundamental to the diagnosis and longitudinal assessment of neurological diseases. Here we sought to develop a convolutional neural network for automated multiclass tissue segmentation of brain MRIs that was robust at typical clinical resolutions and in the presence of a variety of lesions. We trained a 3D U-Net for full brain multiclass tissue segmentation from a prior atlas-based segmentation method on an internal dataset that consisted of 558 clinical T1-weighted brain MRIs (453/52/53; training/validation/test) of patients with one of 50 different diagnostic entities (n = 362) or with a normal brain MRI (n = 196). We then used transfer learning to refine our model on an external dataset that consisted of 7 patients with hand-labeled tissue types. We evaluated the tissue-wise and intra-lesion performance with different loss functions and spatial prior information in the validation set and applied the best performing model to the internal and external test sets. The network achieved an average overall Dice score of 0.87 and volume similarity of 0.97 in the internal test set. Further, the network achieved a median intra-lesion tissue segmentation accuracy of 0.85 inside lesions within white matter and 0.61 inside lesions within gray matter. After transfer learning, the network achieved an average overall Dice score of 0.77 and volume similarity of 0.96 in the external dataset compared to human raters. The network had equivalent or better performance than the original atlas-based method on which it was trained across all metrics and produced segmentations in a hundredth of the time. We anticipate that this pipeline will be a useful tool for clinical decision support and quantitative analysis of clinical brain MRIs in the presence of lesions.

A Cascade-SEME network for COVID-19 detection in chest x-ray images

PURPOSE

The worldwide spread of the SARS-CoV-2 virus poses unprecedented challenges to medical resources and infection prevention and control measures around the world. In this case, a rapid and effective detection method for COVID-19 can not only relieve the pressure of the medical system but find and isolate patients in time, to a certain extent, slow down the development of the epidemic. In this paper, we propose a method that can quickly and accurately diagnose whether pneumonia is viral pneumonia, and classify viral pneumonia in a fine-grained way to diagnose COVID-19.

METHODS

We proposed a Cascade Squeeze-Excitation and Moment Exchange (Cascade-SEME) framework that can effectively detect COVID-19 cases by evaluating the chest x-ray images, where SE is the structure we designed in the network which has attention mechanism, and ME is a method for image enhancement from feature dimension. The framework integrates a model for a coarse level detection of virus cases among other forms of lung infection, and a model for fine-grained categorisation of pneumonia types identifying COVID-19 cases. In addition, a Regional Learning approach is proposed to mitigate the impact of non-lesion features on network training. The network output is also visualised, highlighting the likely areas of lesion, to assist experts' assessment and diagnosis of COVID-19.

RESULTS

Three datasets were used: a set of Chest x-ray Images for Classification with bacterial pneumonia, viral pneumonia and normal chest x-rays, a COVID chest x-ray dataset with COVID-19, and a Lung Segmentation dataset containing 1000 chest x-rays with masks in the lung region. We evaluated all the models on the test set. The results shows the proposed SEME structure significantly improves the performance of the models: in the task of pneumonia infection type diagnosis, the sensitivity, specificity, accuracy and F1 score of ResNet50 with SEME structure are significantly improved in each category, and the accuracy and AUC of the whole test set are also enhanced; in the detection task of COVID-19, the evaluation results shows that when SEME structure was added to the task, the sensitivities, specificities, accuracy and F1 scores of ResNet50 and DenseNet169 are improved. Although the sensitivities and specificities are not significantly promoted, SEME well balanced these two significant indicators. Regional learning also plays an important role. Experiments show that Regional Learning can effectively correct the impact of non-lesion features on the network, which can be seen in the Grad-CAM method.

CONCLUSIONS

Experiments show that after the application of SEME structure in the network, the performance of SEME-ResNet50 and SEME-DenseNet169 in both two datasets show a clear enhancement. And the proposed regional learning method effectively directs the network's attention to focus on relevant pathological regions in the lung radiograph, ensuring the performance of the proposed framework even when a small training set is used. The visual interpretation step using Grad-CAM finds that the region of attention on radiographs of different types of pneumonia are located in different regions of the lungs.

BS-Net: Learning COVID-19 pneumonia severity on a large chest X-ray dataset

In this work we design an end-to-end deep learning architecture for predicting, on Chest X-rays images (CXR), a multi-regional score conveying the degree of lung compromise in COVID-19 patients. Such semi-quantitative scoring system, namely Brixia score, is applied in serial monitoring of such patients, showing significant prognostic value, in one of the hospitals that experienced one of the highest pandemic peaks in Italy. To solve such a challenging visual task, we adopt a weakly supervised learning strategy structured to handle different tasks (segmentation, spatial alignment, and score estimation) trained with a “from-the-part-to-the-whole” procedure involving different datasets. In particular, we exploit a clinical dataset of almost 5,000 CXR annotated images collected in the same hospital. Our BS-Net demonstrates self-attentive behavior and a high degree of accuracy in all processing stages. Through inter-rater agreement tests and a gold standard comparison, we show that our solution outperforms single human annotators in rating accuracy and consistency, thus supporting the possibility of using this tool in contexts of computer-assisted monitoring. Highly resolved (super-pixel level) explainability maps are also generated, with an original technique, to visually help the understanding of the network activity on the lung areas. We also consider other scores proposed in literature and provide a comparison with a recently proposed non-specific approach. We eventually test the performance robustness of our model on an assorted public COVID-19 dataset, for which we also provide Brixia score annotations, observing good direct generalization and fine-tuning capabilities that highlight the portability of BS-Net in other clinical settings. The CXR dataset along with the source code and the trained model are publicly released for research purposes.

Differentiable neural architecture search for optimal spatial/temporal brain function network decomposition

It has been a key topic to decompose the brain's spatial/temporal function networks from 4D functional magnetic resonance imaging (fMRI) data. With the advantages of robust and meaningful brain pattern extraction, deep neural networks have been shown to be more powerful and flexible in fMRI data modeling than other traditional methods. However, the challenge of designing neural network architecture for high-dimensional and complex fMRI data has also been realized recently. In this paper, we propose a new spatial/temporal differentiable neural architecture search algorithm (ST-DARTS) for optimal brain network decomposition. The core idea of ST-DARTS is to optimize the inner cell structure of the vanilla recurrent neural network (RNN) in order to effectively decompose spatial/temporal brain function networks from fMRI data. Based on the evaluations on all seven fMRI tasks in human connectome project (HCP) dataset, the ST-DARTS model is shown to perform promisingly, both spatially (i.e., it can recognize the most stimuli-correlated spatial brain network activation that is very similar to the benchmark) and temporally (i.e., its temporal activity is highly positively correlated with the task-design). To further improve the efficiency of ST-DARTS model, we introduce a flexible early-stopping mechanism, named as ST-DARTS+, which further improves experimental results significantly. To our best knowledge, the proposed ST-DARTS and ST-DARTS+ models are among the early efforts in optimally decomposing spatial/temporal brain function networks from fMRI data with neural architecture search strategy and they demonstrate great promise.