Exploring uncertainty measures in deep networks for Multiple sclerosis lesion detection and segmentation

A review of uncertainty quantification in medical image analysis: Probabilistic and non-probabilistic methods

The comprehensive integration of machine learning healthcare models within clinical practice remains suboptimal, notwithstanding the proliferation of high-performing solutions reported in the literature. A predominant factor hindering widespread adoption pertains to an insufficiency of evidence affirming the reliability of the aforementioned models. Recently, uncertainty quantification methods have been proposed as a potential solution to quantify the reliability of machine learning models and thus increase the interpretability and acceptability of the results. In this review, we offer a comprehensive overview of the prevailing methods proposed to quantify the uncertainty inherent in machine learning models developed for various medical image tasks. Contrary to earlier reviews that exclusively focused on probabilistic methods, this review also explores non-probabilistic approaches, thereby furnishing a more holistic survey of research pertaining to uncertainty quantification for machine learning models. Analysis of medical images with the summary and discussion on medical applications and the corresponding uncertainty evaluation protocols are presented, which focus on the specific challenges of uncertainty in medical image analysis. We also highlight some potential future research work at the end. Generally, this review aims to allow researchers from both clinical and technical backgrounds to gain a quick and yet in-depth understanding of the research in uncertainty quantification for medical image analysis machine learning models.

Clinical applications of deep learning in neuroinflammatory diseases: A scoping review

BACKGROUND

Deep learning (DL) is an artificial intelligence technology that has aroused much excitement for predictive medicine due to its ability to process raw data modalities such as images, text, and time series of signals.

OBJECTIVES

Here, we intend to give the clinical reader elements to understand this technology, taking neuroinflammatory diseases as an illustrative use case of clinical translation efforts. We reviewed the scope of this rapidly evolving field to get quantitative insights about which clinical applications concentrate the efforts and which data modalities are most commonly used.

METHODS

We queried the PubMed database for articles reporting DL algorithms for clinical applications in neuroinflammatory diseases and the radiology.healthairegister.com website for commercial algorithms.

RESULTS

The review included 148 articles published between 2018 and 2024 and five commercial algorithms. The clinical applications could be grouped as computer-aided diagnosis, individual prognosis, functional assessment, the segmentation of radiological structures, and the optimization of data acquisition. Our review highlighted important discrepancies in efforts. The segmentation of radiological structures and computer-aided diagnosis currently concentrate most efforts with an overrepresentation of imaging. Various model architectures have addressed different applications, relatively low volume of data, and diverse data modalities. We report the high-level technical characteristics of the algorithms and synthesize narratively the clinical applications. Predictive performances and some common a priori on this topic are finally discussed.

CONCLUSION

The currently reported efforts position DL as an information processing technology, enhancing existing modalities of paraclinical investigations and bringing perspectives to make innovative ones actionable for healthcare.

Deep neural network uncertainty estimation for early oral cancer diagnosis

BACKGROUND

Early diagnosis in oral cancer is essential to reduce both morbidity and mortality. This study explores the use of uncertainty estimation in deep learning for early oral cancer diagnosis.

METHODS

We develop a Bayesian deep learning model termed 'Probabilistic HRNet', which utilizes the ensemble MC dropout method on HRNet. Additionally, two oral lesion datasets with distinct distributions are created. We conduct a retrospective study to assess the predictive performance and uncertainty of Probabilistic HRNet across these datasets.

RESULTS

Probabilistic HRNet performs optimally on the In-domain test set, achieving an F1 score of 95.3% and an AUC of 96.9% by excluding the top 30% high-uncertainty samples. For evaluations on the Domain-shift test set, the results show an F1 score of 64.9% and an AUC of 80.3%. After excluding 30% of the high-uncertainty samples, these metrics improve to an F1 score of 74.4% and an AUC of 85.6%.

CONCLUSION

Redirecting samples with high uncertainty to experts for subsequent diagnosis significantly decreases the rates of misdiagnosis, which highlights that uncertainty estimation is vital to ensure safe decision making for computer-aided early oral cancer diagnosis.

An Optimized Uncertainty-Aware Training Framework for Neural Networks

Uncertainty quantification (UQ) for predictions generated by neural networks (NNs) is of vital importance in safety-critical applications. An ideal model is supposed to generate low uncertainty for correct predictions and high uncertainty for incorrect predictions. The main focus of state-of-the-art training algorithms is to optimize the NN parameters to improve the accuracy-related metrics. Training based on uncertainty metrics has been fully ignored or overlooked in the literature. This article introduces a novel uncertainty-aware training algorithm for classification tasks. A novel predictive uncertainty estimate-based objective function is defined and optimized using the stochastic gradient descent method. This new multiobjective loss function covers both accuracy and uncertainty accuracy (UA) simultaneously during training. The performance of the proposed training framework is compared from different aspects with other UQ techniques for different benchmarks. The obtained results demonstrate the effectiveness of the proposed framework for developing the NN models capable of generating reliable uncertainty estimates.

Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis

The full acceptance of Deep Learning (DL) models in the clinical field is rather low with respect to the quantity of high-performing solutions reported in the literature. End users are particularly reluctant to rely on the opaque predictions of DL models. Uncertainty quantification methods have been proposed in the literature as a potential solution, to reduce the black-box effect of DL models and increase the interpretability and the acceptability of the result by the final user. In this review, we propose an overview of the existing methods to quantify uncertainty associated with DL predictions. We focus on applications to medical image analysis, which present specific challenges due to the high dimensionality of images and their variable quality, as well as constraints associated with real-world clinical routine. Moreover, we discuss the concept of structural uncertainty, a corpus of methods to facilitate the alignment of segmentation uncertainty estimates with clinical attention. We then discuss the evaluation protocols to validate the relevance of uncertainty estimates. Finally, we highlight the open challenges for uncertainty quantification in the medical field.

Anomaly guided segmentation: Introducing semantic context for lesion segmentation in retinal OCT using weak context supervision from anomaly detection

Automated lesion detection in retinal optical coherence tomography (OCT) scans has shown promise for several clinical applications, including diagnosis, monitoring and guidance of treatment decisions. However, segmentation models still struggle to achieve the desired results for some complex lesions or datasets that commonly occur in real-world, e.g. due to variability of lesion phenotypes, image quality or disease appearance. While several techniques have been proposed to improve them, one line of research that has not yet been investigated is the incorporation of additional semantic context through the application of anomaly detection models. In this study we experimentally show that incorporating weak anomaly labels to standard segmentation models consistently improves lesion segmentation results. This can be done relatively easy by detecting anomalies with a separate model and then adding these output masks as an extra class for training the segmentation model. This provides additional semantic context without requiring extra manual labels. We empirically validated this strategy using two in-house and two publicly available retinal OCT datasets for multiple lesion targets, demonstrating the potential of this generic anomaly guided segmentation approach to be used as an extra tool for improving lesion detection models.

Quantifying U-Net uncertainty in multi-parametric MRI-based glioma segmentation by spherical image projection

BACKGROUND

Uncertainty quantification in deep learning is an important research topic. For medical image segmentation, the uncertainty measurements are usually reported as the likelihood that each pixel belongs to the predicted segmentation region. In potential clinical applications, the uncertainty result reflects the algorithm's robustness and supports the confidence and trust of the segmentation result when the ground-truth result is absent. For commonly studied deep learning models, novel methods for quantifying segmentation uncertainty are in demand.

PURPOSE

To develop a U-Net segmentation uncertainty quantification method based on spherical image projection of multi-parametric MRI (MP-MRI) in glioma segmentation.

METHODS

The projection of planar MRI data onto a spherical surface is equivalent to a nonlinear image transformation that retains global anatomical information. By incorporating this image transformation process in our proposed spherical projection-based U-Net (SPU-Net) segmentation model design, multiple independent segmentation predictions can be obtained from a single MRI. The final segmentation is the average of all available results, and the variation can be visualized as a pixel-wise uncertainty map. An uncertainty score was introduced to evaluate and compare the performance of uncertainty measurements. The proposed SPU-Net model was implemented on the basis of 369 glioma patients with MP-MRI scans (T1, T1-Ce, T2, and FLAIR). Three SPU-Net models were trained to segment enhancing tumor (ET), tumor core (TC), and whole tumor (WT), respectively. The SPU-Net model was compared with (1) the classic U-Net model with test-time augmentation (TTA) and (2) linear scaling-based U-Net (LSU-Net) segmentation models in terms of both segmentation accuracy (Dice coefficient, sensitivity, specificity, and accuracy) and segmentation uncertainty (uncertainty map and uncertainty score).

RESULTS

The developed SPU-Net model successfully achieved low uncertainty for correct segmentation predictions (e.g., tumor interior or healthy tissue interior) and high uncertainty for incorrect results (e.g., tumor boundaries). This model could allow the identification of missed tumor targets or segmentation errors in U-Net. Quantitatively, the SPU-Net model achieved the highest uncertainty scores for three segmentation targets (ET/TC/WT): 0.826/0.848/0.936, compared to 0.784/0.643/0.872 using the U-Net with TTA and 0.743/0.702/0.876 with the LSU-Net (scaling factor = 2). The SPU-Net also achieved statistically significantly higher Dice coefficients, underscoring the improved segmentation accuracy.

CONCLUSION

The SPU-Net model offers a powerful tool to quantify glioma segmentation uncertainty while improving segmentation accuracy. The proposed method can be generalized to other medical image-related deep-learning applications for uncertainty evaluation.

Curriculum Consistency Learning and Multi-Scale Contrastive Constraint in Semi-Supervised Medical Image Segmentation

Data scarcity poses a significant challenge in medical image segmentation, thereby highlighting the importance of leveraging sparse annotation data. In addressing this issue, semi-supervised learning has emerged as an effective approach for training neural networks using limited labeled data. In this study, we introduced a curriculum consistency constraint within the context of semi-supervised medical image segmentation, thus drawing inspiration from the human learning process. By dynamically comparing patch features with full image features, we enhanced the network's ability to learn. Unlike existing methods, our approach adapted the patch size to simulate the human curriculum process, thereby progressing from easy to hard tasks. This adjustment guided the model toward improved convergence optima and generalization. Furthermore, we employed multi-scale contrast learning to enhance the representation of features. Our method capitalizes on the features extracted from multiple layers to explore additional semantic information and point-wise representations. To evaluate the effectiveness of our proposed approach, we conducted experiments on the Kvasir-SEG polyp dataset and the ISIC 2018 skin lesion dataset. The experimental results demonstrated that our method surpassed state-of-the-art semi-supervised methods by achieving a 9.2% increase in the mean intersection over union (mIoU) for the Kvasir-SEG dataset. This improvement substantiated the efficacy of our proposed curriculum consistency constraint and multi-scale contrastive loss.

Deep learning, data ramping, and uncertainty estimation for detecting artifacts in large, imbalanced databases of MRI images

Magnetic resonance imaging (MRI) is increasingly being used to delineate morphological changes underlying neurological disorders. Successfully detecting these changes depends on the MRI data quality. Unfortunately, image artifacts frequently compromise the MRI utility, making it critical to screen the data. Currently, quality assessment requires visual inspection, a time-consuming process that suffers from inter-rater variability. Automated methods to detect MRI artifacts could improve the efficiency of the process. Such automated methods have achieved high accuracy using small datasets, with balanced proportions of MRI data with and without artifacts. With the current trend towards big data in neuroimaging, there is a need for automated methods that achieve accurate detection in large and imbalanced datasets. Deep learning (DL) is the ideal MRI artifact detection algorithm for large neuroimaging databases. However, the inference generated by DL does not commonly include a measure of uncertainty. Here, we present the first stochastic DL algorithm to generate automated, high-performing MRI artifact detection implemented on a large and imbalanced neuroimaging database. We implemented Monte Carlo dropout in a 3D AlexNet to generate probabilities and epistemic uncertainties. We then developed a method to handle class imbalance, namely data-ramping to transfer the learning by extending the dataset size and the proportion of the artifact-free data instances. We used a 34,800 scans (98% clean) dataset. At baseline, we obtained 89.3% testing accuracy (F1 = 0.230). Following the transfer learning (with data-ramping), we obtained 94.9% testing accuracy (F1 = 0.357) outperforming focal cross-entropy (92.9% testing accuracy, F1 = 0.304) incorporated for comparison at handling class imbalance. By implementing epistemic uncertainties, we improved the testing accuracy to 99.5% (F1 = 0.834), outperforming the results obtained in previous comparable studies. In addition, we estimated aleatoric uncertainties by incorporating random flips to the MRI volumes, and demonstrated that aleatoric uncertainty can be implemented as part of the pipeline. The methods we introduce enhance the efficiency of managing large databases and the exclusion of artifact images from big data analyses.

Latent Transformer Models for out-of-distribution detection

Any clinically-deployed image-processing pipeline must be robust to the full range of inputs it may be presented with. One popular approach to this challenge is to develop predictive models that can provide a measure of their uncertainty. Another approach is to use generative modelling to quantify the likelihood of inputs. Inputs with a low enough likelihood are deemed to be out-of-distribution and are not presented to the downstream predictive model. In this work, we evaluate several approaches to segmentation with uncertainty for the task of segmenting bleeds in 3D CT of the head. We show that these models can fail catastrophically when operating in the far out-of-distribution domain, often providing predictions that are both highly confident and wrong. We propose to instead perform out-of-distribution detection using the Latent Transformer Model: a VQ-GAN is used to provide a highly compressed latent representation of the input volume, and a transformer is then used to estimate the likelihood of this compressed representation of the input. We demonstrate this approach can identify images that are both far- and near- out-of-distribution, as well as provide spatial maps that highlight the regions considered to be out-of-distribution. Furthermore, we find a strong relationship between an image's likelihood and the quality of a model's segmentation on it, demonstrating that this approach is viable for filtering out unsuitable images.

Uncertainty-guided transformer for brain tumor segmentation

Multi-model data can enhance brain tumor segmentation for the rich information it provides. However, it also introduces some redundant information that interferes with the segmentation estimation, as some modalities may catch features irrelevant to the tissue of interest. Besides, the ambiguous boundaries and irregulate shapes of different grade tumors lead to a non-confidence estimate of segmentation quality. Given these concerns, we exploit an uncertainty-guided U-shaped transformer with multiple heads to construct drop-out format masks for robust training. Specifically, our drop-out masks are composed of boundary mask, prior probability mask, and conditional probability mask, which can help our approach focus more on uncertainty regions. Extensive experimental results show that our method achieves comparable or higher results than previous state-of-the-art brain tumor segmentation methods, achieving average dice coefficients of [Formula: see text] and Hausdorff distance of 4.91 on the BraTS2021 dataset. Our code is freely available at https://github.com/chaineypung/BTS-UGT.

Uncertainty-inspired open set learning for retinal anomaly identification

Failure to recognize samples from the classes unseen during training is a major limitation of artificial intelligence in the real-world implementation for recognition and classification of retinal anomalies. We establish an uncertainty-inspired open set (UIOS) model, which is trained with fundus images of 9 retinal conditions. Besides assessing the probability of each category, UIOS also calculates an uncertainty score to express its confidence. Our UIOS model with thresholding strategy achieves an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set, external target categories (TC)-JSIEC dataset and TC-unseen testing set, respectively, compared to the F1 score of 92.20%, 80.69% and 64.74% by the standard AI model. Furthermore, UIOS correctly predicts high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images. UIOS provides a robust method for real-world screening of retinal anomalies.

Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013-2023)

Uncertainty estimation in healthcare involves quantifying and understanding the inherent uncertainty or variability associated with medical predictions, diagnoses, and treatment outcomes. In this era of Artificial Intelligence (AI) models, uncertainty estimation becomes vital to ensure safe decision-making in the medical field. Therefore, this review focuses on the application of uncertainty techniques to machine and deep learning models in healthcare. A systematic literature review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our analysis revealed that Bayesian methods were the predominant technique for uncertainty quantification in machine learning models, with Fuzzy systems being the second most used approach. Regarding deep learning models, Bayesian methods emerged as the most prevalent approach, finding application in nearly all aspects of medical imaging. Most of the studies reported in this paper focused on medical images, highlighting the prevalent application of uncertainty quantification techniques using deep learning models compared to machine learning models. Interestingly, we observed a scarcity of studies applying uncertainty quantification to physiological signals. Thus, future research on uncertainty quantification should prioritize investigating the application of these techniques to physiological signals. Overall, our review highlights the significance of integrating uncertainty techniques in healthcare applications of machine learning and deep learning models. This can provide valuable insights and practical solutions to manage uncertainty in real-world medical data, ultimately improving the accuracy and reliability of medical diagnoses and treatment recommendations.

Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise

In the past years, deep learning has seen an increase in usage in the domain of histopathological applications. However, while these approaches have shown great potential, in high-risk environments deep learning models need to be able to judge their uncertainty and be able to reject inputs when there is a significant chance of misclassification. In this work, we conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images, with a focus on the task of selective classification, where the model should reject the classification in situations in which it is uncertain. We conduct our experiments on tile-level under the aspects of domain shift and label noise, as well as on slide-level. In our experiments, we compare Deep Ensembles, Monte-Carlo Dropout, Stochastic Variational Inference, Test-Time Data Augmentation as well as ensembles of the latter approaches. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise, while contrary to results from classical computer vision benchmarks no systematic gain of the other methods can be shown. Across methods, a rejection of the most uncertain samples reliably leads to a significant increase in classification accuracy on both in-distribution as well as out-of-distribution data. Furthermore, we conduct experiments comparing these methods under varying conditions of label noise. Lastly, we publish our code framework to facilitate further research on uncertainty estimation on histopathological data.

Few-shot out-of-distribution detection for automated screening in retinal OCT images using deep learning

Deep neural networks have been increasingly proposed for automated screening and diagnosis of retinal diseases from optical coherence tomography (OCT), but often provide high-confidence predictions on out-of-distribution (OOD) cases, compromising their clinical usage. With this in mind, we performed an in-depth comparative analysis of the state-of-the-art uncertainty estimation methods for OOD detection in retinal OCT imaging. The analysis was performed within the use-case of automated screening and staging of age-related macular degeneration (AMD), one of the leading causes of blindness worldwide, where we achieved a macro-average area under the curve (AUC) of 0.981 for AMD classification. We focus on a few-shot Outlier Exposure (OE) method and the detection of near-OOD cases that share pathomorphological characteristics with the inlier AMD classes. Scoring the OOD case based on the Cosine distance in the feature space from the penultimate network layer proved to be a robust approach for OOD detection, especially in combination with the OE. Using Cosine distance and only 8 outliers exposed per class, we were able to improve the near-OOD detection performance of the OE with Reject Bucket method by [Formula: see text] 10% compared to without OE, reaching an AUC of 0.937. The Cosine distance served as a robust metric for OOD detection of both known and unknown classes and should thus be considered as an alternative to the reject bucket class probability in OE approaches, especially in the few-shot scenario. The inclusion of these methodologies did not come at the expense of classification performance, and can substantially improve the reliability and trustworthiness of the resulting deep learning-based diagnostic systems in the context of retinal OCT.

Learning and predicting the unknown class using evidential deep learning

In practical deep-learning applications, such as medical image analysis, autonomous driving, and traffic simulation, the uncertainty of a classification model's output is critical. Evidential deep learning (EDL) can output this uncertainty for the prediction; however, its accuracy depends on a user-defined threshold, and it cannot handle training data with unknown classes that are unexpectedly contaminated or deliberately mixed for better classification of unknown class. To address these limitations, I propose a classification method called modified-EDL that extends classical EDL such that it outputs a prediction, i.e. an input belongs to a collective unknown class along with a probability. Although other methods handle unknown classes by creating new unknown classes and attempting to learn each class efficiently, the proposed m-EDL outputs, in a natural way, the "uncertainty of the prediction" of classical EDL and uses the output as the probability of an unknown class. Although classical EDL can also classify both known and unknown classes, experiments on three datasets from different domains demonstrated that m-EDL outperformed EDL on known classes when there were instances of unknown classes. Moreover, extensive experiments under different conditions established that m-EDL can predict unknown classes even when the unknown classes in the training and test data have different properties. If unknown class data are to be mixed intentionally during training to increase the discrimination accuracy of unknown classes, it is necessary to mix such data that the characteristics of the mixed data are as close as possible to those of known class data. This ability extends the range of practical applications that can benefit from deep learning-based classification and prediction models.

Bayesian uncertainty estimation for detection of long-tailed and unseen conditions in medical images

Purpose

Deep supervised learning provides an effective approach for developing robust models for various computer-aided diagnosis tasks. However, there is often an underlying assumption that the frequencies of the samples between the different classes of the training dataset are either similar or balanced. In real-world medical data, the samples of positive classes often occur too infrequently to satisfy this assumption. Thus, there is an unmet need for deep-learning systems that can automatically identify and adapt to the real-world conditions of imbalanced data.

Approach

We propose a deep Bayesian ensemble learning framework to address the representation learning problem of long-tailed and out-of-distribution (OOD) samples when training from medical images. By estimating the relative uncertainties of the input data, our framework can adapt to imbalanced data for learning generalizable classifiers. We trained and tested our framework on four public medical imaging datasets with various imbalance ratios and imaging modalities across three different learning tasks: semantic medical image segmentation, OOD detection, and in-domain generalization. We compared the performance of our framework with those of state-of-the-art comparator methods.

Results

Our proposed framework outperformed the comparator models significantly across all performance metrics (pairwise t -test: p < 0.01 ) in the semantic segmentation of high-resolution CT and MR images as well as in the detection of OOD samples (p < 0.01 ), thereby showing significant improvement in handling the associated long-tailed data distribution. The results of the in-domain generalization also indicated that our framework can enhance the prediction of retinal glaucoma, contributing to clinical decision-making processes.

Conclusions

Training of the proposed deep Bayesian ensemble learning framework with dynamic Monte-Carlo dropout and a combination of losses yielded the best generalization to unseen samples from imbalanced medical imaging datasets across different learning tasks.

Uncertainty quantification and attention-aware fusion guided multi-modal MR brain tumor segmentation

Brain tumor is one of the most aggressive cancers in the world, accurate brain tumor segmentation plays a critical role in clinical diagnosis and treatment planning. Although deep learning models have presented remarkable success in medical segmentation, they can only obtain the segmentation map without capturing the segmentation uncertainty. To achieve accurate and safe clinical results, it is necessary to produce extra uncertainty maps to assist the subsequent segmentation revision. To this end, we propose to exploit the uncertainty quantification in the deep learning model and apply it to multi-modal brain tumor segmentation. In addition, we develop an effective attention-aware multi-modal fusion method to learn the complimentary feature information from the multiple MR modalities. First, a multi-encoder-based 3D U-Net is proposed to obtain the initial segmentation results. Then, an estimated Bayesian model is presented to measure the uncertainty of the initial segmentation results. Finally, the obtained uncertainty maps are integrated into a deep learning-based segmentation network, serving as an additional constraint information to further refine the segmentation results. The proposed network is evaluated on publicly available BraTS 2018 and BraTS 2019 datasets. The experimental results demonstrate that the proposed method outperforms the previous state-of-the-art methods on Dice score, Hausdorff distance and Sensitivity metrics. Furthermore, the proposed components could be easily applied to other network architectures and other computer vision fields.

A universal lesion detection method based on partially supervised learning

Partially supervised learning (PSL) is urgently necessary to explore to construct an efficient universal lesion detection (ULD) segmentation model. An annotated dataset is crucial but hard to acquire because of too many Computed tomography (CT) images and the lack of professionals in computer-aided detection/diagnosis (CADe/CADx). To address this problem, we propose a novel loss function to reduce the proportion of negative anchors which is extremely likely to classify the lesion area (positive samples) as a negative bounding box, further leading to an unexpected performance. Before calculating loss, we generate a mask to intentionally choose fewer negative anchors which will backward wrongful loss to the network. During the process of loss calculation, we set a parameter to reduce the proportion of negative samples, and it significantly reduces the adverse effect of misclassification on the model. Our experiments are implemented in a 3D framework by feeding a partially annotated dataset named DeepLesion, a large-scale public dataset for universal lesion detection from CT. We implement a lot of experiments to choose the most suitable parameter, and the result shows that the proposed method has greatly improved the performance of a ULD detector. Our code can be obtained at https://github.com/PLuld0/PLuldl.

Bayesian statistical modeling to predict observer-specific optimal windowing parameters in magnetic resonance imaging

Magnetic resonance (MR) images require a process known as windowing for optimizing the display conditions. However, the conventional windowing process often fails to achieve the preferred display conditions for observers due to various factors. This study proposes a novel framework for predicting the preferred windowing parameters for each observer using Bayesian statistical modeling. MR images obtained from 1000 patients were divided into training and test sets at a 7:3 ratio. The image intensity and windowing parameters were standardized using previously reported methods. Bayesian statistical modeling was utilized to predict the windowing parameters preferred by three MR imaging (MRI) operators. The performance of the proposed framework was evaluated by assessing the mean relative error (MRE), mean absolute error (MAE), and Pearson's correlation coefficient (ρ) of the test set. In addition, the naive method, which presumes that the average value of the windowing parameters for each acquisition sequence and body region in the training set is optimal, was also used for comparison. Three MRI operators and three radiologists conducted visual assessments. The mean MRE, MAE, and ρ values for the window level and width (WL/WW) in the proposed framework were 12.6 and 13.9, 42.9 and 85.4, and 0.98 and 0.98, respectively. These results outperformed those obtained using the naive method. The visual assessments revealed no significant differences between the original and predicted display conditions, indicating that the proposed framework accurately predicts individualized windowing parameters with the additional advantages of robustness and ease of use. Thus, the proposed framework can effectively predict the windowing parameters preferred by each observer.

A convolutional neural network with self-attention for fully automated metabolic tumor volume delineation of head and neck cancer in [Formula: see text]F]FDG PET/CT

PURPOSE

PET-derived metabolic tumor volume (MTV) and total lesion glycolysis of the primary tumor are known to be prognostic of clinical outcome in head and neck cancer (HNC). Including evaluation of lymph node metastases can further increase the prognostic value of PET but accurate manual delineation and classification of all lesions is time-consuming and prone to interobserver variability. Our goal, therefore, was development and evaluation of an automated tool for MTV delineation/classification of primary tumor and lymph node metastases in PET/CT investigations of HNC patients.

METHODS

Automated lesion delineation was performed with a residual 3D U-Net convolutional neural network (CNN) incorporating a multi-head self-attention block. 698 [Formula: see text]F]FDG PET/CT scans from 3 different sites and 5 public databases were used for network training and testing. An external dataset of 181 [Formula: see text]F]FDG PET/CT scans from 2 additional sites was employed to assess the generalizability of the network. In these data, primary tumor and lymph node (LN) metastases were interactively delineated and labeled by two experienced physicians. Performance of the trained network models was assessed by 5-fold cross-validation in the main dataset and by pooling results from the 5 developed models in the external dataset. The Dice similarity coefficient (DSC) for individual delineation tasks and the primary tumor/metastasis classification accuracy were used as evaluation metrics. Additionally, a survival analysis using univariate Cox regression was performed comparing achieved group separation for manual and automated delineation, respectively.

RESULTS

In the cross-validation experiment, delineation of all malignant lesions with the trained U-Net models achieves DSC of 0.885, 0.805, and 0.870 for primary tumor, LN metastases, and the union of both, respectively. In external testing, the DSC reaches 0.850, 0.724, and 0.823 for primary tumor, LN metastases, and the union of both, respectively. The voxel classification accuracy was 98.0% and 97.9% in cross-validation and external data, respectively. Univariate Cox analysis in the cross-validation and the external testing reveals that manually and automatically derived total MTVs are both highly prognostic with respect to overall survival, yielding essentially identical hazard ratios (HR) ([Formula: see text]; [Formula: see text] vs. [Formula: see text]; [Formula: see text] in cross-validation and [Formula: see text]; [Formula: see text] vs. [Formula: see text]; [Formula: see text] in external testing).

CONCLUSION

To the best of our knowledge, this work presents the first CNN model for successful MTV delineation and lesion classification in HNC. In the vast majority of patients, the network performs satisfactory delineation and classification of primary tumor and lymph node metastases and only rarely requires more than minimal manual correction. It is thus able to massively facilitate study data evaluation in large patient groups and also does have clear potential for supervised clinical application.

Multiple sclerosis lesion segmentation: revisiting weighting mechanisms for federated learning

Background and introduction

Federated learning (FL) has been widely employed for medical image analysis to facilitate multi-client collaborative learning without sharing raw data. Despite great success, FL's applications remain suboptimal in neuroimage analysis tasks such as lesion segmentation in multiple sclerosis (MS), due to variance in lesion characteristics imparted by different scanners and acquisition parameters.

Methods

In this work, we propose the first FL MS lesion segmentation framework via two effective re-weighting mechanisms. Specifically, a learnable weight is assigned to each local node during the aggregation process, based on its segmentation performance. In addition, the segmentation loss function in each client is also re-weighted according to the lesion volume for the data during training.

Results

The proposed method has been validated on two FL MS segmentation scenarios using public and clinical datasets. Specifically, the case-wise and voxel-wise Dice score of the proposed method under the first public dataset is 65.20 and 74.30, respectively. On the second in-house dataset, the case-wise and voxel-wise Dice score is 53.66, and 62.31, respectively.

Discussions and conclusions

The Comparison experiments on two FL MS segmentation scenarios using public and clinical datasets have demonstrated the effectiveness of the proposed method by significantly outperforming other FL methods. Furthermore, the segmentation performance of FL incorporating our proposed aggregation mechanism can achieve comparable performance to that from centralized training with all the raw data.

Uncertainty Estimation for Heatmap-Based Landmark Localization

Automatic anatomical landmark localization has made great strides by leveraging deep learning methods in recent years. The ability to quantify the uncertainty of these predictions is a vital component needed for these methods to be adopted in clinical settings, where it is imperative that erroneous predictions are caught and corrected. We propose Quantile Binning, a data-driven method to categorize predictions by uncertainty with estimated error bounds. Our framework can be applied to any continuous uncertainty measure, allowing straightforward identification of the best subset of predictions with accompanying estimated error bounds. We facilitate easy comparison between uncertainty measures by constructing two evaluation metrics derived from Quantile Binning. We compare and contrast three epistemic uncertainty measures (two baselines, and a proposed method combining aspects of the two), derived from two heatmap-based landmark localization model paradigms (U-Net and patch-based). We show results across three datasets, including a publicly available Cephalometric dataset. We illustrate how filtering out gross mispredictions caught in our Quantile Bins significantly improves the proportion of predictions under an acceptable error threshold. Finally, we demonstrate that Quantile Binning remains effective on landmarks with high aleatoric uncertainty caused by inherent landmark ambiguity, and offer recommendations on which uncertainty measure to use and how to use it. The code and data are available at https://github.com/schobs/qbin.

Uncertainty Estimation in Unsupervised MR-CT Synthesis of Scoliotic Spines

Uncertainty estimations through approximate Bayesian inference provide interesting insights to deep neural networks' behavior. In unsupervised learning tasks, where expert labels are unavailable, it becomes ever more important to critique the model through uncertainties. This paper presents a proof-of-concept for generalizing the aleatoric and epistemic uncertainties in unsupervised MR-CT synthesis of scoliotic spines. A novel adaptation of the cycle-consistency constraint in CycleGAN is proposed such that the model predicts the aleatoric uncertainty maps in addition to the standard volume-to-volume translation between Magnetic Resonance (MR) and Computed Tomography (CT) data. Ablation experiments were performed to understand uncertainty estimation as an implicit regularizer and a measure of the model's confidence. The aleatoric uncertainty helps in distinguishing between the bone and soft-tissue regions in CT and MR data during translation, while the epistemic uncertainty provides interpretable information to the user for downstream tasks.

CarveMix: A simple data augmentation method for brain lesion segmentation

Brain lesion segmentation provides a valuable tool for clinical diagnosis and research, and convolutional neural networks (CNNs) have achieved unprecedented success in the segmentation task. Data augmentation is a widely used strategy to improve the training of CNNs. In particular, data augmentation approaches that mix pairs of annotated training images have been developed. These methods are easy to implement and have achieved promising results in various image processing tasks. However, existing data augmentation approaches based on image mixing are not designed for brain lesions and may not perform well for brain lesion segmentation. Thus, the design of this type of simple data augmentation method for brain lesion segmentation is still an open problem. In this work, we propose a simple yet effective data augmentation approach, dubbed as CarveMix, for CNN-based brain lesion segmentation. Like other mixing-based methods, CarveMix stochastically combines two existing annotated images (annotated for brain lesions only) to obtain new labeled samples. To make our method more suitable for brain lesion segmentation, CarveMix is lesion-aware, where the image combination is performed with a focus on the lesions and preserves the lesion information. Specifically, from one annotated image we carve a region of interest (ROI) according to the lesion location and geometry with a variable ROI size. The carved ROI then replaces the corresponding voxels in a second annotated image to synthesize new labeled images for network training, and additional harmonization steps are applied for heterogeneous data where the two annotated images can originate from different sources. Besides, we further propose to model the mass effect that is unique to whole brain tumor segmentation during image mixing. To evaluate the proposed method, experiments were performed on multiple publicly available or private datasets, and the results show that our method improves the accuracy of brain lesion segmentation. The code of the proposed method is available at https://github.com/ZhangxinruBIT/CarveMix.git.

Automated detection of schizophrenia using deep learning: a review for the last decade

Schizophrenia (SZ) is a devastating mental disorder that disrupts higher brain functions like thought, perception, etc., with a profound impact on the individual's life. Deep learning (DL) can detect SZ automatically by learning signal data characteristics hierarchically without the need for feature engineering associated with traditional machine learning. We performed a systematic review of DL models for SZ detection. Various deep models like long short-term memory, convolution neural networks, AlexNet, etc., and composite methods have been published based on electroencephalographic signals, and structural and/or functional magnetic resonance imaging acquired from SZ patients and healthy patients control subjects in diverse public and private datasets. The studies, the study datasets, and model methodologies are reported in detail. In addition, the challenges of DL models for SZ diagnosis and future works are discussed.

Semi-Supervised Dual Stream Segmentation Network for Fundus Lesion Segmentation

Accurate segmentation of retinal images can assist ophthalmologists to determine the degree of retinopathy and diagnose other systemic diseases. However, the structure of the retina is complex, and different anatomical structures often affect the segmentation of fundus lesions. In this paper, a new segmentation strategy called a dual stream segmentation network embedded into a conditional generative adversarial network is proposed to improve the accuracy of retinal lesion segmentation. First, a dual stream encoder is proposed to utilize the capabilities of two different networks and extract more feature information. Second, a multiple level fuse block is proposed to decode the richer and more effective features from the two different parallel encoders. Third, the proposed network is further trained in a semi-supervised adversarial manner to leverage from labeled images and unlabeled images with high confident pseudo labels, which are selected by the dual stream Bayesian segmentation network. An annotation discriminator is further proposed to reduce the negativity that prediction tends to become increasingly similar to the inaccurate predictions of unlabeled images. The proposed method is cross-validated in 384 clinical fundus fluorescein angiography images and 1040 optical coherence tomography images. Compared to state-of-the-art methods, the proposed method can achieve better segmentation of retinal capillary non-perfusion region and choroidal neovascularization.

Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation

Supervised deep learning-based methods yield accurate results for medical image segmentation. However, they require large labeled datasets for this, and obtaining them is a laborious task that requires clinical expertise. Semi/self-supervised learning-based approaches address this limitation by exploiting unlabeled data along with limited annotated data. Recent self-supervised learning methods use contrastive loss to learn good global level representations from unlabeled images and achieve high performance in classification tasks on popular natural image datasets like ImageNet. In pixel-level prediction tasks such as segmentation, it is crucial to also learn good local level representations along with global representations to achieve better accuracy. However, the impact of the existing local contrastive loss-based methods remains limited for learning good local representations because similar and dissimilar local regions are defined based on random augmentations and spatial proximity; not based on the semantic label of local regions due to lack of large-scale expert annotations in the semi/self-supervised setting. In this paper, we propose a local contrastive loss to learn good pixel level features useful for segmentation by exploiting semantic label information obtained from pseudo-labels of unlabeled images alongside limited annotated images with ground truth (GT) labels. In particular, we define the proposed contrastive loss to encourage similar representations for the pixels that have the same pseudo-label/GT label while being dissimilar to the representation of pixels with different pseudo-label/GT label in the dataset. We perform pseudo-label based self-training and train the network by jointly optimizing the proposed contrastive loss on both labeled and unlabeled sets and segmentation loss on only the limited labeled set. We evaluated the proposed approach on three public medical datasets of cardiac and prostate anatomies, and obtain high segmentation performance with a limited labeled set of one or two 3D volumes. Extensive comparisons with the state-of-the-art semi-supervised and data augmentation methods and concurrent contrastive learning methods demonstrate the substantial improvement achieved by the proposed method. The code is made publicly available at https://github.com/krishnabits001/pseudo_label_contrastive_training.

Deep learning aided oropharyngeal cancer segmentation with adaptive thresholding for predicted tumor probability in FDG PET and CT images

Objective. Tumor segmentation is a fundamental step for radiotherapy treatment planning. To define an accurate segmentation of the primary tumor (GTVp) of oropharyngeal cancer patients (OPC) each image volume is explored slice-by-slice from different orientations on different image modalities. However, the manual fixed boundary of segmentation neglects the spatial uncertainty known to occur in tumor delineation. This study proposes a novel deep learning-based method that generates probability maps which capture the model uncertainty in the segmentation task.Approach. We included 138 OPC patients treated with (chemo)radiation in our institute. Sequences of 3 consecutive 2D slices of concatenated FDG-PET/CT images and GTVp contours were used as input. Our framework exploits inter and intra-slice context using attention mechanisms and bi-directional long short term memory (Bi-LSTM). Each slice resulted in three predictions that were averaged. A 3-fold cross validation was performed on sequences extracted from the axial, sagittal, and coronal plane. 3D volumes were reconstructed and single- and multi-view ensembling were performed to obtain final results. The output is a tumor probability map determined by averaging multiple predictions.Main Results. Model performance was assessed on 25 patients at different probability thresholds. Predictions were the closest to the GTVp at a threshold of 0.9 (mean surface DSC of 0.81, median HD₉₅of 3.906 mm).Significance. The promising results of the proposed method show that is it possible to offer the probability maps to radiation oncologists to guide them in a in a slice-by-slice adaptive GTVp segmentation.

3D pyramidal densely connected network with cross-frame uncertainty guidance for intravascular ultrasound sequence segmentation

Objective. Automatic extraction of external elastic membrane border (EEM) and lumen-intima border (LIB) in intravascular ultrasound (IVUS) sequences aids atherosclerosis diagnosis. Existing IVUS segmentation networks ignored longitudinal relations among sequential images and neglected that IVUS images of different vascular conditions vary largely in intricacy and informativeness. As a result, they suffered from performance degradation in complicated parts in IVUS sequences.Approach. In this paper, we develop a 3D Pyramidal Densely-connected Network (PDN) with Adaptive learning and post-Correction guided by a novel cross-frame uncertainty (CFU). The proposed method is named PDN-AC. Specifically, the PDN enables the longitudinal information exploitation and the effective perception of size-varied vessel regions in IVUS samples, by pyramidally connecting multi-scale 3D dilated convolutions. Additionally, the CFU enhances the robustness of the method to complicated pathology from the frame-level (f-CFU) and pixel-level (p-CFU) via exploiting cross-frame knowledge in IVUS sequences. The f-CFU weighs the complexity of IVUS frames and steers an adaptive sampling during the PDN training. The p-CFU visualizes uncertain pixels probably misclassified by the PDN and guides an active contour-based post-correction.Main results. Human and animal experiments were conducted on IVUS datasets acquired from atherosclerosis patients and pigs. Results showed that the f-CFU weighted adaptive sampling reduced the Hausdorff distance (HD) by 10.53%/7.69% in EEM/LIB detection. Improvements achieved by the p-CFU guided post-correction were 2.94%/5.56%.Significance. The PDN-AC attained mean Jaccard values of 0.90/0.87 and HD values of 0.33/0.34 mm in EEM/LIB detection, preferable to state-of-the-art IVUS segmentation methods.

Automatic uncertainty-based quality controlled T1 mapping and ECV analysis from native and post-contrast cardiac T1 mapping images using Bayesian vision transformer

Deep learning-based methods for cardiac MR segmentation have achieved state-of-the-art results. However, these methods can generate incorrect segmentation results which can lead to wrong clinical decisions in the downstream tasks. Automatic and accurate analysis of downstream tasks, such as myocardial tissue characterization, is highly dependent on the quality of the segmentation results. Therefore, it is of paramount importance to use quality control methods to detect the failed segmentations before further analysis. In this work, we propose a fully automatic uncertainty-based quality control framework for T1 mapping and extracellular volume (ECV) analysis. The framework consists of three parts. The first one focuses on segmentation of cardiac structures from a native and post-contrast T1 mapping dataset (n=295) using a Bayesian Swin transformer-based U-Net. In the second part, we propose a novel uncertainty-based quality control (QC) to detect inaccurate segmentation results. The QC method utilizes image-level uncertainty features as input to a random forest-based classifier/regressor to determine the quality of the segmentation outputs. The experimental results from four different types of segmentation results show that the proposed QC method achieves a mean area under the ROC curve (AUC) of 0.927 on binary classification and a mean absolute error (MAE) of 0.021 on Dice score regression, significantly outperforming other state-of-the-art uncertainty based QC methods. The performance gap is notably higher in predicting the segmentation quality from poor-performing models which shows the robustness of our method in detecting failed segmentations. After the inaccurate segmentation results are detected and rejected by the QC method, in the third part, T1 mapping and ECV values are computed automatically to characterize the myocardial tissues of healthy and cardiac pathological cases. The native myocardial T1 and ECV values computed from automatic and manual segmentations show an excellent agreement yielding Pearson coefficients of 0.990 and 0.975 (on the combined validation and test sets), respectively. From the results, we observe that the automatically computed myocardial T1 and ECV values have the ability to characterize myocardial tissues of healthy and cardiac diseases like myocardial infarction, amyloidosis, Tako-Tsubo syndrome, dilated cardiomyopathy, and hypertrophic cardiomyopathy.

Deep Learning Applied to Intracranial Hemorrhage Detection

Intracranial hemorrhage is a serious medical problem that requires rapid and often intensive medical care. Identifying the location and type of any hemorrhage present is a critical step in the treatment of the patient. Detection of, and diagnosis of, a hemorrhage that requires an urgent procedure is a difficult and time-consuming process for human experts. In this paper, we propose methods based on EfficientDet's deep-learning technology that can be applied to the diagnosis of hemorrhages at a patient level and which could, thus, become a decision-support system. Our proposal is two-fold. On the one hand, the proposed technique classifies slices of computed tomography scans for the presence of hemorrhage or its lack of, and evaluates whether the patient is positive in terms of hemorrhage, and achieving, in this regard, 92.7% accuracy and 0.978 ROC AUC. On the other hand, our methodology provides visual explanations of the chosen classification using the Grad-CAM methodology.

Segmentation ability map: Interpret deep features for medical image segmentation

Deep convolutional neural networks (CNNs) have been widely used for medical image segmentation. In most studies, only the output layer is exploited to compute the final segmentation results and the hidden representations of the deep learned features have not been well understood. In this paper, we propose a prototype segmentation (ProtoSeg) method to compute a binary segmentation map based on deep features. We measure the segmentation abilities of the features by computing the Dice between the feature segmentation map and ground-truth, named as the segmentation ability score (SA score for short). The corresponding SA score can quantify the segmentation abilities of deep features in different layers and units to understand the deep neural networks for segmentation. In addition, our method can provide a mean SA score which can give a performance estimation of the output on the test images without ground-truth. Finally, we use the proposed ProtoSeg method to compute the segmentation map directly on input images to further understand the segmentation ability of each input image. Results are presented on segmenting tumors in brain MRI, lesions in skin images, COVID-related abnormality in CT images, prostate segmentation in abdominal MRI, and pancreatic mass segmentation in CT images. Our method can provide new insights for interpreting and explainable AI systems for medical image segmentation. Our code is available on: https://github.com/shengfly/ProtoSeg.

A survey on deep learning applied to medical images: from simple artificial neural networks to generative models

Deep learning techniques, in particular generative models, have taken on great importance in medical image analysis. This paper surveys fundamental deep learning concepts related to medical image generation. It provides concise overviews of studies which use some of the latest state-of-the-art models from last years applied to medical images of different injured body areas or organs that have a disease associated with (e.g., brain tumor and COVID-19 lungs pneumonia). The motivation for this study is to offer a comprehensive overview of artificial neural networks (NNs) and deep generative models in medical imaging, so more groups and authors that are not familiar with deep learning take into consideration its use in medicine works. We review the use of generative models, such as generative adversarial networks and variational autoencoders, as techniques to achieve semantic segmentation, data augmentation, and better classification algorithms, among other purposes. In addition, a collection of widely used public medical datasets containing magnetic resonance (MR) images, computed tomography (CT) scans, and common pictures is presented. Finally, we feature a summary of the current state of generative models in medical image including key features, current challenges, and future research paths.

Predictive uncertainty estimation for out-of-distribution detection in digital pathology

Machine learning model deployment in clinical practice demands real-time risk assessment to identify situations in which the model is uncertain. Once deployed, models should be accurate for classes seen during training while providing informative estimates of uncertainty to flag abnormalities and unseen classes for further analysis. Although recent developments in uncertainty estimation have resulted in an increasing number of methods, a rigorous empirical evaluation of their performance on large-scale digital pathology datasets is lacking. This work provides a benchmark for evaluating prevalent methods on multiple datasets by comparing the uncertainty estimates on both in-distribution and realistic near and far out-of-distribution (OOD) data on a whole-slide level. To this end, we aggregate uncertainty values from patch-based classifiers to whole-slide level uncertainty scores. We show that results found in classical computer vision benchmarks do not always translate to the medical imaging setting. Specifically, we demonstrate that deep ensembles perform best at detecting far-OOD data but can be outperformed on a more challenging near-OOD detection task by multi-head ensembles trained for optimal ensemble diversity. Furthermore, we demonstrate the harmful impact OOD data can have on the performance of deployed machine learning models. Overall, we show that uncertainty estimates can be used to discriminate in-distribution from OOD data with high AUC scores. Still, model deployment might require careful tuning based on prior knowledge of prospective OOD data.

Quantifying deep neural network uncertainty for atrial fibrillation detection with limited labels

Atrial fibrillation (AF) is the most common arrhythmia found in the intensive care unit (ICU), and is associated with many adverse outcomes. Effective handling of AF and similar arrhythmias is a vital part of modern critical care, but obtaining knowledge about both disease burden and effective interventions often requires costly clinical trials. A wealth of continuous, high frequency physiological data such as the waveforms derived from electrocardiogram telemetry are promising sources for enriching clinical research. Automated detection using machine learning and in particular deep learning has been explored as a solution for processing these data. However, a lack of labels, increased presence of noise, and inability to assess the quality and trustworthiness of many machine learning model predictions pose challenges to interpretation. In this work, we propose an approach for training deep AF models on limited, noisy data and report uncertainty in their predictions. Using techniques from the fields of weakly supervised learning, we leverage a surrogate model trained on non-ICU data to create imperfect labels for a large ICU telemetry dataset. We combine these weak labels with techniques to estimate model uncertainty without the need for extensive human data annotation. AF detection models trained using this process demonstrated higher classification performance (0.64-0.67 F1 score) and improved calibration (0.05-0.07 expected calibration error).

Region-based evidential deep learning to quantify uncertainty and improve robustness of brain tumor segmentation

Despite recent advances in the accuracy of brain tumor segmentation, the results still suffer from low reliability and robustness. Uncertainty estimation is an efficient solution to this problem, as it provides a measure of confidence in the segmentation results. The current uncertainty estimation methods based on quantile regression, Bayesian neural network, ensemble, and Monte Carlo dropout are limited by their high computational cost and inconsistency. In order to overcome these challenges, Evidential Deep Learning (EDL) was developed in recent work but primarily for natural image classification and showed inferior segmentation results. In this paper, we proposed a region-based EDL segmentation framework that can generate reliable uncertainty maps and accurate segmentation results, which is robust to noise and image corruption. We used the Theory of Evidence to interpret the output of a neural network as evidence values gathered from input features. Following Subjective Logic, evidence was parameterized as a Dirichlet distribution, and predicted probabilities were treated as subjective opinions. To evaluate the performance of our model on segmentation and uncertainty estimation, we conducted quantitative and qualitative experiments on the BraTS 2020 dataset. The results demonstrated the top performance of the proposed method in quantifying segmentation uncertainty and robustly segmenting tumors. Furthermore, our proposed new framework maintained the advantages of low computational cost and easy implementation and showed the potential for clinical application.

Post-operative glioblastoma multiforme segmentation with uncertainty estimation

Segmentation of post-operative glioblastoma multiforme (GBM) is essential for the planning of Tumor Treating Fields (TTFields) treatment and other clinical applications. Recent methods developed for pre-operative GBM segmentation perform poorly on post-operative GBM MRI scans. In this paper we present a method for the segmentation of GBM in post-operative patients. Our method incorporates an ensemble of segmentation networks and the Kullback–Leibler divergence agreement score in the objective function to estimate the prediction label uncertainty and cope with noisy labels and inter-observer variability. Moreover, our method integrates the surgery type and computes non-tumorous tissue delineation to automatically segment the tumor. We trained and validated our method on a dataset of 340 enhanced T1 MRI scans of patients that were treated with TTFields (270 scans for train and 70 scans for test). For validation, we developed a tool that uses the uncertainty map along with the segmentation result. Our tool allows visualization and fast editing of the tissues to improve the results dependent on user preference. Three physicians reviewed and graded our segmentation and editing tool on 12 different MRI scans. The validation set average (SD) Dice scores were 0.81 (0.11), 0.71 (0.24), 0.64 (0.25), and 0.68 (0.19) for whole-tumor, resection, necrotic-core, and enhancing-tissue, respectively. The physicians rated 72% of the segmented GBMs acceptable for treatment planning or better. Another 22% can be edited manually in a reasonable time to achieve a clinically acceptable result. According to these results, the proposed method for GBM segmentation can be integrated into TTFields treatment planning software in order to shorten the planning process. To conclude, we have extended a state-of-the-art pre-operative GBM segmentation method with surgery-type, anatomical information, and uncertainty visualization to facilitate a clinically viable segmentation of post-operative GBM for TTFields treatment planning.

Prediction of Lung Nodule Progression with an Uncertainty-Aware Hierarchical Probabilistic Network

Predicting whether a lung nodule will grow, remain stable or regress over time, especially early in its follow-up, would help doctors prescribe personalized treatments and better surgical planning. However, the multifactorial nature of lung tumour progression hampers the identification of growth patterns. In this work, we propose a deep hierarchical generative and probabilistic network that, given an initial image of the nodule, predicts whether it will grow, quantifies its future size and provides its expected semantic appearance at a future time. Unlike previous solutions, our approach also estimates the uncertainty in the predictions from the intrinsic noise in medical images and the inter-observer variability in the annotations. The evaluation of this method on an independent test set reported a future tumour growth size mean absolute error of 1.74 mm, a nodule segmentation Dice’s coefficient of 78% and a tumour growth accuracy of 84% on predictions made up to 24 months ahead. Due to the lack of similar methods for providing future lung tumour growth predictions, along with their associated uncertainty, we adapted equivalent deterministic and alternative generative networks (i.e., probabilistic U-Net, Bayesian test dropout and Pix2Pix). Our method outperformed all these methods, corroborating the adequacy of our approach.

Evaluating the use of synthetic T1-w images in new T2 lesion detection in multiple sclerosis

The assessment of disease activity using serial brain MRI scans is one of the most valuable strategies for monitoring treatment response in patients with multiple sclerosis (MS) receiving disease-modifying treatments. Recently, several deep learning approaches have been proposed to improve this analysis, obtaining a good trade-off between sensitivity and specificity, especially when using T1-w and T2-FLAIR images as inputs. However, the need to acquire two different types of images is time-consuming, costly and not always available in clinical practice. In this paper, we investigate an approach to generate synthetic T1-w images from T2-FLAIR images and subsequently analyse the impact of using original and synthetic T1-w images on the performance of a state-of-the-art approach for longitudinal MS lesion detection. We evaluate our approach on a dataset containing 136 images from MS patients, and 73 images with lesion activity (the appearance of new T2 lesions in follow-up scans). To evaluate the synthesis of the images, we analyse the structural similarity index metric and the median absolute error and obtain consistent results. To study the impact of synthetic T1-w images, we evaluate the performance of the new lesion detection approach when using (1) both T2-FLAIR and T1-w original images, (2) only T2-FLAIR images, and (3) both T2-FLAIR and synthetic T1-w images. Sensitivities of 0.75, 0.63, and 0.81, respectively, were obtained at the same false-positive rate (0.14) for all experiments. In addition, we also present the results obtained when using the data from the international MSSEG-2 challenge, showing also an improvement when including synthetic T1-w images. In conclusion, we show that the use of synthetic images can support the lack of data or even be used instead of the original image to homogenize the contrast of the different acquisitions in new T2 lesions detection algorithms.

Cortical lesions, central vein sign, and paramagnetic rim lesions in multiple sclerosis: Emerging machine learning techniques and future avenues

The current diagnostic criteria for multiple sclerosis (MS) lack specificity, and this may lead to misdiagnosis, which remains an issue in present-day clinical practice. In addition, conventional biomarkers only moderately correlate with MS disease progression. Recently, some MS lesional imaging biomarkers such as cortical lesions (CL), the central vein sign (CVS), and paramagnetic rim lesions (PRL), visible in specialized magnetic resonance imaging (MRI) sequences, have shown higher specificity in differential diagnosis. Moreover, studies have shown that CL and PRL are potential prognostic biomarkers, the former correlating with cognitive impairments and the latter with early disability progression. As machine learning-based methods have achieved extraordinary performance in the assessment of conventional imaging biomarkers, such as white matter lesion segmentation, several automated or semi-automated methods have been proposed as well for CL, PRL, and CVS. In the present review, we first introduce these MS biomarkers and their imaging methods. Subsequently, we describe the corresponding machine learning-based methods that were proposed to tackle these clinical questions, putting them into context with respect to the challenges they are facing, including non-standardized MRI protocols, limited datasets, and moderate inter-rater variability. We conclude by presenting the current limitations that prevent their broader deployment and suggesting future research directions.

Bayesian deep learning outperforms clinical trial estimators of intracerebral and intraventricular hemorrhage volume

BACKGROUND AND PURPOSE

Intracerebral hemorrhage (ICH) and intraventricular hemorrhage (IVH) clinical trials rely on manual linear and semi-quantitative (LSQ) estimators like the ABC/2, modified Graeb and IVH scores for timely volumetric estimation from CT. Deep learning (DL) volumetrics of ICH have recently approached the accuracy of gold-standard planimetry. However, DL and LSQ strategies have been limited by unquantified uncertainty, in particular when ICH and IVH estimates intersect. Bayesian deep learning methods can be used to approximate uncertainty, presenting an opportunity to improve quality assurance in clinical trials.

METHODS

A DL model was trained to simultaneously segment ICH and IVH using diagnostic CT data from the Minimally Invasive Surgery Plus Alteplase for ICH Evacuation (MISTIE) III and Clot Lysis: Evaluating Accelerated Resolution of IVH (CLEAR) III clinical trials. Bayesian uncertainty approximation was performed using Monte-Carlo dropout. We compared the performance of our model with estimators used in the CLEAR IVH and MISTIE II trials. The reliability of planimetry, DL, and LSQ volumetrics in the setting of high ICH and IVH intersection is quantified using consensus estimates.

RESULTS

Our DL model produced volume correlations and median Dice scores of .994 and .946 for ICH in MISTIE II, and .980 and .863 for IVH in CLEAR IVH, respectively, outperforming LSQ estimates from the clinical trials. We found significant linear relationships between ICH uncertainty, Dice scores (r = -.849), and relative volume difference (r = .735).

CONCLUSION

In our validation clinical trial dataset, DL models with Bayesian uncertainty approximation provided superior volumetric estimates to LSQ methods with real-time estimates of model uncertainty.

Segmentation of macular neovascularization and leakage in fluorescein angiography images in neovascular age-related macular degeneration using deep learning

BACKGROUND/OBJECTIVES

We aim to develop an objective fully automated Artificial intelligence (AI) algorithm for MNV lesion size and leakage area segmentation on fluorescein angiography (FA) in patients with neovascular age-related macular degeneration (nAMD).

SUBJECTS/METHODS

Two FA image datasets collected form large prospective multicentre trials consisting of 4710 images from 513 patients and 4558 images from 514 patients were used to develop and evaluate a deep learning-based algorithm to detect CNV lesion size and leakage area automatically. Manual segmentation of was performed by certified FA graders of the Vienna Reading Center. Precision, Recall and F1 score between AI predictions and manual annotations were computed. In addition, two masked retina experts conducted a clinical-applicability evaluation, comparing the quality of AI based and manual segmentations.

RESULTS

For CNV lesion size and leakage area segmentation, we obtained F1 scores of 0.73 and 0.65, respectively. Expert review resulted in a slight preference for the automated segmentations in both datasets. The quality of automated segmentations was slightly more often judged as good compared to manual annotations.

CONCLUSIONS

CNV lesion size and leakage area can be segmented by our automated model at human-level performance, its output being well-accepted during clinical applicability testing. The results provide proof-of-concept that an automated deep learning approach can improve efficacy of objective biomarker analysis in FA images and will be well-suited for clinical application.

Self-normalized density map (SNDM) for counting microbiological objects

The statistical properties of the density map (DM) approach to counting microbiological objects on images are studied in detail. The DM is given by U\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^2$$\end{document}2-Net. Two statistical methods for deep neural networks are utilized: the bootstrap and the Monte Carlo (MC) dropout. The detailed analysis of the uncertainties for the DM predictions leads to a deeper understanding of the DM model’s deficiencies. Based on our investigation, we propose a self-normalization module in the network. The improved network model, called Self-Normalized Density Map (SNDM), can correct its output density map by itself to accurately predict the total number of objects in the image. The SNDM architecture outperforms the original model. Moreover, both statistical frameworks—bootstrap and MC dropout—have consistent statistical results for SNDM, which were not observed in the original model. The SNDM efficiency is comparable with the detector-base models, such as Faster and Cascade R-CNN detectors.

Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency

The interest in machine learning (ML) has grown tremendously in recent years, partly due to the performance leap that occurred with new techniques of deep learning, convolutional neural networks for images, increased computational power, and wider availability of large datasets. Most fields of medicine follow that popular trend and, notably, radiation oncology is one of those that are at the forefront, with already a long tradition in using digital images and fully computerized workflows. ML models are driven by data, and in contrast with many statistical or physical models, they can be very large and complex, with countless generic parameters. This inevitably raises two questions, namely, the tight dependence between the models and the datasets that feed them, and the interpretability of the models, which scales with its complexity. Any problems in the data used to train the model will be later reflected in their performance. This, together with the low interpretability of ML models, makes their implementation into the clinical workflow particularly difficult. Building tools for risk assessment and quality assurance of ML models must involve then two main points: interpretability and data-model dependency. After a joint introduction of both radiation oncology and ML, this paper reviews the main risks and current solutions when applying the latter to workflows in the former. Risks associated with data and models, as well as their interaction, are detailed. Next, the core concepts of interpretability, explainability, and data-model dependency are formally defined and illustrated with examples. Afterwards, a broad discussion goes through key applications of ML in workflows of radiation oncology as well as vendors' perspectives for the clinical implementation of ML.

Generalisation effects of predictive uncertainty estimation in deep learning for digital pathology

Deep learning (DL) has shown great potential in digital pathology applications. The robustness of a diagnostic DL-based solution is essential for safe clinical deployment. In this work we evaluate if adding uncertainty estimates for DL predictions in digital pathology could result in increased value for the clinical applications, by boosting the general predictive performance or by detecting mispredictions. We compare the effectiveness of model-integrated methods (MC dropout and Deep ensembles) with a model-agnostic approach (Test time augmentation, TTA). Moreover, four uncertainty metrics are compared. Our experiments focus on two domain shift scenarios: a shift to a different medical center and to an underrepresented subtype of cancer. Our results show that uncertainty estimates increase reliability by reducing a model’s sensitivity to classification threshold selection as well as by detecting between 70 and 90% of the mispredictions done by the model. Overall, the deep ensembles method achieved the best performance closely followed by TTA.