USE-Evaluator: Performance metrics for medical image segmentation models supervised by uncertain, small or empty reference annotations in neuroimaging

Towards U-Net-based intraoperative 2D dose prediction in high dose rate prostate brachytherapy

BACKGROUND

Poor needle placement in prostate high-dose-rate brachytherapy (HDR-BT) results in sub-optimal dosimetry and mentally predicting these effects during HDR-BT is difficult, creating a barrier to widespread availability of high-quality prostate HDR-BT.

PURPOSE

To provide earlier feedback on needle implantation quality, we trained machine learning models to predict 2D dosimetry for prostate HDR-BT on axial TRUS images.

METHODS AND MATERIALS

Clinical treatment plans from 248 prostate HDR-BT patients were retrospectively collected and randomly split 80/20 for training/testing. Fifteen U-Net models were implemented to predict the 90%, 100%, 120%, 150%, and 200% isodose levels in the prostate base, midgland, and apex. Predicted isodose lines were compared to delivered dose using Dice similarity coefficient (DSC), precision, recall, average symmetric surface distance, area percent difference, and 95th percentile Hausdorff distance. To benchmark performance, 10 cases were retrospectively replanned and compared against the clinical plans using the same metrics.

RESULTS

Models predicting 90% and 100% isodose lines at midgland performed best, with median DSC of 0.97 and 0.96, respectively. Performance declined as isodose level increased, with median DSC of 0.90, 0.79, and 0.65 in the 120%, 150%, and 200% models. In the base, median DSC was 0.94 for 90% and decreased to 0.64 for 200%. In the apex, median DSC was 0.93 for 90% and decreased to 0.63 for 200%. Median prediction time was 25 ms.

CONCLUSION

U-Net models accurately predicted HDR-BT isodose lines on 2D TRUS images sufficiently quickly for real-time use. Incorporating auto-segmentation algorithms will allow intra-operative feedback on needle implantation quality.

Exploring transformer reliability in clinically significant prostate cancer segmentation: A comprehensive in-depth investigation

Despite the growing prominence of transformers in medical image segmentation, their application to clinically significant prostate cancer (csPCa) has been overlooked. Minimal attention has been paid to domain shift analysis and uncertainty assessment, critical for safely implementing computer-aided diagnosis (CAD) systems. Domain shift in medical imagery refers to differences between the data used to train a model and the data evaluated later, arising from variations in imaging equipment, protocols, patient populations, and acquisition noise. While recent models enhance in-domain performance, areas such as robustness and uncertainty estimation in out-of-domain distributions have received limited investigation, creating indecisiveness about model reliability. In contrast, our study addresses csPCa at voxel, lesion, and image levels, investigating models from traditional U-Net to cutting-edge transformers. We focus on four key points: robustness, calibration, out-of-distribution (OOD), and misclassification detection (MD). Findings show that transformer-based models exhibit enhanced robustness at image and lesion levels, both in and out of domain. However, this improvement is not fully translated to the voxel level, where Convolutional Neural Networks (CNNs) outperform in most robustness metrics. Regarding uncertainty, hybrid transformers and transformer encoders performed better, but this trend depends on misclassification or out-of-distribution tasks.

DDANet: A deep dilated attention network for intracerebral haemorrhage segmentation

Intracranial haemorrhage (ICH) is an urgent and potentially fatal medical condition caused by brain blood vessel rupture, leading to blood accumulation in the brain tissue. Due to the pressure and damage it causes to brain tissue, ICH results in severe neurological impairment or even death. Recently, deep neural networks have been widely applied to enhance the speed and precision of ICH detection yet they are still challenged by small or subtle hemorrhages. The authors introduce DDANet, a novel haematoma segmentation model for brain CT images. Specifically, a dilated convolution pooling block is introduced in the intermediate layers of the encoder to enhance feature extraction capabilities of middle layers. Additionally, the authors incorporate a self-attention mechanism to capture global semantic information of high-level features to guide the extraction and processing of low-level features, thereby enhancing the model's understanding of the overall structure while maintaining details. DDANet also integrates residual networks, channel attention, and spatial attention mechanisms for joint optimisation, effectively mitigating the severe class imbalance problem and aiding the training process. Experiments show that DDANet outperforms current methods, achieving the Dice coefficient, Jaccard index, sensitivity, accuracy, and a specificity of 0.712, 0.601, 0.73, 0.997, and 0.998, respectively. The code is available at https://github.com/hpguo1982/DDANet.

Segmentation and multiparametric evaluation of corneal whorl-like nerves for in vivo confocal microscopy images in dry eye disease

OBJECTIVE

To establish an automated corneal nerve analysis system for corneal in vivo confocal microscopy (IVCM) images from both the whorl-like corneal nerves in the inferior whorl (IW) region and the straight ones in the central cornea and to characterise the geometric features of cornea nerves in dry eye disease (DED).

METHODS AND ANALYSIS

An encoder-decoder-based semi-supervised method was proposed for corneal nerve segmentation. This model's performance was compared with the ground truth provided by experienced clinicians, using Dice similarity coefficient (DSC), mean intersection over union (mIoU), accuracy (Acc), sensitivity (Sen) and specificity (Spe). The corneal nerve total length (CNFL), tortuosity (CNTor), fractal dimension (CNDf) and number of branching points (CNBP) were used for further analysis in an independent DED dataset including 50 patients with DED and 30 healthy controls.

RESULTS

The model achieved 95.72% Acc, 97.88% Spe, 80.61% Sen, 75.26% DSC, 77.57% mIoU and an area under the curve value of 0.98. For clinical evaluation, the CNFL, CNBP and CNDf for whorl-like and straight nerves showed a significant decrease in DED patients compared with healthy controls (p<0.05). Additionally, significantly elevated CNTor was detected in the IW in DED patients (p<0.05). The CNTor for straight corneal nerves, however, showed no significant alteration in DED patients (p>0.05).

CONCLUSION

The proposed method segments both whorl-like and straight corneal nerves in IVCM images with high accuracy and offered parameters to objectively quantify DED-induced corneal nerve injury. The IW is an effective region to detect alterations of multiple geometric indices in DED patients.

Random expert sampling for deep learning segmentation of acute ischemic stroke on non-contrast CT

BACKGROUND

Outlining acutely infarcted tissue on non-contrast CT is a challenging task for which human inter-reader agreement is limited. We explored two different methods for training a supervised deep learning algorithm: one that used a segmentation defined by majority vote among experts and another that trained randomly on separate individual expert segmentations.

METHODS

The data set consisted of 260 non-contrast CT studies in 233 patients with acute ischemic stroke recruited from the multicenter DEFUSE 3 (Endovascular Therapy Following Imaging Evaluation for Ischemic Stroke 3) trial. Additional external validation was performed using 33 patients with matched stroke onset times from the University Hospital Lausanne. A benchmark U-Net was trained on the reference annotations of three experienced neuroradiologists to segment ischemic brain tissue using majority vote and random expert sampling training schemes. The median of volume, overlap, and distance segmentation metrics were determined for agreement in lesion segmentations between (1) three experts, (2) the majority model and each expert, and (3) the random model and each expert. The two sided Wilcoxon signed rank test was used to compare performances (1) to 2) and (1) to (3). We further compared volumes with the 24 hour follow-up diffusion weighted imaging (DWI, final infarct core) and correlations with clinical outcome (modified Rankin Scale (mRS) at 90 days) with the Spearman method.

RESULTS

The random model outperformed the inter-expert agreement ((1) to (2)) and the majority model ((1) to (3)) (dice 0.51±0.04 vs 0.36±0.05 (P<0.0001) vs 0.45±0.05 (P<0.0001)). The random model predicted volume correlated with clinical outcome (0.19, P<0.05), whereas the median expert volume and majority model volume did not. There was no significant difference when comparing the volume correlations between random model, median expert volume, and majority model to 24 hour follow-up DWI volume (P>0.05, n=51).

CONCLUSION

The random model for ischemic injury delineation on non-contrast CT surpassed the inter-expert agreement ((1) to (2)) and the performance of the majority model ((1) to (3)). We showed that the random model volumetric measures of the model were consistent with 24 hour follow-up DWI.

Deep learning-based postoperative glioblastoma segmentation and extent of resection evaluation: Development, external validation, and model comparison

Background

The pursuit of automated methods to assess the extent of resection (EOR) in glioblastomas is challenging, requiring precise measurement of residual tumor volume. Many algorithms focus on preoperative scans, making them unsuitable for postoperative studies. Our objective was to develop a deep learning-based model for postoperative segmentation using magnetic resonance imaging (MRI). We also compared our model's performance with other available algorithms.

Methods

To develop the segmentation model, a training cohort from 3 research institutions and 3 public databases was used. Multiparametric MRI scans with ground truth labels for contrast-enhancing tumor (ET), edema, and surgical cavity, served as training data. The models were trained using MONAI and nnU-Net frameworks. Comparisons were made with currently available segmentation models using an external cohort from a research institution and a public database. Additionally, the model's ability to classify EOR was evaluated using the RANO-Resect classification system. To further validate our best-trained model, an additional independent cohort was used.

Results

The study included 586 scans: 395 for model training, 52 for model comparison, and 139 scans for independent validation. The nnU-Net framework produced the best model with median Dice scores of 0.81 for contrast ET, 0.77 for edema, and 0.81 for surgical cavities. Our best-trained model classified patients into maximal and submaximal resection categories with 96% accuracy in the model comparison dataset and 84% in the independent validation cohort.

Conclusions

Our nnU-Net-based model outperformed other algorithms in both segmentation and EOR classification tasks, providing a freely accessible tool with promising clinical applicability.

Non-inferiority of deep learning ischemic stroke segmentation on non-contrast CT within 16-hours compared to expert neuroradiologists

We determined if a convolutional neural network (CNN) deep learning model can accurately segment acute ischemic changes on non-contrast CT compared to neuroradiologists. Non-contrast CT (NCCT) examinations from 232 acute ischemic stroke patients who were enrolled in the DEFUSE 3 trial were included in this study. Three experienced neuroradiologists independently segmented hypodensity that reflected the ischemic core on each scan. The neuroradiologist with the most experience (expert A) served as the ground truth for deep learning model training. Two additional neuroradiologists' (experts B and C) segmentations were used for data testing. The 232 studies were randomly split into training and test sets. The training set was further randomly divided into 5 folds with training and validation sets. A 3-dimensional CNN architecture was trained and optimized to predict the segmentations of expert A from NCCT. The performance of the model was assessed using a set of volume, overlap, and distance metrics using non-inferiority thresholds of 20%, 3 ml, and 3 mm, respectively. The optimized model trained on expert A was compared to test experts B and C. We used a one-sided Wilcoxon signed-rank test to test for the non-inferiority of the model-expert compared to the inter-expert agreement. The final model performance for the ischemic core segmentation task reached a performance of 0.46 ± 0.09 Surface Dice at Tolerance 5mm and 0.47 ± 0.13 Dice when trained on expert A. Compared to the two test neuroradiologists the model-expert agreement was non-inferior to the inter-expert agreement, [Formula: see text]. The before, CNN accurately delineates the hypodense ischemic core on NCCT in acute ischemic stroke patients with an accuracy comparable to neuroradiologists.