Understanding metric-related pitfalls in image analysis validation

Image-level supervision and self-training for transformer-based cross-modality tumor segmentation

Deep neural networks are commonly used for automated medical image segmentation, but models will frequently struggle to generalize well across different imaging modalities. This issue is particularly problematic due to the limited availability of annotated data, both in the target as well as the source modality, making it difficult to deploy these models on a larger scale. To overcome these challenges, we propose a new semi-supervised training strategy called MoDATTS. Our approach is designed for accurate cross-modality 3D tumor segmentation on unpaired bi-modal datasets. An image-to-image translation strategy between modalities is used to produce synthetic but annotated images and labels in the desired modality and improve generalization to the unannotated target modality. We also use powerful vision transformer architectures for both image translation (TransUNet) and segmentation (Medformer) tasks and introduce an iterative self-training procedure in the later task to further close the domain gap between modalities, thus also training on unlabeled images in the target modality. MoDATTS additionally allows the possibility to exploit image-level labels with a semi-supervised objective that encourages the model to disentangle tumors from the background. This semi-supervised methodology helps in particular to maintain downstream segmentation performance when pixel-level label scarcity is also present in the source modality dataset, or when the source dataset contains healthy controls. The proposed model achieves superior performance compared to other methods from participating teams in the CrossMoDA 2022 vestibular schwannoma (VS) segmentation challenge, as evidenced by its reported top Dice score of 0.87±0.04 for the VS segmentation. MoDATTS also yields consistent improvements in Dice scores over baselines on a cross-modality adult brain gliomas segmentation task composed of four different contrasts from the BraTS 2020 challenge dataset, where 95% of a target supervised model performance is reached when no target modality annotations are available. We report that 99% and 100% of this maximum performance can be attained if 20% and 50% of the target data is additionally annotated, which further demonstrates that MoDATTS can be leveraged to reduce the annotation burden.

Subcutaneous edema segmentation on abdominal CT using multi-class labels and iterative annotation

PURPOSE

Anasarca is a condition that results from organ dysfunctions, such as heart, kidney, or liver failure, characterized by the presence of edema throughout the body. The quantification of accumulated edema may have potential clinical benefits. This work focuses on accurately estimating the amount of edema non-invasively using abdominal CT scans, with minimal false positives. However, edema segmentation is challenging due to the complex appearance of edema and the lack of manually annotated volumes.

METHODS

We propose a weakly supervised approach for edema segmentation using initial edema labels from the current state-of-the-art method for edema segmentation (Intensity Prior), along with labels of surrounding tissues as anatomical priors. A multi-class 3D nnU-Net was employed as the segmentation network, and training was performed using an iterative annotation workflow.

RESULTS

We evaluated segmentation accuracy on a test set of 25 patients with edema. The average Dice Similarity Coefficient of the proposed method was similar to Intensity Prior (61.5% vs. 61.7%; p = 0.83 ). However, the proposed method reduced the average False Positive Rate significantly, from 1.8% to 1.1% ( p < 0.001 ). Edema volumes computed using automated segmentation had a strong correlation with manual annotation (R 2 = 0.87 ).

CONCLUSION

Weakly supervised learning using 3D multi-class labels and iterative annotation is an efficient way to perform high-quality edema segmentation with minimal false positives. Automated edema segmentation can produce edema volume estimates that are highly correlated with manual annotation. The proposed approach is promising for clinical applications to monitor anasarca using estimated edema volumes.

Evaluation of an automated laminar cartilage T2 relaxation time analysis method in an early osteoarthritis model

OBJECTIVE

A fully automated laminar cartilage composition (MRI-based T2) analysis method was technically and clinically validated by comparing radiographically normal knees with (CL-JSN) and without contra-lateral joint space narrowing or other signs of radiographic osteoarthritis (OA, CL-noROA).

MATERIALS AND METHODS

2D U-Nets were trained from manually segmented femorotibial cartilages (n = 72) from all 7 echoes (AllE), or from the 1st echo only (1stE) of multi-echo-spin-echo (MESE) MRIs acquired by the Osteoarthritis Initiative (OAI). Because of its greater accuracy, only the AllE U-Net was then applied to knees from the OAI healthy reference cohort (n = 10), CL-JSN (n = 39), and (1:1) matched CL-noROA knees (n = 39) that all had manual expert segmentation, and to 982 non-matched CL-noROA knees without expert segmentation.

RESULTS

The agreement (Dice similarity coefficient) between automated vs. manual expert cartilage segmentation was between 0.82 ± 0.05/0.79 ± 0.06 (AllE/1stE) and 0.88 ± 0.03/0.88 ± 0.03 (AllE/1stE) across femorotibial cartilage plates. The deviation between automated vs. manually derived laminar T2 reached up to - 2.2 ± 2.6 ms/ + 4.1 ± 10.2 ms (AllE/1stE). The AllE U-Net showed a similar sensitivity to cross-sectional laminar T2 differences between CL-JSN and CL-noROA knees in the matched (Cohen's D ≤ 0.54) and the non-matched (D ≤ 0.54) comparison as the matched manual analyses (D ≤ 0.48). Longitudinally, the AllE U-Net also showed a similar sensitivity to CL-JSN vs. CS-noROA differences in the matched (D ≤ 0.51) and the non-matched (D ≤ 0.43) comparison as matched manual analyses (D ≤ 0.41).

CONCLUSION

The fully automated T2 analysis showed a high agreement, acceptable accuracy, and similar sensitivity to cross-sectional and longitudinal laminar T2 differences in an early OA model, compared with manual expert analysis.

TRIAL REGISTRATION

Clinicaltrials.gov identification: NCT00080171.

Vascular liver segmentation: a narrative review on methods and new insights brought by artificial intelligence

Liver vessel segmentation from routinely performed medical imaging is a useful tool for diagnosis, treatment planning and delivery, and prognosis evaluation for many diseases, particularly liver cancer. A precise representation of liver anatomy is crucial to define the extent of the disease and, when suitable, the consequent resective or ablative procedure, in order to guarantee a radical treatment without sacrificing an excessive volume of healthy liver. Once mainly performed manually, with notable cost in terms of time and human energies, vessel segmentation is currently realized through the application of artificial intelligence (AI), which has gained increased interest and development of the field. Many different AI-driven models adopted for this aim have been described and can be grouped into different categories: thresholding methods, edge- and region-based methods, model-based methods, and machine learning models. The latter includes neural network and deep learning models that now represent the principal algorithms exploited for vessel segmentation. The present narrative review describes how liver vessel segmentation can be realized through AI models, with a summary of model results in terms of accuracy, and an overview on the future progress of this topic.

The impact of multicentric datasets for the automated tumor delineation in primary prostate cancer using convolutional neural networks on 18F-PSMA-1007 PET

PURPOSE

Convolutional Neural Networks (CNNs) have emerged as transformative tools in the field of radiation oncology, significantly advancing the precision of contouring practices. However, the adaptability of these algorithms across diverse scanners, institutions, and imaging protocols remains a considerable obstacle. This study aims to investigate the effects of incorporating institution-specific datasets into the training regimen of CNNs to assess their generalization ability in real-world clinical environments. Focusing on a data-centric analysis, the influence of varying multi- and single center training approaches on algorithm performance is conducted.

METHODS

nnU-Net is trained using a dataset comprising 161 18F-PSMA-1007 PET images collected from four distinct institutions (Freiburg: n = 96, Munich: n = 19, Cyprus: n = 32, Dresden: n = 14). The dataset is partitioned such that data from each center are systematically excluded from training and used solely for testing to assess the model's generalizability and adaptability to data from unfamiliar sources. Performance is compared through a 5-Fold Cross-Validation, providing a detailed comparison between models trained on datasets from single centers to those trained on aggregated multi-center datasets. Dice Similarity Score, Hausdorff distance and volumetric analysis are used as primary evaluation metrics.

RESULTS

The mixed training approach yielded a median DSC of 0.76 (IQR: 0.64-0.84) in a five-fold cross-validation, showing no significant differences (p = 0.18) compared to models trained with data exclusion from each center, which performed with a median DSC of 0.74 (IQR: 0.56-0.86). Significant performance improvements regarding multi-center training were observed for the Dresden cohort (multi-center median DSC 0.71, IQR: 0.58-0.80 vs. single-center 0.68, IQR: 0.50-0.80, p < 0.001) and Cyprus cohort (multi-center 0.74, IQR: 0.62-0.83 vs. single-center 0.72, IQR: 0.54-0.82, p < 0.01). While Munich and Freiburg also showed performance improvements with multi-center training, results showed no statistical significance (Munich: multi-center DSC 0.74, IQR: 0.60-0.80 vs. single-center 0.72, IQR: 0.59-0.82, p > 0.05; Freiburg: multi-center 0.78, IQR: 0.53-0.87 vs. single-center 0.71, IQR: 0.53-0.83, p = 0.23).

CONCLUSION

CNNs trained for auto contouring intraprostatic GTV in 18F-PSMA-1007 PET on a diverse dataset from multiple centers mostly generalize well to unseen data from other centers. Training on a multicentric dataset can improve performance compared to training exclusively with a single-center dataset regarding intraprostatic 18F-PSMA-1007 PET GTV segmentation. The segmentation performance of the same CNN can vary depending on the dataset employed for training and testing.

In silico labeling in cell biology: Potential and limitations

In silico labeling is the computational cross-modality image translation where the output modality is a subcellular marker that is not specifically encoded in the input image, for example, in silico localization of organelles from transmitted light images. In principle, in silico labeling has the potential to facilitate rapid live imaging of multiple organelles with reduced photobleaching and phototoxicity, a technology enabling a major leap toward understanding the cell as an integrated complex system. However, five years have passed since feasibility was attained, without any demonstration of using in silico labeling to uncover new biological insight. In here, we discuss the current state of in silico labeling, the limitations preventing it from becoming a practical tool, and how we can overcome these limitations to reach its full potential.

aiSEGcell: User-friendly deep learning-based segmentation of nuclei in transmitted light images

Segmentation is required to quantify cellular structures in microscopic images. This typically requires their fluorescent labeling. Convolutional neural networks (CNNs) can detect these structures also in only transmitted light images. This eliminates the need for transgenic or dye fluorescent labeling, frees up imaging channels, reduces phototoxicity and speeds up imaging. However, this approach currently requires optimized experimental conditions and computational specialists. Here, we introduce "aiSEGcell" a user-friendly CNN-based software to segment nuclei and cells in bright field images. We extensively evaluated it for nucleus segmentation in different primary cell types in 2D cultures from different imaging modalities in hand-curated published and novel imaging data sets. We provide this curated ground-truth data with 1.1 million nuclei in 20,000 images. aiSEGcell accurately segments nuclei from even challenging bright field images, very similar to manual segmentation. It retains biologically relevant information, e.g. for demanding quantification of noisy biosensors reporting signaling pathway activity dynamics. aiSEGcell is readily adaptable to new use cases with only 32 images required for retraining. aiSEGcell is accessible through both a command line, and a napari graphical user interface. It is agnostic to computational environments and does not require user expert coding experience.

Insights into traditional Large Deformation Diffeomorphic Metric Mapping and unsupervised deep-learning for diffeomorphic registration and their evaluation

This paper explores the connections between traditional Large Deformation Diffeomorphic Metric Mapping methods and unsupervised deep-learning approaches for non-rigid registration, particularly emphasizing diffeomorphic registration. The study provides useful insights and establishes connections between the methods, thereby facilitating a profound understanding of the methodological landscape. The methods considered in our study are extensively evaluated in T1w MRI images using traditional NIREP and Learn2Reg OASIS evaluation protocols with a focus on fairness, to establish equitable benchmarks and facilitate informed comparisons. Through a comprehensive analysis of the results, we address key questions, including the intricate relationship between accuracy and transformation quality in performance, the disentanglement of the influence of registration ingredients on performance, and the determination of benchmark methods and baselines. We offer valuable insights into the strengths and limitations of both traditional and deep-learning methods, shedding light on their comparative performance and guiding future advancements in the field.

Connectivity of the Piriform Cortex and its Implications in Temporal Lobe Epilepsy

Background

The piriform cortex has been implicated in the initiation, spread and termination of epileptic seizures. This understanding has extended to surgical management of epilepsy, where it has been shown that resection or ablation of the piriform cortex can result in better outcomes. How and why the piriform cortex may play such a crucial role in seizure networks is not well understood. To answer these questions, we investigated the functional and structural connectivity of the piriform cortex in both healthy controls and temporal lobe epilepsy (TLE) patients.

Methods

We studied a retrospective cohort of 55 drug-resistant unilateral TLE patients and 26 healthy controls who received structural and functional neuroimaging. Using seed-to-voxel connectivity we compared the normative whole-brain connectivity of the piriform to that of the hippocampus, a region commonly involved in epilepsy, to understand the differential contribution of the piriform to the epileptogenic network. We subsequently measured the inter-piriform coupling (IPC) to quantify similarities in the inter-hemispheric cortical functional connectivity profile between the two piriform cortices. We related differences in IPC in TLE back to aberrations in normative piriform connectivity, whole brain functional properties, and structural connectivity.

Results

We find that relative to the hippocampus, the piriform is functionally connected to the anterior insula and the rest of the salience ventral attention network (SAN). We also find that low IPC is a sensitive metric of poor surgical outcome (sensitivity: 85.71%, 95% CI: [19.12%, 99.64%]); and differences in IPC within TLE were related to disconnectivity and hyperconnectivity to the anterior insula and the SAN. More globally, we find that low IPC is associated with whole-brain functional and structural segregation, marked by decreased functional small-worldness and fractional anisotropy.

Conclusions

Our study presents novel insights into the functional and structural neural network alterations associated with this structure, laying the foundation for future work to carefully consider its connectivity during the presurgical management of epilepsy.

Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update

To address the rapid evolution of artificial intelligence in medical imaging, the authors present the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) 2024 Update.

Image Quality Assessment Tool for Conventional and Dynamic Magnetic Resonance Imaging Acquisitions

Image quality assessment of magnetic resonance imaging (MRI) data is an important factor not only for conventional diagnosis and protocol optimization but also for fairness, trustworthiness, and robustness of artificial intelligence (AI) applications, especially on large heterogeneous datasets. Information on image quality in multi-centric studies is important to complement the contribution profile from each data node along with quantity information, especially when large variability is expected, and certain acceptance criteria apply. The main goal of this work was to present a tool enabling users to assess image quality based on both subjective criteria as well as objective image quality metrics used to support the decision on image quality based on evidence. The evaluation can be performed on both conventional and dynamic MRI acquisition protocols, while the latter is also checked longitudinally across dynamic series. The assessment provides an overall image quality score and information on the types of artifacts and degrading factors as well as a number of objective metrics for automated evaluation across series (BRISQUE score, Total Variation, PSNR, SSIM, FSIM, MS-SSIM). Moreover, the user can define specific regions of interest (ROIs) to calculate the regional signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR), thus individualizing the quality output to specific use cases, such as tissue-specific contrast or regional noise quantification.

MarShie: a clearing protocol for 3D analysis of single cells throughout the bone marrow at subcellular resolution

Analyzing immune cell interactions in the bone marrow is vital for understanding hematopoiesis and bone homeostasis. Three-dimensional analysis of the complete, intact bone marrow within the cortex of whole long bones remains a challenge, especially at subcellular resolution. We present a method that stabilizes the marrow and provides subcellular resolution of fluorescent signals throughout the murine femur, enabling identification and spatial characterization of hematopoietic and stromal cell subsets. By combining a pre-processing algorithm for stripe artifact removal with a machine-learning approach, we demonstrate reliable cell segmentation down to the deepest bone marrow regions. This reveals age-related changes in the marrow. It highlights the interaction between CX3CR1+ cells and the vascular system in homeostasis, in contrast to other myeloid cell types, and reveals their spatial characteristics after injury. The broad applicability of this method will contribute to a better understanding of bone marrow biology.

Metrics reloaded: recommendations for image analysis validation

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.