Annotation-efficient deep learning for automatic medical image segmentation

Automatic medical imaging segmentation via self-supervising large-scale convolutional neural networks

PURPOSE

This study aims to develop a robust, large-scale deep learning model for medical image segmentation, leveraging self-supervised learning to overcome the limitations of supervised learning and data variability in clinical settings.

METHODS AND MATERIALS

We curated a substantial multi-center CT dataset for self-supervised pre-training using masked image modeling with sparse submanifold convolution. We designed a series of Sparse Submanifold U-Nets (SS-UNets) of varying sizes and performed self-supervised pre-training. We fine-tuned the SS-UNets on the TotalSegmentator dataset. The evaluation encompassed robustness tests on four unseen datasets and transferability assessments on three additional datasets.

RESULTS

Our SS-UNets exhibited superior performance in comparison to state-of-the-art self-supervised methods, demonstrating higher Dice Similarity Coefficient (DSC) and Surface Dice Coefficient (SDC) metrics. SS-UNet-B achieved 84.3 % DSC and 88.0 % SDC in TotalSegmentator. We further demonstrated the scalability of our networks, with segmentation performance increasing with model size, demonstrated from 58 million to 1.4 billion parameters:4.6 % DSC and 3.2 % SDC improvement in TotalSegmentator from SS-UNet-B to SS-UNet-H.

CONCLUSIONS

We demonstrate the efficacy of self-supervised learning for medical image segmentation in the CT, MRI and PET domains. Our approach significantly reduces reliance on extensively labeled data, mitigates risks of overfitting, and enhances model generalizability. Future applications may allow accurate segmentation of organs and lesions across several imaging domains, potentially streamlining cancer detection and radiotherapy treatment planning.

Computer vision algorithms in healthcare: Recent advancements and future challenges

Computer vision has emerged as a promising technology with numerous applications in healthcare. This systematic review provides an overview of advancements and challenges associated with computer vision in healthcare. The review highlights the application areas where computer vision has made significant strides, including medical imaging, surgical assistance, remote patient monitoring, and telehealth. Additionally, it addresses the challenges related to data quality, privacy, model interpretability, and integration with existing healthcare systems. Ethical and legal considerations, such as patient consent and algorithmic bias, are also discussed. The review concludes by identifying future directions and opportunities for research, emphasizing the potential impact of computer vision on healthcare delivery and outcomes. Overall, this systematic review underscores the importance of understanding both the advancements and challenges in computer vision to facilitate its responsible implementation in healthcare.

Large language model-augmented learning for auto-delineation of treatment targets in head-and-neck cancer radiotherapy

BACKGROUND AND PURPOSE

Radiation therapy (RT) is highly effective, but its success depends on accurate, manual target delineation, which is time-consuming, labor-intensive, and prone to variability. Despite AI advancements in auto-contouring normal tissues, accurate RT target volume delineation remains challenging. This study presents Radformer, a novel visual language model that integrates text-rich clinical data with medical imaging for accurate automated RT target volume delineation.

MATERIALS AND METHODS

We developed Radformer, an innovative network that utilizes a hierarchical vision transformer as its backbone and integrates large language models (LLMs) to extract and embed clinical data in text-rich form. The model features a novel visual language attention module (VLAM) to combine visual and linguistic features, enabling language-aware visual encoding (LAVE). The Radformer was evaluated on a dataset of 2985 patients with head-and-neck cancer who underwent RT. Quantitative evaluations were performed utilizing metrics such as the Dice similarity coefficient (DSC), intersection over union (IOU), and 95th percentile Hausdorff distance (HD95).

RESULTS

The Radformer demonstrated superior performance in segmenting RT target volumes compared to state-of-the-art models. On the head-and-neck cancer dataset, Radformer achieved a mean DSC of 0.76 ± 0.09 versus 0.66 ± 0.09, a mean IOU of 0.69 ± 0.08 versus 0.59 ± 0.07, and a mean HD95 of 7.82 ± 6.87 mm versus 14.28 ± 6.85 mm for gross tumor volume delineation, compared to the baseline 3D-UNETR.

CONCLUSIONS

The Radformer model offers a clinically optimal means of RT target auto-delineation by integrating both imaging and clinical data through a visual language model. This approach improves the accuracy of RT target volume delineation, facilitating broader AI-assisted automation in RT treatment planning.

CT-based Machine Learning Radiomics Modeling: Survival Prediction and Mechanism Exploration in Ovarian Cancer Patients

RATIONALE AND OBJECTIVES

To create a radiomics model based on computed tomography (CT) to predict overall survival in ovarian cancer patients. To combine Rad-score with genomic data to explore the association between gene expression and Rad-score.

MATERIALS AND METHODS

Imaging and clinical data from 455 patients with ovarian cancer were retrospectively analyzed. Patients were categorized into training cohort, validation cohort and test cohort. Cox regression analysis and the least absolute shrinkage and selection operator (LASSO) methods were utilized to identify characteristics and develop the Rad-score. Radiomics models were developed and evaluated for predictive efficacy and clinical incremental value. Application of genomic data from the cancer genome atlas (TCGA) to reveal differential genes in different Rad-score groups. Screening hub genes and exploring the functions of hub genes through bioinformatics analysis and machine learning.

RESULTS

Prognostic models based on FIGO, tumor residual disease and Rad-score were developed. The receiver operating characteristic (ROC) curves showed that the 1, 3, and 5 year area under curves (AUCs) of the model were in the training group (0.816, 0.865 and 0.862, respectively), validation group (0.845, 0.877, 0.869, respectively) and test group (0.899, 0.906 and 0.869, respectively) had good predictive accuracy. Calibration curves showed good agreement between observations and predictions. Decision curve analysis revealed a high net benefit of the clinical-radiomics model. The clinical impact curve (CIC) showed good clinical applicability of the clinical-radiomics model. Analysis of sequencing data from the TCGA database revealed EMP1 as a hub gene for radiomics modeling. It revealed that its biological function may be associated with extracellular matrix organization and focal adhesion.

CONCLUSION

Prognostic models based on FIGO, Tumor residual disease, and Rad-score can effectively predict the overall survival (OS) of ovarian cancer patients. Rad-score may enable prognostic prediction of ovarian cancer patients by revealing the expression level of EMP1 and its biological function.

Context-Aware Dual-Task Deep Network for Concurrent Bone Segmentation and Clinical Assessment to Enhance Shoulder Arthroplasty Preoperative planning

Goal: Effective preoperative planning for shoulder joint replacement requires accurate glenohumeral joint (GH) digital surfaces and reliable clinical staging. Methods: xCEL-UNet was designed as a dual-task deep network for humerus and scapula bone reconstruction in CT scans, and assessment of three GH joint clinical conditions, namely osteophyte size (OS), joint space reduction (JS), and humeroscapular alignment (HSA). Results: Trained on a dataset of 571 patients, the model optimized segmentation and classification through transfer learning. It achieved median root mean squared errors of 0.31 and 0.24 mm, and Hausdorff distances of 2.35 and 3.28 mm for the humerus and scapula, respectively. Classification accuracy was 91 for OS, 93 for JS, and 85% for HSA. GradCAM-based activation maps validated the network's interpretability. Conclusions: this framework delivers accurate 3D bone surface reconstructions and dependable clinical assessments of the GH joint, offering robust support for therapeutic decision-making in shoulder arthroplasty.

Metastasis lesion segmentation from bone scintigrams using encoder-decoder architecture model with multi-attention and multi-scale learning

Background

The limitation in spatial resolution of bone scintigraphy, combined with the vast variations in size, location, and intensity of bone metastasis (BM) lesions, poses challenges for accurate diagnosis by human experts. Deep learning-based analysis has emerged as a preferred approach for automating the identification and delineation of BM lesions. This study aims to develop a deep learning-based approach to automatically segment bone scintigrams for improving diagnostic accuracy.

Methods

This study introduces a deep learning-based segmentation model structured around an encoder-decoder architecture. The model employs a multi-attention learning scheme to enhance the contrast of the skeleton outline against the background and a multi-scale learning strategy to highlight the hotspots within skeletal areas. The multi-attention strategies include the Non-local Attention scheme and the vision transformer (ViT), while the multi-scale learning incorporates the multi-scale feature learning strategy and the multi-pooling learning strategy. This combination enables the proposed model to accurately detect and extract lesions of varying sizes with high randomness in location and intensity.

Results

Experimental evaluation conducted on clinical data of single photon emission computed tomography (SPECT) bone scintigrams showed the superior performance of the proposed model, achieving the highest-ever dice similarity coefficient (DSC) score of 0.6720. A comparative analysis on the same dataset demonstrated increased scores of 5.6%, 2.03%, and 7.9% for DSC, Precision, and Recall, respectively, compared to the existing models.

Conclusions

The proposed segmentation model can be used as a promising tool for automatically extracting metastasis lesions from SPECT bone scintigrams, offering significant support to the development of deep learning-based automated analysis for characterizing BM.

Predicting overall survival and prophylactic cranial irradiation benefit in small cell lung cancer patients: a multicenter cohort study

BACKGROUND

To construct a CT-based radiomics nomogram, enabling the estimation of overall survival (OS) in small cell lung cancer (SCLC) patients and facilitating the identification of prophylactic cranial irradiation (PCI) beneficiaries through risk stratification using the radiomics score (RS).

METHODS

A retrospective recruitment of 375 patients with pathologically confirmed SCLC was conducted across three medical centers, followed by their division into different cohorts. To generate the RS, a series of analyses were performed, including Pearson correlation analysis, univariate Cox analysis, and least absolute shrinkage and selection operator (LASSO) Cox regression analysis. Subsequently, patients were stratified into either the low RS or high RS group, determined by identifying the optimal RS cutoff value. Subsequently, a radiomics nomogram was constructed using the RS, followed by assessments of its discrimination, calibration, clinical utility and reclassification. Moreover, we evaluated the potential benefits of PCI following stratification by RS.

RESULTS

For the internal and external validation cohorts, the radiomics nomogram (concordance index [C-index]: 0.770, 0.763) outperformed clinical nomogram (C-index: 0.625, 0.570) in predicting OS. Besides, patients with high RS had survival benefit from PCI in both the limited and extensive stage (hazard ratio [HR]: 0.304, 95% confidence interval [CI]: 0.087-1.065, P = 0.003; HR: 0.481, 95% CI: 0.270-0.860, P = 0.019, respectively), while no significant association were observed in patients with low RS.

CONCLUSION

A radiomics nomogram based on CT shows potential in predicting OS for individuals with SCLC. The RS could assist in tailoring treatment plans to identify patients likely to benefit from PCI.

An automated and time-efficient framework for simulation of coronary blood flow under steady and pulsatile conditions

BACKGROUND AND OBJECTIVE

Invasive fractional flow reserve (FFR) measurement is the gold standard method for coronary artery disease (CAD) diagnosis. FFR-CT exploits computational fluid dynamics (CFD) for non-invasive evaluation of FFR, simulating coronary flow in virtual geometries reconstructed from computed tomography (CT), but suffers from cost-intensive computing process and uncertainties in the definition of patient specific boundary conditions (BCs). In this work, we investigated the use of time-averaged steady BCs, compared to pulsatile to reduce the computational time and deployed a self-adjusting method for the tuning of BCs to patient-specific clinical data.

METHODS

133 coronary arteries were reconstructed form CT images of patients suffering from CAD. For each vessel, invasive FFR was measured. After segmentation, the geometries were prepared for CFD simulation by clipping the outlets and discretizing into tetrahedral mesh. Steady BCs were defined in two steps: (i) rest BCs were extrapolated from clinical and image-derived data; (ii) hyperemic BCs were computed from resting conditions. Flow rate was iteratively adjusted during the simulation, until patient's aortic pressure was matched. Pulsatile BCs were defined exploiting the convergence values of steady BCs. After CFD simulation, lesion-specific hemodynamic indexes were computed and compared between group of patients for which surgery was indicated and not. The whole pipeline was implemented as a straightforward process, in which each single step is performed automatically.

RESULTS

Steady and pulsatile FFR-CT yielded a strong correlation (r = 0.988, p < 0.001) and correlated with invasive FFR (r = 0.797, p < 0.001). The per-point difference between the pressure and FFR-CT field predicted by the two methods was below 1 % and 2 %, respectively. Both approaches exhibited a good diagnostic performance: accuracy was 0.860 and 0.864, the AUC was 0.923 and 0.912, for steady and pulsatile case, respectively. The computational time required by steady BCs CFD was approximatively 30-folds lower than pulsatile case.

CONCLUSIONS

This work shows the feasibility of using steady BCs CFD for computing the FFR-CT in coronary arteries, as well as its computational and diagnostic performance within a fully automated pipeline.

DSIFNet: Implicit feature network for nasal cavity and vestibule segmentation from 3D head CT

This study is dedicated to accurately segment the nasal cavity and its intricate internal anatomy from head CT images, which is critical for understanding nasal physiology, diagnosing diseases, and planning surgeries. Nasal cavity and it's anatomical structures such as the sinuses, and vestibule exhibit significant scale differences, with complex shapes and variable microstructures. These features require the segmentation method to have strong cross-scale feature extraction capabilities. To effectively address this challenge, we propose an image segmentation network named the Deeply Supervised Implicit Feature Network (DSIFNet). This network uniquely incorporates an Implicit Feature Function Module Guided by Local and Global Positional Information (LGPI-IFF), enabling effective fusion of features across scales and enhancing the network's ability to recognize details and overall structures. Additionally, we introduce a deep supervision mechanism based on implicit feature functions in the network's decoding phase, optimizing the utilization of multi-scale feature information, thus improving segmentation precision and detail representation. Furthermore, we constructed a dataset comprising 7116 CT volumes (including 1,292,508 slices) and implemented PixPro-based self-supervised pretraining to utilize unlabeled data for enhanced feature extraction. Our tests on nasal cavity and vestibule segmentation, conducted on a dataset comprising 128 head CT volumes (including 34,006 slices), demonstrate the robustness and superior performance of proposed method, achieving leading results across multiple segmentation metrics.

A CT-Based Clinical-Radiomics Nomogram for Predicting the Overall Survival to TACE Combined with Camrelizumab and Apatinib in Patients with Advanced Hepatocellular Carcinoma

RATIONALE AND OBJECTIVES

To construct a computed tomography (CT)-based clinical-radiomics nomogram for estimating overall survival (OS) in advanced hepatocellular carcinoma (HCC) patients receiving transcatheter arterial chemoembolization (TACE) in combination with camrelizumab and apatinib.

METHODS

A retrospective recruitment of 150 patients with clinically or pathologically confirmed HCC was conducted, followed by their division into training cohort (n = 105) and test cohort (n = 45). To generate the radiomics score (Rad-score), a series of analyses were performed, including Pearson correlation analysis, univariate Cox analysis, and least absolute shrinkage and selection operator Cox regression analysis. Subsequently, a clinical-radiomics nomogram was constructed using the Rad-score combined with independent clinical prognostic factors, followed by assessments of its calibration, discrimination, reclassification, and clinical utility.

RESULTS

Five CT radiomics features were selected. The Rad-score showed a significant correlation with OS (P < 0.001). The clinical-radiomics nomogram demonstrated superior performance in estimating OS, with a concordance index (C-index) of 0.840, compared to the radiomics nomogram (C-index: 0.817) and the clinical nomogram (C-index: 0.661). It also exhibited high 1-year and 2-year area under the curves of 0.936 and 0.946, respectively. Additionally, the clinical-radiomics nomogram markedly enhanced classification accuracy for OS outcomes, as evidenced by net reclassification improvement and integrated discrimination improvement. Decision curve analysis confirmed its clinical utility.

CONCLUSION

A CT-based clinical-radiomics nomogram exhibits strong potential for predicting OS in advanced HCC patients undergoing TACE combined with camrelizumab and apatinib.

Kruskal Szekeres generative adversarial network augmented deep autoencoder for colorectal cancer detection

Cancer involves abnormal cell growth, with types like intestinal and oesophageal cancer often diagnosed in advanced stages, making them hard to cure. Symptoms are like burning sensations in the stomach and swallowing difficulties are specified as colorectal cancer. Deep learning significantly impacts the medical image processing and diagnosis, offering potential improvements in accuracy and efficiency. The Kruskal Szekeres Generative Adversarial Network Augmented Deep Autoencoder (KSGANA-DA) is introduced for early colorectal cancer detection and it comprises two stages; Initial stage, data augmentation uses Affine Transform via Random Horizontal Rotation and Geometric Transform via Kruskal-Szekeres that coordinates to improve the training dataset diversity, boosting detection performance. The second stage, a Deep Autoencoder Anatomical Landmark-based Image Segmentation preserves edge pixel spatial locations, improving precision and recall for early boundary detection. Experiments validate KSGANA-DA performance and different existing methods are implemented into Python. The results of KSGANA-DA are to provide higher precision by 41%, recall by 7%, and lesser training time by 46% than compared to conventional methods.

Neural Network Methods in the Development of MEMS Sensors

As a kind of long-term favorable device, the microelectromechanical system (MEMS) sensor has become a powerful dominator in the detection applications of commercial and industrial areas. There have been a series of mature solutions to address the possible issues in device design, optimization, fabrication, and output processing. The recent involvement of neural networks (NNs) has provided a new paradigm for the development of MEMS sensors and greatly accelerated the research cycle of high-performance devices. In this paper, we present an overview of the progress, applications, and prospects of NN methods in the development of MEMS sensors. The superiority of leveraging NN methods in structural design, device fabrication, and output compensation/calibration is reviewed and discussed to illustrate how NNs have reformed the development of MEMS sensors. Relevant issues in the usage of NNs, such as available models, dataset construction, and parameter optimization, are presented. Many application scenarios have demonstrated that NN methods can enhance the speed of predicting device performance, rapidly generate device-on-demand solutions, and establish more accurate calibration and compensation models. Along with the improvement in research efficiency, there are also several critical challenges that need further exploration in this area.

The AI revolution in glaucoma: Bridging challenges with opportunities

Recent advancements in artificial intelligence (AI) herald transformative potentials for reshaping glaucoma clinical management, improving screening efficacy, sharpening diagnosis precision, and refining the detection of disease progression. However, incorporating AI into healthcare usages faces significant hurdles in terms of developing algorithms and putting them into practice. When creating algorithms, issues arise due to the intensive effort required to label data, inconsistent diagnostic standards, and a lack of thorough testing, which often limits the algorithms' widespread applicability. Additionally, the "black box" nature of AI algorithms may cause doctors to be wary or skeptical. When it comes to using these tools, challenges include dealing with lower-quality images in real situations and the systems' limited ability to work well with diverse ethnic groups and different diagnostic equipment. Looking ahead, new developments aim to protect data privacy through federated learning paradigms, improving algorithm generalizability by diversifying input data modalities, and augmenting datasets with synthetic imagery. The integration of smartphones appears promising for using AI algorithms in both clinical and non-clinical settings. Furthermore, bringing in large language models (LLMs) to act as interactive tool in medicine may signify a significant change in how healthcare will be delivered in the future. By navigating through these challenges and leveraging on these as opportunities, the field of glaucoma AI will not only have improved algorithmic accuracy and optimized data integration but also a paradigmatic shift towards enhanced clinical acceptance and a transformative improvement in glaucoma care.

Self-supervised learning for accurately modelling hierarchical evolutionary patterns of cerebrovasculature

Cerebrovascular abnormalities are critical indicators of stroke and neurodegenerative diseases like Alzheimer's disease (AD). Understanding the normal evolution of brain vessels is essential for detecting early deviations and enabling timely interventions. Here, for the first time, we proposed a pipeline exploring the joint evolution of cortical volumes (CVs) and arterial volumes (AVs) in a large cohort of 2841 individuals. Using advanced deep learning for vessel segmentation, we built normative models of CVs and AVs across spatially hierarchical brain regions. We found that while AVs generally decline with age, distinct trends appear in regions like the circle of Willis. Comparing healthy individuals with those affected by AD or stroke, we identified significant reductions in both CVs and AVs, wherein patients with AD showing the most severe impact. Our findings reveal gender-specific effects and provide critical insights into how these conditions alter brain structure, potentially guiding future clinical assessments and interventions.

Deep Learning-Assisted Assessing of Single Circulating Tumor Cell Viability via Cellular Morphology

Circulating tumor cells (CTCs) are closely associated with cancer metastasis and recurrence, so the assessment of CTC viability is crucial for diagnosis, prognosis evaluation, and efficacy judgment of cancer. Due to the extreme scarcity of CTCs in human blood, it is difficult to accurately evaluate the viability of a single CTC. In this study, a deep learning model based on a convolutional neural network was constructed and trained to extract the morphological features of CTCs with different viabilities defined by cell counting kit-8, achieve accurate CTC identification, and assess the viability of a single CTC. Being efficient, accurate, and noninvasive, it has a broad application prospect in biomedical directions.

Collaborative Learning for Annotation-Efficient Volumetric MR Image Segmentation

BACKGROUND

Deep learning has presented great potential in accurate MR image segmentation when enough labeled data are provided for network optimization. However, manually annotating three-dimensional (3D) MR images is tedious and time-consuming, requiring experts with rich domain knowledge and experience.

PURPOSE

To build a deep learning method exploring sparse annotations, namely only a single two-dimensional slice label for each 3D training MR image.

STUDY TYPE

Retrospective.

POPULATION

Three-dimensional MR images of 150 subjects from two publicly available datasets were included. Among them, 50 (1377 image slices) are for prostate segmentation. The other 100 (8800 image slices) are for left atrium segmentation. Five-fold cross-validation experiments were carried out utilizing the first dataset. For the second dataset, 80 subjects were used for training and 20 were used for testing.

FIELD STRENGTH/SEQUENCE

1.5 T and 3.0 T; axial T2-weighted and late gadolinium-enhanced, 3D respiratory navigated, inversion recovery prepared gradient echo pulse sequence.

ASSESSMENT

A collaborative learning method by integrating the strengths of semi-supervised and self-supervised learning schemes was developed. The method was trained using labeled central slices and unlabeled noncentral slices. Segmentation performance on testing set was reported quantitatively and qualitatively.

STATISTICAL TESTS

Quantitative evaluation metrics including boundary intersection-over-union (B-IoU), Dice similarity coefficient, average symmetric surface distance, and relative absolute volume difference were calculated. Paired t test was performed, and P < 0.05 was considered statistically significant.

RESULTS

Compared to fully supervised training with only the labeled central slice, mean teacher, uncertainty-aware mean teacher, deep co-training, interpolation consistency training (ICT), and ambiguity-consensus mean teacher, the proposed method achieved a substantial improvement in segmentation accuracy, increasing the mean B-IoU significantly by more than 10.0% for prostate segmentation (proposed method B-IoU: 70.3% ± 7.6% vs. ICT B-IoU: 60.3% ± 11.2%) and by more than 6.0% for left atrium segmentation (proposed method B-IoU: 66.1% ± 6.8% vs. ICT B-IoU: 60.1% ± 7.1%).

DATA CONCLUSIONS

A collaborative learning method trained using sparse annotations can segment prostate and left atrium with high accuracy.

LEVEL OF EVIDENCE

0 TECHNICAL EFFICACY: Stage 1.

Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement

Recently, vision-language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care. The underlying hypothesis is that the rich knowledge embedded in radiology reports can effectively assist and guide the learning process, reducing the need for additional labels. However, these reports tend to be complex and sometimes even consist of redundant descriptions that make the representation learning too challenging to capture the key semantic information. This paper develops a novel iterative vision-language representation learning framework by proposing a key semantic knowledge-emphasized report refinement method. Particularly, raw radiology reports are refined to highlight the key information according to a constructed clinical dictionary and two model-optimized knowledge-enhancement metrics. The iterative framework is designed to progressively learn, starting from gaining a general understanding of the patient's condition based on raw reports and gradually refines and extracts critical information essential to the fine-grained analysis tasks. The effectiveness of the proposed framework is validated on various downstream medical image analysis tasks, including disease classification, region-of-interest segmentation, and phrase grounding. Our framework surpasses seven state-of-the-art methods in both fine-tuning and zero-shot settings, demonstrating its encouraging potential for different clinical applications.

Impact of annotation imperfections and auto-curation for deep learning-based organ-at-risk segmentation

Background and purpose

Segmentation imperfections (noise) in radiotherapy organ-at-risk segmentation naturally arise from specialist experience and image quality. Using clinical contours can result in sub-optimal convolutional neural network (CNN) training and performance, but manual curation is costly. We address the impact of simulated and clinical segmentation noise on CNN parotid gland (PG) segmentation performance and provide proof-of-concept for an easily implemented auto-curation countermeasure.

Methods and Materials

The impact of segmentation imperfections was investigated by simulating noise in clean, high-quality segmentations. Curation efficacy was tested by removing lowest-scoring Dice similarity coefficient (DSC) cases early during CNN training, both in simulated (5-fold) and clinical (10-fold) settings, using our full radiotherapy clinical cohort (RTCC; N = 1750 individual PGs). Statistical significance was assessed using Bonferroni-corrected Wilcoxon signed-rank tests. Curation efficacies were evaluated using DSC and mean surface distance (MSD) on in-distribution and out-of-distribution data and visual inspection.

Results

The curation step correctly removed median(range) 98(90-100)% of corrupted segmentations and restored the majority (1.2 %/1.3 %) of DSC lost from training with 30 % corrupted segmentations. This effect was masked when using typical (non-curated) validation data. In RTCC, 20 % curation showed improved model generalizability which significantly improved out-of-distribution DSC and MSD (p < 1.0e-12, p < 1.0e-6). Improved consistency was observed in particularly the medial and anterior lobes.

Conclusions

Up to 30% case removal, the curation benefit outweighed the training variance lost through curation. Considering the notable ease of implementation, high sensitivity in simulations and performance gains already at lower curation fractions, as a conservative middle ground, we recommend 15% curation of training cases when training CNNs using clinical PG contours.

Cross-site Validation of AI Segmentation and Harmonization in Breast MRI

This work aims to perform a cross-site validation of automated segmentation for breast cancers in MRI and to compare the performance to radiologists. A three-dimensional (3D) U-Net was trained to segment cancers in dynamic contrast-enhanced axial MRIs using a large dataset from Site 1 (n = 15,266; 449 malignant and 14,817 benign). Performance was validated on site-specific test data from this and two additional sites, and common publicly available testing data. Four radiologists from each of the three clinical sites provided two-dimensional (2D) segmentations as ground truth. Segmentation performance did not differ between the network and radiologists on the test data from Sites 1 and 2 or the common public data (median Dice score Site 1, network 0.86 vs. radiologist 0.85, n = 114; Site 2, 0.91 vs. 0.91, n = 50; common: 0.93 vs. 0.90). For Site 3, an affine input layer was fine-tuned using segmentation labels, resulting in comparable performance between the network and radiologist (0.88 vs. 0.89, n = 42). Radiologist performance differed on the common test data, and the network numerically outperformed 11 of the 12 radiologists (median Dice: 0.85-0.94, n = 20). In conclusion, a deep network with a novel supervised harmonization technique matches radiologists' performance in MRI tumor segmentation across clinical sites. We make code and weights publicly available to promote reproducible AI in radiology.

Deep Learning-driven Automatic Nuclei Segmentation of Label-free Live Cell Chromatin-sensitive Partial Wave Spectroscopic Microscopy Imaging

Chromatin-sensitive Partial Wave Spectroscopic (csPWS) microscopy offers a non-invasive glimpse into the mass density distribution of cellular structures at the nanoscale, leveraging the spectroscopic information. Such capability allows us to analyze the chromatin structure and organization and the global transcriptional state of the cell nuclei for the study of its role in carcinogenesis. Accurate segmentation of the nuclei in csPWS microscopy images is an essential step in isolating them for further analysis. However, manual segmentation is error-prone, biased, time-consuming, and laborious, resulting in disrupted nuclear boundaries with partial or over-segmentation. Here, we present an innovative deep-learning-driven approach to automate the accurate nuclei segmentation of label-free live cell csPWS microscopy imaging data. Our approach, csPWS-seg, harnesses the Convolutional Neural Networks-based U-Net model with an attention mechanism to automate the accurate cell nuclei segmentation of csPWS microscopy images. We leveraged the structural, physical, and biological differences between the cytoplasm, nucleus, and nuclear periphery to construct three distinct csPWS feature images for nucleus segmentation. Using these images of HCT116 cells, csPWS-seg achieved superior performance with a median Intersection over Union (IoU) of 0.80 and a Dice Similarity Coefficient (DSC) score of 0.88. The csPWS-seg overcame the segmentation performance over the baseline U-Net model and another attention-based model, SE-U-Net, marking a significant improvement in segmentation accuracy. Further, we analyzed the performance of our proposed model with four loss functions: binary cross-entropy loss, focal loss, dice loss, and Jaccard loss. The csPWS-seg with focal loss provided the best results compared to other loss functions. The automatic and accurate nuclei segmentation offered by the csPWS-seg not only automates, accelerates, and streamlines csPWS data analysis but also enhances the reliability of subsequent chromatin analysis research, paving the way for more accurate diagnostics, treatment, and understanding of cellular mechanisms for carcinogenesis.

Reducing Manual Annotation Costs for Cell Segmentation by Upgrading Low-Quality Annotations

Deep-learning algorithms for cell segmentation typically require large data sets with high-quality annotations to be trained with. However, the annotation cost for obtaining such sets may prove to be prohibitively expensive. Our work aims to reduce the time necessary to create high-quality annotations of cell images by using a relatively small well-annotated data set for training a convolutional neural network to upgrade lower-quality annotations, produced at lower annotation costs. We investigate the performance of our solution when upgrading the annotation quality for labels affected by three types of annotation error: omission, inclusion, and bias. We observe that our method can upgrade annotations affected by high error levels from 0.3 to 0.9 Dice similarity with the ground-truth annotations. We also show that a relatively small well-annotated set enlarged with samples with upgraded annotations can be used to train better-performing cell segmentation networks compared to training only on the well-annotated set. Moreover, we present a use case where our solution can be successfully employed to increase the quality of the predictions of a segmentation network trained on just 10 annotated samples.

Large Language Model-Augmented Auto-Delineation of Treatment Target Volume in Radiation Therapy

Radiation therapy (RT) is one of the most effective treatments for cancer, and its success relies on the accurate delineation of targets. However, target delineation is a comprehensive medical decision that currently relies purely on manual processes by human experts. Manual delineation is time-consuming, laborious, and subject to interobserver variations. Although the advancements in artificial intelligence (AI) techniques have significantly enhanced the auto-contouring of normal tissues, accurate delineation of RT target volumes remains a challenge. In this study, we propose a visual language model-based RT target volume auto-delineation network termed Radformer. The Radformer utilizes a hierarchical vision transformer as the backbone and incorporates large language models to extract text-rich features from clinical data. We introduce a visual language attention module (VLAM) for integrating visual and linguistic features for language-aware visual encoding (LAVE). The Radformer has been evaluated on a dataset comprising 2985 patients with head-and-neck cancer who underwent RT. Metrics, including the Dice similarity coefficient (DSC), intersection over union (IOU), and 95th percentile Hausdorff distance (HD95), were used to evaluate the performance of the model quantitatively. Our results demonstrate that the Radformer has superior segmentation performance compared to other state-of-the-art models, validating its potential for adoption in RT practice.

Weakly Supervised Deep Learning in Radiology

Deep learning (DL) is currently the standard artificial intelligence tool for computer-based image analysis in radiology. Traditionally, DL models have been trained with strongly supervised learning methods. These methods depend on reference standard labels, typically applied manually by experts. In contrast, weakly supervised learning is more scalable. Weak supervision comprises situations in which only a portion of the data are labeled (incomplete supervision), labels refer to a whole region or case as opposed to a precisely delineated image region (inexact supervision), or labels contain errors (inaccurate supervision). In many applications, weak labels are sufficient to train useful models. Thus, weakly supervised learning can unlock a large amount of otherwise unusable data for training DL models. One example of this is using large language models to automatically extract weak labels from free-text radiology reports. Here, we outline the key concepts in weakly supervised learning and provide an overview of applications in radiologic image analysis. With more fundamental and clinical translational work, weakly supervised learning could facilitate the uptake of DL in radiology and research workflows by enabling large-scale image analysis and advancing the development of new DL-based biomarkers.

Incremental value of automatically segmented perirenal adipose tissue for pathological grading of clear cell renal cell carcinoma: a multicenter cohort study

OBJECTIVES

Accurate preoperative prediction of the pathological grade of clear cell renal cell carcinoma (ccRCC) is crucial for optimal treatment planning and patient outcomes. This study aims to develop and validate a deep-learning (DL) algorithm to automatically segment renal tumours, kidneys, and perirenal adipose tissue (PRAT) from computed tomography (CT) images and extract radiomics features to predict the pathological grade of ccRCC.

METHODS

In this cross-ethnic retrospective study, a total of 614 patients were divided into a training set (383 patients from the local hospital), an internal validation set (88 patients from the local hospital), and an external validation set (143 patients from the public dataset). A two-dimensional TransUNet-based DL model combined with the train-while-annotation method was trained for automatic volumetric segmentation of renal tumours, kidneys, and visceral adipose tissue (VAT) on images from two groups of datasets. PRAT was extracted using a dilation algorithm by calculating voxels of VAT surrounding the kidneys. Radiomics features were subsequently extracted from three regions of interest of CT images, adopting multiple filtering strategies. The least absolute shrinkage and selection operator (LASSO) regression was used for feature selection, and the support vector machine (SVM) for developing the pathological grading model. Ensemble learning was used for imbalanced data classification. Performance evaluation included the Dice coefficient for segmentation and metrics such as accuracy and area under curve (AUC) for classification. The WHO/International Society of Urological Pathology (ISUP) grading models were finally interpreted and visualized using the SHapley Additive exPlanations (SHAP) method.

RESULTS

For automatic segmentation, the mean Dice coefficient achieved 0.836 for renal tumours and 0.967 for VAT on the internal validation dataset. For WHO/ISUP grading, a model built with features of PRAT achieved a moderate AUC of 0.711 (95% CI, 0.604-0.802) in the internal validation set, coupled with a sensitivity of 0.400 and a specificity of 0.781. While model built with combination features of the renal tumour, kidney, and PRAT showed an AUC of 0.814 (95% CI, 0.717-0.889) in the internal validation set, with a sensitivity of 0.800 and a specificity of 0.753, significantly higher than the model built with features solely from tumour lesion (0.760; 95% CI, 0.657-0.845), with a sensitivity of 0.533 and a specificity of 0.767.

CONCLUSION

Automated segmentation of kidneys and visceral adipose tissue (VAT) through TransUNet combined with a conventional image morphology processing algorithm offers a standardized approach to extract PRAT with high reproducibility. The radiomics features of PRAT and tumour lesions, along with machine learning, accurately predict the pathological grade of ccRCC and reveal the incremental significance of PRAT in this prediction.

Predicting the effort required to manually mend auto-segmentations

Auto-segmentation is one of the critical and foundational steps for medical image analysis. The quality of auto-segmentation techniques influences the efficiency of precision radiology and radiation oncology since high- quality auto-segmentations usually require limited manual correction. Segmentation metrics are necessary and important to evaluate auto-segmentation results and guide the development of auto-segmentation techniques. Currently widely applied segmentation metrics usually compare the auto-segmentation with the ground truth in terms of the overlapping area (e.g., Dice Coefficient (DC)) or the distance between boundaries (e.g., Hausdorff Distance (HD)). However, these metrics may not well indicate the manual mending effort required when observing the auto-segmentation results in clinical practice. In this article, we study different segmentation metrics to explore the appropriate way of evaluating auto-segmentations with clinical demands. The mending time for correcting auto-segmentations by experts is recorded to indicate the required mending effort. Five well-defined metrics, the overlapping area-based metric DC, the segmentation boundary distance-based metric HD, the segmentation boundary length-based metrics surface DC (surDC) and added path length (APL), and a newly proposed hybrid metric Mendability Index (MI) are discussed in the correlation analysis experiment and regression experiment. In addition to these explicitly defined metrics, we also preliminarily explore the feasibility of using deep learning models to predict the mending effort, which takes segmentation masks and the original images as the input. Experiments are conducted using datasets of 7 objects from three different institutions, which contain the original computed tomography (CT) images, the ground truth segmentations, the auto-segmentations, the corrected segmentations, and the recorded mending time. According to the correlation analysis and regression experiments for the five well-defined metrics, the variety of MI shows the best performance to indicate the mending effort for sparse objects, while the variety of HD works best when assessing the mending effort for non-sparse objects. Moreover, the deep learning models could well predict efforts required to mend auto-segmentations, even without the need of ground truth segmentations, demonstrating the potential of a novel and easy way to evaluate and boost auto-segmentation techniques.

SG-Transunet: A segmentation-guided Transformer U-Net model for KRAS gene mutation status identification in colorectal cancer

Accurately identifying the Kirsten rat sarcoma virus (KRAS) gene mutation status in colorectal cancer (CRC) patients can assist doctors in deciding whether to use specific targeted drugs for treatment. Although deep learning methods are popular, they are often affected by redundant features from non-lesion areas. Moreover, existing methods commonly extract spatial features from imaging data, which neglect important frequency domain features and may degrade the performance of KRAS gene mutation status identification. To address this deficiency, we propose a segmentation-guided Transformer U-Net (SG-Transunet) model for KRAS gene mutation status identification in CRC. Integrating the strength of convolutional neural networks (CNNs) and Transformers, SG-Transunet offers a unique approach for both lesion segmentation and KRAS mutation status identification. Specifically, for precise lesion localization, we employ an encoder-decoder to obtain segmentation results and guide the KRAS gene mutation status identification task. Subsequently, a frequency domain supplement block is designed to capture frequency domain features, integrating it with high-level spatial features extracted in the encoding path to derive advanced spatial-frequency domain features. Furthermore, we introduce a pre-trained Xception block to mitigate the risk of overfitting associated with small-scale datasets. Following this, an aggregate attention module is devised to consolidate spatial-frequency domain features with global information extracted by the Transformer at shallow and deep levels, thereby enhancing feature discriminability. Finally, we propose a mutual-constrained loss function that simultaneously constrains the segmentation mask acquisition and gene status identification process. Experimental results demonstrate the superior performance of SG-Transunet over state-of-the-art methods in discriminating KRAS gene mutation status.

Diagnosis of Forme Fruste Keratoconus Using Corvis ST Sequences with Digital Image Correlation and Machine Learning

PURPOSE

This study aimed to employ the incremental digital image correlation (DIC) method to obtain displacement and strain field data of the cornea from Corvis ST (CVS) sequences and access the performance of embedding these biomechanical data with machine learning models to distinguish forme fruste keratoconus (FFKC) from normal corneas.

METHODS

100 subjects were categorized into normal (N = 50) and FFKC (N = 50) groups. Image sequences depicting the horizontal cross-section of the human cornea under air puff were captured using the Corvis ST tonometer. The high-speed evolution of full-field corneal displacement, strain, velocity, and strain rate was reconstructed utilizing the incremental DIC approach. Maximum (max-) and average (ave-) values of full-field displacement V, shear strain γxy, velocity VR, and shear strain rate γxyR were determined over time, generating eight evolution curves denoting max-V, max-γxy, max-VR, max-γxyR, ave-V, ave-γxy, ave-VR, and ave-γxyR, respectively. These evolution data were inputted into two machine learning (ML) models, specifically Naïve Bayes (NB) and Random Forest (RF) models, which were subsequently employed to construct a voting classifier. The performance of the models in diagnosing FFKC from normal corneas was compared to existing CVS parameters.

RESULTS

The Normal group and the FFKC group each included 50 eyes. The FFKC group did not differ from healthy controls for age (p = 0.26) and gender (p = 0.36) at baseline, but they had significantly lower bIOP (p < 0.001) and thinner central cornea thickness (CCT) (p < 0.001). The results demonstrated that the proposed voting ensemble model yielded the highest performance with an AUC of 1.00, followed by the RF model with an AUC of 0.99. Radius and A2 Time emerged as the best-performing CVS parameters with AUC values of 0.948 and 0.938, respectively. Nonetheless, no existing Corvis ST parameters outperformed the ML models. A progressive enhancement in performance of the ML models was observed with incremental time points during the corneal deformation.

CONCLUSION

This study represents the first instance where displacement and strain data following incremental DIC analysis of Corvis ST images were integrated with machine learning models to effectively differentiate FFKC corneas from normal ones, achieving superior accuracy compared to existing CVS parameters. Considering biomechanical responses of the inner cornea and their temporal pattern changes may significantly improve the early detection of keratoconus.

Promoting fast MR imaging pipeline by full-stack AI

Magnetic resonance imaging (MRI) is a widely used imaging modality in clinics for medical disease diagnosis, staging, and follow-up. Deep learning has been extensively used to accelerate k-space data acquisition, enhance MR image reconstruction, and automate tissue segmentation. However, these three tasks are usually treated as independent tasks and optimized for evaluation by radiologists, thus ignoring the strong dependencies among them; this may be suboptimal for downstream intelligent processing. Here, we present a novel paradigm, full-stack learning (FSL), which can simultaneously solve these three tasks by considering the overall imaging process and leverage the strong dependence among them to further improve each task, significantly boosting the efficiency and efficacy of practical MRI workflows. Experimental results obtained on multiple open MR datasets validate the superiority of FSL over existing state-of-the-art methods on each task. FSL has great potential to optimize the practical workflow of MRI for medical diagnosis and radiotherapy.

Artificial intelligence in advancing occupational health and safety: an encapsulation of developments

OBJECTIVES

In an era characterized by dynamic technological advancements, the well-being of the workforce remains a cornerstone of progress and sustainability. The evolving industrial landscape in the modern world has had a considerable influence on occupational health and safety (OHS). Ensuring the well-being of workers and creating safe working environments are not only ethical imperatives but also integral to maintaining operational efficiency and productivity. We aim to review the advancements that have taken place with a potential to reshape workplace safety with integration of artificial intelligence (AI)-driven new technologies to prevent occupational diseases and promote safety solutions.

METHODS

The published literature was identified using scientific databases of Embase, PubMed, and Google scholar including a lower time bound of 1974 to capture chronological advances in occupational disease detection and technological solutions employed in industrial set-ups.

RESULTS

AI-driven technologies are revolutionizing how organizations approach health and safety, offering predictive insights, real-time monitoring, and risk mitigation strategies that not only minimize accidents and hazards but also pave the way for a more proactive and responsive approach to safeguarding the workforce.

CONCLUSION

As industries embrace the transformative potential of AI, a new frontier of possibilities emerges for enhancing workplace safety. This synergy between OHS and AI marks a pivotal moment in the quest for safer, healthier, and more sustainable workplaces.

The role of large language models in medical image processing: a narrative review

Background and Objective

The rapid advancement of artificial intelligence (AI) has ushered in a new era in natural language processing (NLP), with large language models (LLMs) like ChatGPT leading the way. This paper explores the profound impact of AI, particularly LLMs, in the field of medical image processing. The objective is to provide insights into the transformative potential of AI in improving healthcare by addressing historical challenges associated with manual image interpretation.

Methods

A comprehensive literature search was conducted on the Web of Science and PubMed databases from 2013 to 2023, focusing on the transformations of LLMs in Medical Imaging Processing. Recent publications on the arXiv database were also reviewed. Our search criteria included all types of articles, including abstracts, review articles, letters, and editorials. The language of publications was restricted to English to facilitate further content analysis.

Key Content and Findings

The review reveals that AI, driven by LLMs, has revolutionized medical image processing by streamlining the interpretation process, traditionally characterized by time-intensive manual efforts. AI's impact on medical care quality and patient well-being is substantial. With their robust interactivity and multimodal learning capabilities, LLMs offer immense potential for enhancing various aspects of medical image processing. Additionally, the Transformer architecture, foundational to LLMs, is gaining prominence in this domain.

Conclusions

In conclusion, this review underscores the pivotal role of AI, especially LLMs, in advancing medical image processing. These technologies have the capacity to enhance transfer learning efficiency, integrate multimodal data, facilitate clinical interactivity, and optimize cost-efficiency in healthcare. The potential applications of LLMs in clinical settings are promising, with far-reaching implications for future research, clinical practice, and healthcare policy. The transformative impact of AI in medical image processing is undeniable, and its continued development and implementation are poised to reshape the healthcare landscape for the better.

Unveiling New Strategies Facilitating the Implementation of Artificial Intelligence in Neuroimaging for the Early Detection of Alzheimer's Disease

Alzheimer's disease (AD) is a chronic neurodegenerative disorder with a global impact. The past few decades have witnessed significant strides in comprehending the underlying pathophysiological mechanisms and developing diagnostic methodologies for AD, such as neuroimaging approaches. Neuroimaging techniques, including positron emission tomography and magnetic resonance imaging, have revolutionized the field by providing valuable insights into the structural and functional alterations in the brains of individuals with AD. These imaging modalities enable the detection of early biomarkers such as amyloid-β plaques and tau protein tangles, facilitating early and precise diagnosis. Furthermore, the emerging technologies encompassing blood-based biomarkers and neurochemical profiling exhibit promising results in the identification of specific molecular signatures for AD. The integration of machine learning algorithms and artificial intelligence has enhanced the predictive capacity of these diagnostic tools when analyzing complex datasets. In this review article, we will highlight not only some of the most used diagnostic imaging approaches in neurodegeneration research but focus much more on new tools like artificial intelligence, emphasizing their application in the realm of AD. These advancements hold immense potential for early detection and intervention, thereby paving the way for personalized therapeutic strategies and ultimately augmenting the quality of life for individuals affected by AD.

Annotation-efficient training of medical image segmentation network based on scribble guidance in difficult areas

PURPOSE

The training of deep medical image segmentation networks usually requires a large amount of human-annotated data. To alleviate the burden of human labor, many semi- or non-supervised methods have been developed. However, due to the complexity of clinical scenario, insufficient training labels still causes inaccurate segmentation in some difficult local areas such as heterogeneous tumors and fuzzy boundaries.

METHODS

We propose an annotation-efficient training approach, which only requires scribble guidance in the difficult areas. A segmentation network is initially trained with a small amount of fully annotated data and then used to produce pseudo labels for more training data. Human supervisors draw scribbles in the areas of incorrect pseudo labels (i.e., difficult areas), and the scribbles are converted into pseudo label maps using a probability-modulated geodesic transform. To reduce the influence of the potential errors in the pseudo labels, a confidence map of the pseudo labels is generated by jointly considering the pixel-to-scribble geodesic distance and the network output probability. The pseudo labels and confidence maps are iteratively optimized with the update of the network, and the network training is promoted by the pseudo labels and the confidence maps in turn.

RESULTS

Cross-validation based on two data sets (brain tumor MRI and liver tumor CT) showed that our method significantly reduces the annotation time while maintains the segmentation accuracy of difficult areas (e.g., tumors). Using 90 scribble-annotated training images (annotated time: ~ 9 h), our method achieved the same performance as using 45 fully annotated images (annotation time: > 100 h) but required much shorter annotation time.

CONCLUSION

Compared to the conventional full annotation approaches, the proposed method significantly saves the annotation efforts by focusing the human supervisions on the most difficult regions. It provides an annotation-efficient way for training medical image segmentation networks in complex clinical scenario.

Attention-gated 3D CapsNet for robust hippocampal segmentation

Purpose

The hippocampus is organized in subfields (HSF) involved in learning and memory processes and widely implicated in pathologies at different ages of life, from neonatal hypoxia to temporal lobe epilepsy or Alzheimer's disease. Getting a highly accurate and robust delineation of sub-millimetric regions such as HSF to investigate anatomo-functional hypotheses is a challenge. One of the main difficulties encountered by those methodologies is related to the small size and anatomical variability of HSF, resulting in the scarcity of manual data labeling. Recently introduced, capsule networks solve analogous problems in medical imaging, providing deep learning architectures with rotational equivariance. Nonetheless, capsule networks are still two-dimensional and unassessed for the segmentation of HSF.

Approach

We released a public 3D Capsule Network (3D-AGSCaps, https://github.com/clementpoiret/3D-AGSCaps) and compared it to equivalent architectures using classical convolutions on the automatic segmentation of HSF on small and atypical datasets (incomplete hippocampal inversion, IHI). We tested 3D-AGSCaps on three datasets with manually labeled hippocampi.

Results

Our main results were: (1) 3D-AGSCaps produced segmentations with a better Dice Coefficient compared to CNNs on rotated hippocampi (p = 0.004 , cohen's d = 0.179 ); (2) on typical subjects, 3D-AGSCaps produced segmentations with a Dice coefficient similar to CNNs while having 15 times fewer parameters (2.285M versus 35.069M). This may greatly facilitate the study of atypical subjects, including healthy and pathological cases like those presenting an IHI.

Conclusion

We expect our newly introduced 3D-AGSCaps to allow a more accurate and fully automated segmentation on atypical populations, small datasets, as well as on and large cohorts where manual segmentations are nearly intractable.

Sparse annotation learning for dense volumetric MR image segmentation with uncertainty estimation

Objective.Training neural networks for pixel-wise or voxel-wise image segmentation is a challenging task that requires a considerable amount of training samples with highly accurate and densely delineated ground truth maps. This challenge becomes especially prominent in the medical imaging domain, where obtaining reliable annotations for training samples is a difficult, time-consuming, and expert-dependent process. Therefore, developing models that can perform well under the conditions of limited annotated training data is desirable.Approach.In this study, we propose an innovative framework called the extremely sparse annotation neural network (ESA-Net) that learns with only the single central slice label for 3D volumetric segmentation which explores both intra-slice pixel dependencies and inter-slice image correlations with uncertainty estimation. Specifically, ESA-Net consists of four specially designed distinct components: (1) an intra-slice pixel dependency-guided pseudo-label generation module that exploits uncertainty in network predictions while generating pseudo-labels for unlabeled slices with temporal ensembling; (2) an inter-slice image correlation-constrained pseudo-label propagation module which propagates labels from the labeled central slice to unlabeled slices by self-supervised registration with rotation ensembling; (3) a pseudo-label fusion module that fuses the two sets of generated pseudo-labels with voxel-wise uncertainty guidance; and (4) a final segmentation network optimization module to make final predictions with scoring-based label quantification.Main results.Extensive experimental validations have been performed on two popular yet challenging magnetic resonance image segmentation tasks and compared to five state-of-the-art methods.Significance.Results demonstrate that our proposed ESA-Net can consistently achieve better segmentation performances even under the extremely sparse annotation setting, highlighting its effectiveness in exploiting information from unlabeled data.

Unveiling breast cancer risk profiles: a survival clustering analysis empowered by an online web application

Aim: To develop a shiny app for doctors to investigate breast cancer treatments through a new approach by incorporating unsupervised clustering and survival information. Materials & methods: Analysis is based on the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) dataset, which contains 1726 subjects and 22 variables. Cox regression was used to identify survival risk factors for K-means clustering. Logrank tests and C-statistics were compared across different cluster numbers and Kaplan-Meier plots were presented. Results & conclusion: Our study fills an existing void by introducing a unique combination of unsupervised learning techniques and survival information on the clinician side, demonstrating the potential of survival clustering as a valuable tool in uncovering hidden structures based on distinct risk profiles.

Automating Ground Truth Annotations for Gland Segmentation Through Immunohistochemistry

Microscopic evaluation of glands in the colon is of utmost importance in the diagnosis of inflammatory bowel disease and cancer. When properly trained, deep learning pipelines can provide a systematic, reproducible, and quantitative assessment of disease-related changes in glandular tissue architecture. The training and testing of deep learning models require large amounts of manual annotations, which are difficult, time-consuming, and expensive to obtain. Here, we propose a method for automated generation of ground truth in digital hematoxylin and eosin (H&E)-stained slides using immunohistochemistry (IHC) labels. The image processing pipeline generates annotations of glands in H&E histopathology images from colon biopsy specimens by transfer of gland masks from KRT8/18, CDX2, or EPCAM IHC. The IHC gland outlines are transferred to coregistered H&E images for training of deep learning models. We compared the performance of the deep learning models to that of manual annotations using an internal held-out set of biopsy specimens as well as 2 public data sets. Our results show that EPCAM IHC provides gland outlines that closely match manual gland annotations (Dice = 0.89) and are resilient to damage by inflammation. In addition, we propose a simple data sampling technique that allows models trained on data from several sources to be adapted to a new data source using just a few newly annotated samples. The best performing models achieved average Dice scores of 0.902 and 0.89 on Gland Segmentation and Colorectal Adenocarcinoma Gland colon cancer public data sets, respectively, when trained with only 10% of annotated cases from either public cohort. Altogether, the performances of our models indicate that automated annotations using cell type-specific IHC markers can safely replace manual annotations. Automated IHC labels from single-institution cohorts can be combined with small numbers of hand-annotated cases from multi-institutional cohorts to train models that generalize well to diverse data sources.

A CT-based radiomics nomogram for predicting the progression-free survival in small cell lung cancer: a multicenter cohort study

PURPOSE

To develop a radiomics nomogram based on computed tomography (CT) to estimate progression-free survival (PFS) in patients with small cell lung cancer (SCLC) and assess its incremental value to the clinical risk factors for individual PFS estimation.

METHODS

558 patients with pathologically confirmed SCLC were retrospectively recruited from three medical centers. A radiomics signature was generated by using the Pearson correlation analysis, univariate Cox analysis, and multivariate Cox analysis. Association of the radiomics signature with PFS was evaluated. A radiomics nomogram was developed based on the radiomics signature, then its calibration, discrimination, reclassification, and clinical usefulness were evaluated.

RESULTS

In total, 6 CT radiomics features were finally selected. The radiomics signature was significantly associated with PFS (hazard ratio [HR] 4.531, 95% confidence interval [CI] 3.524-5.825, p < 0.001). Incorporating the radiomics signature into the radiomics nomogram resulted in better performance for the estimation of PFS (concordance index [C-index] 0.799) than with the clinical nomogram (C-index 0.629), as well as high 6 months and 12 months area under the curves of 0.885 and 0.846, respectively. Furthermore, the radiomics nomogram also significantly improved the classification accuracy for PFS outcomes, based on the net reclassification improvement (33.7%, 95% CI 0.216-0.609, p < 0.05) and integrated discrimination improvement (22.7%, 95% CI 0.168-0.278, p < 0.05). Decision curve analysis demonstrated that in terms of clinical usefulness, the radiomics nomogram outperformed the clinical nomogram.

CONCLUSION

A CT-based radiomics nomogram exhibited a promising performance for predicting PFS in patients with SCLC, which could provide valuable information for individualized treatment.

Application of Machine Learning Algorithm in Predicting Axillary Lymph Node Metastasis from Breast Cancer on Preoperative Chest CT

Axillary lymph node (ALN) status is one of the most critical prognostic factors in patients with breast cancer. However, ALN evaluation with contrast-enhanced CT (CECT) has been challenging. Machine learning (ML) is known to show excellent performance in image recognition tasks. The purpose of our study was to evaluate the performance of the ML algorithm for predicting ALN metastasis by combining preoperative CECT features of both ALN and primary tumor. This was a retrospective single-institutional study of a total of 266 patients with breast cancer who underwent preoperative chest CECT. Random forest (RF), extreme gradient boosting (XGBoost), and neural network (NN) algorithms were used. Statistical analysis and recursive feature elimination (RFE) were adopted as feature selection for ML. The best ML-based ALN prediction model for breast cancer was NN with RFE, which achieved an AUROC of 0.76 ± 0.11 and an accuracy of 0.74 ± 0.12. By comparing NN with RFE model performance with and without ALN features from CECT, NN with RFE model with ALN features showed better performance at all performance evaluations, which indicated the effect of ALN features. Through our study, we were able to demonstrate that the ML algorithm could effectively predict the final diagnosis of ALN metastases from CECT images of the primary tumor and ALN. This suggests that ML has the potential to differentiate between benign and malignant ALNs.

A Multi-Scale Context Aware Attention Model for Medical Image Segmentation

Medical image segmentation is critical for efficient diagnosis of diseases and treatment planning. In recent years, convolutional neural networks (CNN)-based methods, particularly U-Net and its variants, have achieved remarkable results on medical image segmentation tasks. However, they do not always work consistently on images with complex structures and large variations in regions of interest (ROI). This could be due to the fixed geometric structure of the receptive fields used for feature extraction and repetitive down-sampling operations that lead to information loss. To overcome these problems, the standard U-Net architecture is modified in this work by replacing the convolution block with a dilated convolution block to extract multi-scale context features with varying sizes of receptive fields, and adding a dilated inception block between the encoder and decoder paths to alleviate the problem of information recession and the semantic gap between features. Furthermore, the input of each dilated convolution block is added to the output through a squeeze and excitation unit, which alleviates the vanishing gradient problem and improves overall feature representation by re-weighting the channel-wise feature responses. The original inception block is modified by reducing the size of the spatial filter and introducing dilated convolution to obtain a larger receptive field. The proposed network was validated on three challenging medical image segmentation tasks with varying size ROIs: lung segmentation on chest X-ray (CXR) images, skin lesion segmentation on dermoscopy images and nucleus segmentation on microscopy cell images. Improved performance compared to state-of-the-art techniques demonstrates the effectiveness and generalisability of the proposed Dilated Convolution and Inception blocks-based U-Net (DCI-UNet).

Medical volume segmentation by overfitting sparsely annotated data

Purpose

Semantic segmentation is one of the most significant tasks in medical image computing, whereby deep neural networks have shown great success. Unfortunately, supervised approaches are very data-intensive, and obtaining reliable annotations is time-consuming and expensive. Sparsely labeled approaches, such as bounding boxes, have shown some success in reducing the annotation time. However, in 3D volume data, each slice must still be manually labeled.

Approach

We evaluate approaches that reduce the annotation effort by reducing the number of slices that need to be labeled in a 3D volume. In a two-step process, a similarity metric is used to select slices that should be annotated by a trained radiologist. In the second step, a predictor is used to predict the segmentation mask for the rest of the slices. We evaluate different combinations of selectors and predictors on medical CT and MRI volumes. Thus we can determine that combination works best, and how far slice annotations can be reduced.

Results

Our results show that for instance for the Medical Segmentation Decathlon-heart dataset, some selector, and predictor combinations allow for a Dice score 0.969 when only annotating 20% of slices per volume. Experiments on other datasets show a similarly positive trend.

Conclusions

We evaluate a method that supports experts during the labeling of 3D medical volumes. Our approach makes it possible to drastically reduce the number of slices that need to be manually labeled. We present a recommendation in which selector predictor combination to use for different tasks and goals.

DFTNet: Dual-Path Feature Transfer Network for Weakly Supervised Medical Image Segmentation

Medical image segmentation has long suffered from the problem of expensive labels. Acquiring pixel-level annotations is time-consuming, labor-intensive, and relies on extensive expert knowledge. Bounding box annotations, in contrast, are relatively easy to acquire. Thus, in this paper, we explore to segment images through a novel Dual-path Feature Transfer design with only bounding box annotations. Specifically, a Target-aware Reconstructor is proposed to extract target-related features by reconstructing the pixels within the bounding box through the channel and spatial attention module. Then, a sliding Feature Fusion and Transfer Module (FFTM) fuses the extracted features from Reconstructor and transfers them to guide the Segmentor for segmentation. Finally, we present the Confidence Ranking Loss (CRLoss) which dynamically assigns weights to the loss of each pixel based on the network's confidence. CRLoss mitigates the impact of inaccurate pseudo-labels on performance. Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance on the Medical Segmentation Decathlon (MSD) Brain Tumour and PROMISE12 datasets.

Feasibility study of adaptive radiotherapy for esophageal cancer using artificial intelligence autosegmentation based on MR-Linac

Objective

We proposed a scheme for automatic patient-specific segmentation in Magnetic Resonance (MR)-guided online adaptive radiotherapy based on daily updated, small-sample deep learning models to address the time-consuming delineation of the region of interest (ROI) in the adapt-to-shape (ATS) workflow. Additionally, we verified its feasibility in adaptive radiation therapy for esophageal cancer (EC).

Methods

Nine patients with EC who were treated with an MR-Linac were prospectively enrolled. The actual adapt-to-position (ATP) workflow and simulated ATS workflow were performed, the latter of which was embedded with a deep learning autosegmentation (AS) model. The first three treatment fractions of the manual delineations were used as input data to predict the next fraction segmentation, which was modified and then used as training data to update the model daily, forming a cyclic training process. Then, the system was validated in terms of delineation accuracy, time, and dosimetric benefit. Additionally, the air cavity in the esophagus and sternum were added to the ATS workflow (producing ATS+), and the dosimetric variations were assessed.

Results

The mean AS time was 1.40 [1.10-1.78 min]. The Dice similarity coefficient (DSC) of the AS model gradually approached 1; after four training sessions, the DSCs of all ROIs reached a mean value of 0.9 or more. Furthermore, the planning target volume (PTV) of the ATS plan showed a smaller heterogeneity index than that of the ATP plan. Additionally, V5 and V10 in the lungs and heart were greater in the ATS+ group than in the ATS group.

Conclusion

The accuracy and speed of artificial intelligence-based AS in the ATS workflow met the clinical radiation therapy needs of EC. This allowed the ATS workflow to achieve a similar speed to the ATP workflow while maintaining its dosimetric advantage. Fast and precise online ATS treatment ensured an adequate dose to the PTV while reducing the dose to the heart and lungs.

SDPN: A Slight Dual-Path Network With Local-Global Attention Guided for Medical Image Segmentation

Accurate identification of lesions is a key step in surgical planning. However, this task mainly exists two challenges: 1) Due to the complex anatomical shapes of different lesions, most segmentation methods only achieve outstanding performance for a specific structure, rather than other lesions with location differences. 2) The huge number of parameters limits existing transformer-based segmentation models. To overcome these problems, we propose a novel slight dual-path network (SDPN) to segment variable location lesions or organs with significant differences accurately. First, we design a dual-path module to integrate local with global features without obvious memory consumption. Second, a novel Multi-spectrum attention module is proposed to pay further attention to detailed information, which can automatically adapt to the variable segmentation target. Then, the compression module based on tensor ring decomposition is designed to compress convolutional and transformer structures. In the experiment, four datasets, including three benchmark datasets and a clinical dataset, are used to evaluate SDPN. Results of the experiments show that SDPN performs better than other start-of-the-art methods for brain tumor, liver tumor, endometrial tumor and cardiac segmentation. To ensure the generalizability, we train the network on Kvasir-SEG and test on CVC-ClinicDB which collected from a different institution. The quantitative analysis shows that the clinical evaluation results are consistent with the experts. Therefore, this model may be a potential candidate for the segmentation of lesions and organs segmentation with variable locations in clinical applications.

Automated LVO detection and collateral scoring on CTA using a 3D self-configuring object detection network: a multi-center study

The use of deep learning (DL) techniques for automated diagnosis of large vessel occlusion (LVO) and collateral scoring on computed tomography angiography (CTA) is gaining attention. In this study, a state-of-the-art self-configuring object detection network called nnDetection was used to detect LVO and assess collateralization on CTA scans using a multi-task 3D object detection approach. The model was trained on single-phase CTA scans of 2425 patients at five centers, and its performance was evaluated on an external test set of 345 patients from another center. Ground-truth labels for the presence of LVO and collateral scores were provided by three radiologists. The nnDetection model achieved a diagnostic accuracy of 98.26% (95% CI 96.25-99.36%) in identifying LVO, correctly classifying 339 out of 345 CTA scans in the external test set. The DL-based collateral scores had a kappa of 0.80, indicating good agreement with the consensus of the radiologists. These results demonstrate that the self-configuring 3D nnDetection model can accurately detect LVO on single-phase CTA scans and provide semi-quantitative collateral scores, offering a comprehensive approach for automated stroke diagnostics in patients with LVO.

PyMIC: A deep learning toolkit for annotation-efficient medical image segmentation

BACKGROUND AND OBJECTIVE

Open-source deep learning toolkits are one of the driving forces for developing medical image segmentation models that are essential for computer-assisted diagnosis and treatment procedures. Existing toolkits mainly focus on fully supervised segmentation that assumes full and accurate pixel-level annotations are available. Such annotations are time-consuming and difficult to acquire for segmentation tasks, which makes learning from imperfect labels highly desired for reducing the annotation cost. We aim to develop a new deep learning toolkit to support annotation-efficient learning for medical image segmentation, which can accelerate and simplify the development of deep learning models with limited annotation budget, e.g., learning from partial, sparse or noisy annotations.

METHODS

Our proposed toolkit named PyMIC is a modular deep learning library for medical image segmentation tasks. In addition to basic components that support development of high-performance models for fully supervised segmentation, it contains several advanced components that are tailored for learning from imperfect annotations, such as loading annotated and unannounced images, loss functions for unannotated, partially or inaccurately annotated images, and training procedures for co-learning between multiple networks, etc. PyMIC is built on the PyTorch framework and supports development of semi-supervised, weakly supervised and noise-robust learning methods for medical image segmentation.

RESULTS

We present several illustrative medical image segmentation tasks based on PyMIC: (1) Achieving competitive performance on fully supervised learning; (2) Semi-supervised cardiac structure segmentation with only 10% training images annotated; (3) Weakly supervised segmentation using scribble annotations; and (4) Learning from noisy labels for chest radiograph segmentation.

CONCLUSIONS

The PyMIC toolkit is easy to use and facilitates efficient development of medical image segmentation models with imperfect annotations. It is modular and flexible, which enables researchers to develop high-performance models with low annotation cost. The source code is available at:https://github.com/HiLab-git/PyMIC.

Improved Repeatability of Mouse Tibia Volume Segmentation in Murine Myelofibrosis Model Using Deep Learning

A murine model of myelofibrosis in tibia was used in a co-clinical trial to evaluate segmentation methods for application of image-based biomarkers to assess disease status. The dataset (32 mice with 157 3D MRI scans including 49 test-retest pairs scanned on consecutive days) was split into approximately 70% training, 10% validation, and 20% test subsets. Two expert annotators (EA1 and EA2) performed manual segmentations of the mouse tibia (EA1: all data; EA2: test and validation). Attention U-net (A-U-net) model performance was assessed for accuracy with respect to EA1 reference using the average Jaccard index (AJI), volume intersection ratio (AVI), volume error (AVE), and Hausdorff distance (AHD) for four training scenarios: full training, two half-splits, and a single-mouse subsets. The repeatability of computer versus expert segmentations for tibia volume of test-retest pairs was assessed by within-subject coefficient of variance (%wCV). A-U-net models trained on full and half-split training sets achieved similar average accuracy (with respect to EA1 annotations) for test set: AJI = 83-84%, AVI = 89-90%, AVE = 2-3%, and AHD = 0.5 mm-0.7 mm, exceeding EA2 accuracy: AJ = 81%, AVI = 83%, AVE = 14%, and AHD = 0.3 mm. The A-U-net model repeatability wCV [95% CI]: 3 [2, 5]% was notably better than that of expert annotators EA1: 5 [4, 9]% and EA2: 8 [6, 13]%. The developed deep learning model effectively automates murine bone marrow segmentation with accuracy comparable to human annotators and substantially improved repeatability.

Evaluating semi-supervision methods for medical image segmentation: applications in cardiac magnetic resonance imaging

Purpose

Neural networks have potential to automate medical image segmentation but require expensive labeling efforts. While methods have been proposed to reduce the labeling burden, most have not been thoroughly evaluated on large, clinical datasets or clinical tasks. We propose a method to train segmentation networks with limited labeled data and focus on thorough network evaluation.

Approach

We propose a semi-supervised method that leverages data augmentation, consistency regularization, and pseudolabeling and train four cardiac magnetic resonance (MR) segmentation networks. We evaluate the models on multiinstitutional, multiscanner, multidisease cardiac MR datasets using five cardiac functional biomarkers, which are compared to an expert's measurements using Lin's concordance correlation coefficient (CCC), the within-subject coefficient of variation (CV), and the Dice coefficient.

Results

The semi-supervised networks achieve strong agreement using Lin's CCC ( > 0.8 ), CV similar to an expert, and strong generalization performance. We compare the error modes of the semi-supervised networks against fully supervised networks. We evaluate semi-supervised model performance as a function of labeled training data and with different types of model supervision, showing that a model trained with 100 labeled image slices can achieve a Dice coefficient within 1.10% of a network trained with 16,000+ labeled image slices.

Conclusion

We evaluate semi-supervision for medical image segmentation using heterogeneous datasets and clinical metrics. As methods for training models with little labeled data become more common, knowledge about how they perform on clinical tasks, how they fail, and how they perform with different amounts of labeled data is useful to model developers and users.

A Soft Label Method for Medical Image Segmentation with Multirater Annotations

In medical image analysis, collecting multiple annotations from different clinical raters is a typical practice to mitigate possible diagnostic errors. For such multirater labels' learning problems, in addition to majority voting, it is a common practice to use soft labels in the form of full-probability distributions obtained by averaging raters as ground truth to train the model, which benefits from uncertainty contained in soft labels. However, the potential information contained in soft labels is rarely studied, which may be the key to improving the performance of medical image segmentation with multirater annotations. In this work, we aim to improve soft label methods by leveraging interpretable information from multiraters. Considering that mis-segmentation occurs in areas with weak supervision of annotations and high difficulty of images, we propose to reduce the reliance on local uncertain soft labels and increase the focus on image features. Therefore, we introduce local self-ensembling learning with consistency regularization, forcing the model to concentrate more on features rather than annotations, especially in regions with high uncertainty measured by the pixelwise interclass variance. Furthermore, we utilize a label smoothing technique to flatten each rater's annotation, alleviating overconfidence of structural edges in annotations. Without introducing additional parameters, our method improves the accuracy of the soft label baseline by 4.2% and 2.7% on a synthetic dataset and a fundus dataset, respectively. In addition, quantitative comparisons show that our method consistently outperforms existing multirater strategies as well as state-of-the-art methods. This work provides a simple yet effective solution for the widespread multirater label segmentation problems in clinical diagnosis.

Development and Application of a Standardized Testset for an Artificial Intelligence Medical Device Intended for the Computer-Aided Diagnosis of Diabetic Retinopathy

Objective

To explore a centralized approach to build test sets and assess the performance of an artificial intelligence medical device (AIMD) which is intended for computer-aided diagnosis of diabetic retinopathy (DR).

Method

A framework was proposed to conduct data collection, data curation, and annotation. Deidentified colour fundus photographs were collected from 11 partner hospitals with raw labels. Photographs with sensitive information or authenticity issues were excluded during vetting. A team of annotators was recruited through qualification examinations and trained. The annotation process included three steps: initial annotation, review, and arbitration. The annotated data then composed a standardized test set, which was further imported to algorithms under test (AUT) from different developers. The algorithm outputs were compared with the final annotation results (reference standard).

Result

The test set consists of 6327 digital colour fundus photographs. The final labels include 5 stages of DR and non-DR, as well as other ocular diseases and photographs with unacceptable quality. The Fleiss Kappa was 0.75 among the annotators. The Cohen's kappa between raw labels and final labels is 0.5. Using this test set, five AUTs were tested and compared quantitatively. The metrics include accuracy, sensitivity, and specificity. The AUTs showed inhomogeneous capabilities to classify different types of fundus photographs.

Conclusions

This article demonstrated a workflow to build standardized test sets and conduct algorithm testing of the AIMD for computer-aided diagnosis of diabetic retinopathy. It may provide a reference to develop technical standards that promote product verification and quality control, improving the comparability of products.

Visual ensemble selection of deep convolutional neural networks for 3D segmentation of breast tumors on dynamic contrast enhanced MRI

OBJECTIVES

To develop a visual ensemble selection of deep convolutional neural networks (CNN) for 3D segmentation of breast tumors using T1-weighted dynamic contrast-enhanced (T1-DCE) MRI.

METHODS

Multi-center 3D T1-DCE MRI (n = 141) were acquired for a cohort of patients diagnosed with locally advanced or aggressive breast cancer. Tumor lesions of 111 scans were equally divided between two radiologists and segmented for training. The additional 30 scans were segmented independently by both radiologists for testing. Three 3D U-Net models were trained using either post-contrast images or a combination of post-contrast and subtraction images fused at either the image or the feature level. Segmentation accuracy was evaluated quantitatively using the Dice similarity coefficient (DSC) and the Hausdorff distance (HD95) and scored qualitatively by a radiologist as excellent, useful, helpful, or unacceptable. Based on this score, a visual ensemble approach selecting the best segmentation among these three models was proposed.

RESULTS

The mean and standard deviation of DSC and HD95 between the two radiologists were equal to 77.8 ± 10.0% and 5.2 ± 5.9 mm. Using the visual ensemble selection, a DSC and HD95 equal to 78.1 ± 16.2% and 14.1 ± 40.8 mm was reached. The qualitative assessment was excellent (resp. excellent or useful) in 50% (resp. 77%).

CONCLUSION

Using subtraction images in addition to post-contrast images provided complementary information for 3D segmentation of breast lesions by CNN. A visual ensemble selection allowing the radiologist to select the most optimal segmentation obtained by the three 3D U-Net models achieved comparable results to inter-radiologist agreement, yielding 77% segmented volumes considered excellent or useful.

KEY POINTS

• Deep convolutional neural networks were developed using T1-weighted post-contrast and subtraction MRI to perform automated 3D segmentation of breast tumors. • A visual ensemble selection allowing the radiologist to choose the best segmentation among the three 3D U-Net models outperformed each of the three models. • The visual ensemble selection provided clinically useful segmentations in 77% of cases, potentially allowing for a valuable reduction of the manual 3D segmentation workload for the radiologist and greatly facilitating quantitative studies on non-invasive biomarker in breast MRI.