Evaluation of multislice inputs to convolutional neural networks for medical image segmentation

Cross-sectional angle prediction of lipid-rich and calcified tissue on computed tomography angiography images

PURPOSE

The assessment of vulnerable plaque characteristics and distribution is important to stratify cardiovascular risk in a patient. Computed tomography angiography (CTA) offers a promising alternative to invasive imaging but is limited by the fact that the range of Hounsfield units (HU) in lipid-rich areas overlaps with the HU range in fibrotic tissue and that the HU range of calcified plaques overlaps with the contrast within the contrast-filled lumen. This paper is to investigate whether lipid-rich and calcified plaques can be detected more accurately on cross-sectional CTA images using deep learning methodology.

METHODS

Two deep learning (DL) approaches are proposed, a 2.5D Dense U-Net and 2.5D Mask-RCNN, which separately perform the cross-sectional plaque detection in the Cartesian and polar domain. The spread-out view is used to evaluate and show the prediction result of the plaque regions. The accuracy and F1-score are calculated on a lesion level for the DL and conventional plaque detection methods.

RESULTS

For the lipid-rich plaques, the median and mean values of the F1-score calculated by the two proposed DL methods on 91 lesions were approximately 6 and 3 times higher than those of the conventional method. For the calcified plaques, the F1-score of the proposed methods was comparable to those of the conventional method. The median F1-score of the Dense U-Net-based method was 3% higher than that of the conventional method.

CONCLUSION

The two methods proposed in this paper contribute to finer cross-sectional predictions of lipid-rich and calcified plaques compared to studies focusing only on longitudinal prediction. The angular prediction performance of the proposed methods outperforms the convincing conventional method for lipid-rich plaque and is comparable for calcified plaque.

Transformer-based deep learning denoising of single and multi-delay 3D arterial spin labeling

PURPOSE

To present a Swin Transformer-based deep learning (DL) model (SwinIR) for denoising single-delay and multi-delay 3D arterial spin labeling (ASL) and compare its performance with convolutional neural network (CNN) and other Transformer-based methods.

METHODS

SwinIR and CNN-based spatial denoising models were developed for single-delay ASL. The models were trained on 66 subjects (119 scans) and tested on 39 subjects (44 scans) from three different vendors. Spatiotemporal denoising models were developed using another dataset (6 subjects, 10 scans) of multi-delay ASL. A range of input conditions was tested for denoising single and multi-delay ASL, respectively. The performance was evaluated using similarity metrics, spatial SNR and quantification accuracy of cerebral blood flow (CBF), and arterial transit time (ATT).

RESULTS

SwinIR outperformed CNN and other Transformer-based networks, whereas pseudo-3D models performed better than 2D models for denoising single-delay ASL. The similarity metrics and image quality (SNR) improved with more slices in pseudo-3D models and further improved when using M0 as input, but introduced greater biases for CBF quantification. Pseudo-3D models with three slices achieved optimal balance between SNR and accuracy, which can be generalized to different vendors. For multi-delay ASL, spatiotemporal denoising models had better performance than spatial-only models with reduced biases in fitted CBF and ATT maps.

CONCLUSIONS

SwinIR provided better performance than CNN and other Transformer-based methods for denoising both single and multi-delay 3D ASL data. The proposed model offers flexibility to improve image quality and/or reduce scan time for 3D ASL to facilitate its clinical use.

Deep learning algorithm performance in contouring head and neck organs at risk: a systematic review and single-arm meta-analysis

PURPOSE

The contouring of organs at risk (OARs) in head and neck cancer radiation treatment planning is a crucial, yet repetitive and time-consuming process. Recent studies have applied deep learning (DL) algorithms to automatically contour head and neck OARs. This study aims to conduct a systematic review and meta-analysis to summarize and analyze the performance of DL algorithms in contouring head and neck OARs. The objective is to assess the advantages and limitations of DL algorithms in contour planning of head and neck OARs.

METHODS

This study conducted a literature search of Pubmed, Embase and Cochrane Library databases, to include studies related to DL contouring head and neck OARs, and the dice similarity coefficient (DSC) of four categories of OARs from the results of each study are selected as effect sizes for meta-analysis. Furthermore, this study conducted a subgroup analysis of OARs characterized by image modality and image type.

RESULTS

149 articles were retrieved, and 22 studies were included in the meta-analysis after excluding duplicate literature, primary screening, and re-screening. The combined effect sizes of DSC for brainstem, spinal cord, mandible, left eye, right eye, left optic nerve, right optic nerve, optic chiasm, left parotid, right parotid, left submandibular, and right submandibular are 0.87, 0.83, 0.92, 0.90, 0.90, 0.71, 0.74, 0.62, 0.85, 0.85, 0.82, and 0.82, respectively. For subgroup analysis, the combined effect sizes for segmentation of the brainstem, mandible, left optic nerve, and left parotid gland using CT and MRI images are 0.86/0.92, 0.92/0.90, 0.71/0.73, and 0.84/0.87, respectively. Pooled effect sizes using 2D and 3D images of the brainstem, mandible, left optic nerve, and left parotid gland for contouring are 0.88/0.87, 0.92/0.92, 0.75/0.71 and 0.87/0.85.

CONCLUSIONS

The use of automated contouring technology based on DL algorithms is an essential tool for contouring head and neck OARs, achieving high accuracy, reducing the workload of clinical radiation oncologists, and providing individualized, standardized, and refined treatment plans for implementing "precision radiotherapy". Improving DL performance requires the construction of high-quality data sets and enhancing algorithm optimization and innovation.

Novel near-infrared spectroscopy-intravascular ultrasound-based deep-learning methodology for accurate coronary computed tomography plaque quantification and characterization

Aims

Coronary computed tomography angiography (CCTA) is inferior to intravascular imaging in detecting plaque morphology and quantifying plaque burden. We aim to, for the first time, train a deep-learning (DL) methodology for accurate plaque quantification and characterization in CCTA using near-infrared spectroscopy-intravascular ultrasound (NIRS-IVUS).

Methods and results

Seventy patients were prospectively recruited who underwent CCTA and NIRS-IVUS imaging. Corresponding cross sections were matched using an in-house developed software, and the estimations of NIRS-IVUS for the lumen, vessel wall borders, and plaque composition were used to train a convolutional neural network in 138 vessels. The performance was evaluated in 48 vessels and compared against the estimations of NIRS-IVUS and the conventional CCTA expert analysis. Sixty-four patients (186 vessels, 22 012 matched cross sections) were included. Deep-learning methodology provided estimations that were closer to NIRS-IVUS compared with the conventional approach for the total atheroma volume (ΔDL-NIRS-IVUS: -37.8 ± 89.0 vs. ΔConv-NIRS-IVUS: 243.3 ± 183.7 mm3, variance ratio: 4.262, P < 0.001) and percentage atheroma volume (-3.34 ± 5.77 vs. 17.20 ± 7.20%, variance ratio: 1.578, P < 0.001). The DL methodology detected lesions more accurately than the conventional approach (Area under the curve (AUC): 0.77 vs. 0.67, P < 0.001) and quantified minimum lumen area (ΔDL-NIRS-IVUS: -0.35 ± 1.81 vs. ΔConv-NIRS-IVUS: 1.37 ± 2.32 mm2, variance ratio: 1.634, P < 0.001), maximum plaque burden (4.33 ± 11.83% vs. 5.77 ± 16.58%, variance ratio: 2.071, P = 0.004), and calcific burden (-51.2 ± 115.1 vs. -54.3 ± 144.4, variance ratio: 2.308, P < 0.001) more accurately than conventional approach. The DL methodology was able to segment a vessel on CCTA in 0.3 s.

Conclusions

The DL methodology developed for CCTA analysis from co-registered NIRS-IVUS and CCTA data enables rapid and accurate assessment of lesion morphology and is superior to expert analysts (Clinicaltrials.gov: NCT03556644).

Transformer based deep learning denoising of single and multi-delay 3D Arterial Spin Labeling

Purpose

To present a Swin Transformer-based deep learning (DL) model for denoising of single-delay and multi-delay 3D arterial spin labeling (ASL) and compare its performance with convolutional neural network (CNN) methods.

Methods

Swin Transformer and CNN-based spatial denoising models were developed for single-delay ASL. The models were trained on 59 subjects (104 scans) and tested on 44 subjects (57 scans) from 3 different vendors. Spatiotemporal denoising models were developed using another dataset (6 subjects, 10 scans) of multi-delay ASL. A range of input conditions was tested for denoising single and multi-delay ASL respectively. The performance was evaluated using similarity metrics, spatial signal-to-noise ratio (SNR) and quantification accuracy of cerebral blood flow (CBF) and arterial transit time (ATT).

Results

Swin Transformer outperformed CNN-based networks, whereas pseudo-3D models showed better performance than 2D models for denoising single-delay ASL. The similarity metrics and image quality (SNR) improved with more slices in pseudo-3D models, and further improved when using M0 as input but introduced greater biases for CBF quantification. Pseudo-3D models with 3 slices as input achieved optimal balance between SNR and accuracy, which can be generalized to different vendors. For multi-delay, spatiotemporal denoising models had better performance than spatial-only models with reduced biases in fitted CBF and ATT maps.

Conclusions

Swin Transformer DL models provided better performance than CNN methods for denoising both single and multi-delay 3D ASL data. The proposed model offers flexibility to improve image quality and/or reduce scan time for 3D ASL to facilitate its clinical use.

Interobserver Variability Prediction of Primary Gross Tumor in a Patient with Non-Small Cell Lung Cancer

This research addresses the problem of interobserver variability (IOV), in which different oncologists manually delineate varying primary gross tumor volume (pGTV) contours, adding risk to targeted radiation treatments. Thus, a method of IOV reduction is urgently needed. Hypothesizing that the radiation oncologist’s IOV may shrink with the aid of IOV maps, we propose IOV prediction network (IOV-Net), a deep-learning model that uses the fuzzy membership function to produce high-quality maps based on computed tomography (CT) images. To test the prediction accuracy, a ground-truth pGTV IOV map was created using the manual contour delineations of radiation therapy structures provided by five expert oncologists. Then, we tasked IOV-Net with producing a map of its own. The mean squared error (prediction vs. ground truth) and its standard deviation were 0.0038 and 0.0005, respectively. To test the clinical feasibility of our method, CT images were divided into two groups, and oncologists from our institution created manual contours with and without IOV map guidance. The Dice similarity coefficient and Jaccard index increased by ~6 and 7%, respectively, and the Hausdorff distance decreased by 2.5 mm, indicating a statistically significant IOV reduction (p < 0.05). Hence, IOV-net and its resultant IOV maps have the potential to improve radiation therapy efficacy worldwide.

SIP-UNet: Sequential Inputs Parallel UNet Architecture for Segmentation of Brain Tissues from Magnetic Resonance Images

Proper analysis of changes in brain structure can lead to a more accurate diagnosis of specific brain disorders. The accuracy of segmentation is crucial for quantifying changes in brain structure. In recent studies, UNet-based architectures have outperformed other deep learning architectures in biomedical image segmentation. However, improving segmentation accuracy is challenging due to the low resolution of medical images and insufficient data. In this study, we present a novel architecture that combines three parallel UNets using a residual network. This architecture improves upon the baseline methods in three ways. First, instead of using a single image as input, we use three consecutive images. This gives our model the freedom to learn from neighboring images as well. Additionally, the images are individually compressed and decompressed using three different UNets, which prevents the model from merging the features of the images. Finally, following the residual network architecture, the outputs of the UNets are combined in such a way that the features of the image corresponding to the output are enhanced by a skip connection. The proposed architecture performed better than using a single conventional UNet and other UNet variants.

A Data-Adaptive Loss Function for Incomplete Data and Incremental Learning in Semantic Image Segmentation

In the last years, deep learning has dramatically improved the performances in a variety of medical image analysis applications. Among different types of deep learning models, convolutional neural networks have been among the most successful and they have been used in many applications in medical imaging. Training deep convolutional neural networks often requires large amounts of image data to generalize well to new unseen images. It is often time-consuming and expensive to collect large amounts of data in the medical image domain due to expensive imaging systems, and the need for experts to manually make ground truth annotations. A potential problem arises if new structures are added when a decision support system is already deployed and in use. Since the field of radiation therapy is constantly developing, the new structures would also have to be covered by the decision support system. In the present work, we propose a novel loss function to solve multiple problems: imbalanced datasets, partially-labeled data, and incremental learning. The proposed loss function adapts to the available data in order to utilize all available data, even when some have missing annotations. We demonstrate that the proposed loss function also works well in an incremental learning setting, where an existing model is easily adapted to semi-automatically incorporate delineations of new organs when they appear. Experiments on a large in-house dataset show that the proposed method performs on par with baseline models, while greatly reducing the training time and eliminating the hassle of maintaining multiple models in practice.

Deep Diffusion MRI Registration (DDMReg): A Deep Learning Method for Diffusion MRI Registration

In this paper, we present a deep learning method, DDMReg, for accurate registration between diffusion MRI (dMRI) datasets. In dMRI registration, the goal is to spatially align brain anatomical structures while ensuring that local fiber orientations remain consistent with the underlying white matter fiber tract anatomy. DDMReg is a novel method that uses joint whole-brain and tract-specific information for dMRI registration. Based on the successful VoxelMorph framework for image registration, we propose a novel registration architecture that leverages not only whole brain information but also tract-specific fiber orientation information. DDMReg is an unsupervised method for deformable registration between pairs of dMRI datasets: it does not require nonlinearly pre-registered training data or the corresponding deformation fields as ground truth. We perform comparisons with four state-of-the-art registration methods on multiple independently acquired datasets from different populations (including teenagers, young and elderly adults) and different imaging protocols and scanners. We evaluate the registration performance by assessing the ability to align anatomically corresponding brain structures and ensure fiber spatial agreement between different subjects after registration. Experimental results show that DDMReg obtains significantly improved registration performance compared to the state-of-the-art methods. Importantly, we demonstrate successful generalization of DDMReg to dMRI data from different populations with varying ages and acquired using different acquisition protocols and different scanners.

Automatic evaluation of graft orientation during Descemet membrane endothelial keratoplasty using intraoperative OCT

Correct Descemet Membrane Endothelial Keratoplasty (DMEK) graft orientation is imperative for success of DMEK surgery, but intraoperative evaluation can be challenging. We present a method for automatic evaluation of the graft orientation in intraoperative optical coherence tomography (iOCT), exploiting the natural rolling behavior of the graft. The method encompasses a deep learning model for graft segmentation, post-processing to obtain a smooth line representation, and curvature calculations to determine graft orientation. For an independent test set of 100 iOCT-frames, the automatic method correctly identified graft orientation in 78 frames and obtained an area under the receiver operating characteristic curve (AUC) of 0.84. When we replaced the automatic segmentation with the manual masks, the AUC increased to 0.92, corresponding to an accuracy of 86%. In comparison, two corneal specialists correctly identified graft orientation in 90% and 91% of the iOCT-frames.

A Novel and Automated Approach to Classify Radiation Induced Lung Tissue Damage on CT Scans

Radiation-induced lung damage (RILD) is a common side effect of radiotherapy (RT). The ability to automatically segment, classify, and quantify different types of lung parenchymal change is essential to uncover underlying patterns of RILD and their evolution over time. A RILD dedicated tissue classification system was developed to describe lung parenchymal tissue changes on a voxel-wise level. The classification system was automated for segmentation of five lung tissue classes on computed tomography (CT) scans that described incrementally increasing tissue density, ranging from normal lung (Class 1) to consolidation (Class 5). For ground truth data generation, we employed a two-stage data annotation approach, akin to active learning. Manual segmentation was used to train a stage one auto-segmentation method. These results were manually refined and used to train the stage two auto-segmentation algorithm. The stage two auto-segmentation algorithm was an ensemble of six 2D Unets using different loss functions and numbers of input channels. The development dataset used in this study consisted of 40 cases, each with a pre-radiotherapy, 3-, 6-, 12-, and 24-month follow-up CT scans (n = 200 CT scans). The method was assessed on a hold-out test dataset of 6 cases (n = 30 CT scans). The global Dice score coefficients (DSC) achieved for each tissue class were: Class (1) 99% and 98%, Class (2) 71% and 44%, Class (3) 56% and 26%, Class (4) 79% and 47%, and Class (5) 96% and 92%, for development and test subsets, respectively. The lowest values for the test subsets were caused by imaging artefacts or reflected subgroups that occurred infrequently and with smaller overall parenchymal volumes. We performed qualitative evaluation on the test dataset presenting manual and auto-segmentation to a blinded independent radiologist to rate them as 'acceptable', 'minor disagreement' or 'major disagreement'. The auto-segmentation ratings were similar to the manual segmentation, both having approximately 90% of cases rated as acceptable. The proposed framework for auto-segmentation of different lung tissue classes produces acceptable results in the majority of cases and has the potential to facilitate future large studies of RILD.

Liver segmentation from computed tomography images using cascade deep learning

BACKGROUND

Liver segmentation is a fundamental step in the treatment planning and diagnosis of liver cancer. However, manual segmentation of liver is time-consuming because of the large slice quantity and subjectiveness associated with the specialist's experience, which can lead to segmentation errors. Thus, the segmentation process can be automated using computational methods for better time efficiency and accuracy. However, automatic liver segmentation is a challenging task, as the liver can vary in shape, ill-defined borders, and lesions, which affect its appearance. We aim to propose an automatic method for liver segmentation using computed tomography (CT) images.

METHODS

The proposed method, based on deep convolutional neural network models and image processing techniques, comprise of four main steps: (1) image preprocessing, (2) initial segmentation, (3) reconstruction, and (4) final segmentation.

RESULTS

We evaluated the proposed method using 131 CT images from the LiTS image base. An average sensitivity of 95.45%, an average specificity of 99.86%, an average Dice coefficient of 95.64%, an average volumetric overlap error (VOE) of 8.28%, an average relative volume difference (RVD) of -0.41%, and an average Hausdorff distance (HD) of 26.60 mm were achieved.

CONCLUSIONS

This study demonstrates that liver segmentation, even when lesions are present in CT images, can be efficiently performed using a cascade approach and including a reconstruction step based on deep convolutional neural networks.

The impact of training sample size on deep learning-based organ auto-segmentation for head-and-neck patients

To investigate the impact of training sample size on the performance of deep learning-based organ auto-segmentation for head-and-neck cancer patients, a total of 1160 patients with head-and-neck cancer who received radiotherapy were enrolled in this study. Patient planning CT images and regions of interest (ROIs) delineation, including the brainstem, spinal cord, eyes, lenses, optic nerves, temporal lobes, parotids, larynx and body, were collected. An evaluation dataset with 200 patients were randomly selected and combined with Dice similarity index to evaluate the model performances. Eleven training datasets with different sample sizes were randomly selected from the remaining 960 patients to form auto-segmentation models. All models used the same data augmentation methods, network structures and training hyperparameters. A performance estimation model of the training sample size based on the inverse power law function was established. Different performance change patterns were found for different organs. Six organs had the best performance with 800 training samples and others achieved their best performance with 600 training samples or 400 samples. The benefit of increasing the size of the training dataset gradually decreased. Compared to the best performance, optic nerves and lenses reached 95% of their best effect at 200, and the other organs reached 95% of their best effect at 40. For the fitting effect of the inverse power law function, the fitted root mean square errors of all ROIs were less than 0.03 (left eye: 0.024, others: <0.01), and theRsquare of all ROIs except for the body was greater than 0.5. The sample size has a significant impact on the performance of deep learning-based auto-segmentation. The relationship between sample size and performance depends on the inherent characteristics of the organ. In some cases, relatively small samples can achieve satisfactory performance.