Attention-enabled 3D boosted convolutional neural networks for semantic CT segmentation using deep supervision

Automated contouring of CTV and OARs in planning CT scans using novel hybrid convolution-transformer networks for prostate cancer radiotherapy

PURPOSE OBJECTIVE(S)

Manual contouring of the prostate region in planning computed tomography (CT) images is a challenging task due to factors such as low contrast in soft tissues, inter- and intra-observer variability, and variations in organ size and shape. Consequently, the use of automated contouring methods can offer significant advantages. In this study, we aimed to investigate automated male pelvic multi-organ contouring in multi-center planning CT images using a hybrid convolutional neural network-vision transformer (CNN-ViT) that combines convolutional and ViT techniques.

MATERIALS/METHODS

We used retrospective data from 104 localized prostate cancer patients, with delineations of the clinical target volume (CTV) and critical organs at risk (OAR) for external beam radiotherapy. We introduced a novel attention-based fusion module that merges detailed features extracted through convolution with the global features obtained through the ViT.

RESULTS

The average dice similarity coefficients (DSCs) achieved by VGG16-UNet-ViT for the prostate, bladder, rectum, right femoral head (RFH), and left femoral head (LFH) were 91.75%, 95.32%, 87.00%, 96.30%, and 96.34%, respectively. Experiments conducted on multi-center planning CT images indicate that combining the ViT structure with the CNN network resulted in superior performance for all organs compared to pure CNN and transformer architectures. Furthermore, the proposed method achieves more precise contours compared to state-of-the-art techniques.

CONCLUSION

Results demonstrate that integrating ViT into CNN architectures significantly improves segmentation performance. These results show promise as a reliable and efficient tool to facilitate prostate radiotherapy treatment planning.

Novel dosimetric validation of a commercial CT scanner based deep learning automated contour solution for prostate radiotherapy

PURPOSE

OAR delineation accuracy influences: (i) a patient's optimised dose distribution (PD), (ii) the reported doses (RD) presented at approval, which represent plan quality. This study utilised a novel dosimetric validation methodology, comprehensively evaluating a new CT-scanner-based AI contouring solution in terms of PD and RD within an automated planning workflow.

METHODS

20 prostate patients were selected to evaluate AI contouring for rectum, bladder, and proximal femurs. Five planning 'pipelines' were considered; three using AI contours with differing levels of manual editing (nominally none (AIStd), minor editing in specific regions (AIMinEd), and fully corrected (AIFullEd)). Remaining pipelines were manual delineations from two observers (MDOb1, MDOb2). Automated radiotherapy plans were generated for each pipeline. Geometric and dosimetric agreement of contour sets AIStd, AIMinEd, AIFullEd and MDOb2 were evaluated against the reference set MDOb1. Non-inferiority of AI pipelines was assessed, hypothesising that compared to MDOb1, absolute deviations in metrics for AI contouring were no greater than that from MDOb2.

RESULTS

Compared to MDOb1, organ delineation time was reduced by 24.9 min (96 %), 21.4 min (79 %) and 12.2 min (45 %) for AIStd, AIMinEd and AIFullEd respectively. All pipelines exhibited generally good dosimetric agreement with MDOb1. For RD, median deviations were within ± 1.8 cm3, ± 1.7 % and ± 0.6 Gy for absolute volume, relative volume and mean dose metrics respectively. For PD, respective values were within ± 0.4 cm3, ± 0.5 % and ± 0.2 Gy. Statistically (p < 0.05), AIMinEd and AIFullEd were dosimetrically non-inferior to MDOb2.

CONCLUSIONS

This novel dosimetric validation demonstrated that following targeted minor editing (AIMinEd), AI contours were dosimetrically non-inferior to manual delineations, reducing delineation time by 79 %.

Attention-Gated Deep-Learning-Based Automatic Digitization of Interstitial Needles in High-Dose-Rate Brachytherapy for Cervical Cancer

Purpose

Deep learning can be used to automatically digitize interstitial needles in high-dose-rate (HDR) brachytherapy for patients with cervical cancer. The aim of this study was to design a novel attention-gated deep-learning model, which may further improve the accuracy of and better differentiate needles.

Methods and Materials

Seventeen patients with cervical cancer with 56 computed tomography-based interstitial HDR brachytherapy plans from the local hospital were retrospectively chosen with the local institutional review board's approval. Among them, 50 plans were randomly selected as the training set and the rest as the validation set. Spatial and channel attention gates (AGs) were added to 3-dimensional convolutional neural networks (CNNs) to highlight needle features and suppress irrelevant regions; this was supposed to facilitate convergence and improve accuracy of automatic needle digitization. Subsequently, the automatically digitized needles were exported to the Oncentra treatment planning system (Elekta Solutions AB, Stockholm, Sweden) for dose evaluation. The geometric and dosimetric accuracy of automatic needle digitization was compared among 3 methods: (1) clinically approved plans with manual needle digitization (ground truth); (2) the conventional deep-learning (CNN) method; and (3) the attention-added deep-learning (CNN + AG) method, in terms of the Dice similarity coefficient (DSC), tip and shaft positioning errors, dose distribution in the high-risk clinical target volume (HR-CTV), organs at risk, and so on.

Results

The attention-gated CNN model was superior to CNN without AGs, with a greater DSC (approximately 94% for CNN + AG vs 89% for CNN). The needle tip and shaft errors of the CNN + AG method (1.1 mm and 1.8 mm, respectively) were also much smaller than those of the CNN method (2.0 mm and 3.3 mm, respectively). Finally, the dose difference for the HR-CTV D90 using the CNN + AG method was much more accurate than that using CNN (0.4% and 1.7%, respectively).

Conclusions

The attention-added deep-learning model was successfully implemented for automatic needle digitization in HDR brachytherapy, with clinically acceptable geometric and dosimetric accuracy. Compared with conventional deep-learning neural networks, attention-gated deep learning may have superior performance and great clinical potential.

A new architecture combining convolutional and transformer-based networks for automatic 3D multi-organ segmentation on CT images

PURPOSE

Deep learning-based networks have become increasingly popular in the field of medical image segmentation. The purpose of this research was to develop and optimize a new architecture for automatic segmentation of the prostate gland and normal organs in the pelvic, thoracic, and upper gastro-intestinal (GI) regions.

METHODS

We developed an architecture which combines a shifted-window (Swin) transformer with a convolutional U-Net. The network includes a parallel encoder, a cross-fusion block, and a CNN-based decoder to extract local and global information and merge related features on the same scale. A skip connection is applied between the cross-fusion block and decoder to integrate low-level semantic features. Attention gates (AGs) are integrated within the CNN to suppress features in image background regions. Our network is termed "SwinAttUNet." We optimized the architecture for automatic image segmentation. Training datasets consisted of planning-CT datasets from 300 prostate cancer patients from an institutional database and 100 CT datasets from a publicly available dataset (CT-ORG). Images were linearly interpolated and resampled to a spatial resolution of (1.0 × 1.0× 1.5) mm3 . A volume patch (192 × 192 × 96) was used for training and inference, and the dataset was split into training (75%), validation (10%), and test (15%) cohorts. Data augmentation transforms were applied consisting of random flip, rotation, and intensity scaling. The loss function comprised Dice and cross-entropy equally weighted and summed. We evaluated Dice coefficients (DSC), 95th percentile Hausdorff Distances (HD95), and Average Surface Distances (ASD) between results of our network and ground truth data.

RESULTS

SwinAttUNet, DSC values were 86.54 ± 1.21, 94.15 ± 1.17, and 87.15 ± 1.68% and HD95 values were 5.06 ± 1.42, 3.16 ± 0.93, and 5.54 ± 1.63 mm for the prostate, bladder, and rectum, respectively. Respective ASD values were 1.45 ± 0.57, 0.82 ± 0.12, and 1.42 ± 0.38 mm. For the lung, liver, kidneys and pelvic bones, respective DSC values were: 97.90 ± 0.80, 96.16 ± 0.76, 93.74 ± 2.25, and 89.31 ± 3.87%. Respective HD95 values were: 5.13 ± 4.11, 2.73 ± 1.19, 2.29 ± 1.47, and 5.31 ± 1.25 mm. Respective ASD values were: 1.88 ± 1.45, 1.78 ± 1.21, 0.71 ± 0.43, and 1.21 ± 1.11 mm. Our network outperformed several existing deep learning approaches using only attention-based convolutional or Transformer-based feature strategies, as detailed in the results section.

CONCLUSIONS

We have demonstrated that our new architecture combining Transformer- and convolution-based features is able to better learn the local and global context for automatic segmentation of multi-organ, CT-based anatomy.

Real-world validation of Artificial Intelligence-based Computed Tomography auto-contouring for prostate cancer radiotherapy planning

Background and purpose

Artificial Intelligence (AI)-based auto-contouring for treatment planning in radiotherapy needs extensive clinical validation, including the impact of editing after automatic segmentation. The aims of this study were to assess the performance of a commercial system for Clinical Target Volumes (CTVs) (prostate/seminal vesicles) and selected Organs at Risk (OARs) (rectum/bladder/femoral heads + femurs), evaluating also inter-observer variability (manual vs automatic + editing) and the reduction of contouring time.

Materials and methods

Two expert observers contoured CTVs/OARs of 20 patients in our Treatment Planning System (TPS). Computed Tomography (CT) images were sent to the automatic contouring workstation: automatic contours were generated and sent back to TPS, where observers could edit them if necessary. Inter- and intra-observer consistency was estimated using Dice Similarity Coefficients (DSC). Radiation oncologists were also asked to score the quality of automatic contours, ranging from 1 (complete re-contouring) to 5 (no editing). Contouring times (manual vs automatic + edit) were compared.

Results

DSCs (manual vs automatic only) were consistent with inter-observer variability (between 0.65 for seminal vesicles and 0.94 for bladder); editing further improved performances (range: 0.76-0.94). The median clinical score was 4 (little editing) and it was <4 in 3/2 patients for the two observers respectively. Inter-observer variability of automatic + editing contours improved significantly, being lower than manual contouring (e.g.: seminal vesicles: 0.83vs0.73; prostate: 0.86vs0.83; rectum: 0.96vs0.81). Oncologist contouring time reduced from 17 to 24 min of manual contouring time to 3-7 min of editing time for the two observers (p < 0.01).

Conclusion

Automatic contouring with a commercial AI-based system followed by editing can replace manual contouring, resulting in significantly reduced time for segmentation and better consistency between operators.

CBCT-based synthetic CT generated using CycleGAN with HU correction for adaptive radiotherapy of nasopharyngeal carcinoma

This study aims to utilize a hybrid approach of phantom correction and deep learning for synthesized CT (sCT) images generation based on cone-beam CT (CBCT) images for nasopharyngeal carcinoma (NPC). 52 CBCT/CT paired images of NPC patients were used for model training (41), validation (11). Hounsfield Units (HU) of the CBCT images was calibrated by a commercially available CIRS phantom. Then the original CBCT and the corrected CBCT (CBCT_cor) were trained separately with the same cycle generative adversarial network (CycleGAN) to generate SCT1 and SCT2. The mean error and mean absolute error (MAE) were used to quantify the image quality. For validations, the contours and treatment plans in CT images were transferred to original CBCT, CBCT_cor, SCT1 and SCT2 for dosimetric comparison. Dose distribution, dosimetric parameters and 3D gamma passing rate were analyzed. Compared with rigidly registered CT (RCT), the MAE of CBCT, CBCT_cor, SCT1 and SCT2 were 346.11 ± 13.58 HU, 145.95 ± 17.64 HU, 105.62 ± 16.08 HU and 83.51 ± 7.71 HU, respectively. Moreover, the average dosimetric parameter differences for the CBCT_cor, SCT1 and SCT2 were 2.7% ± 1.4%, 1.2% ± 1.0% and 0.6% ± 0.6%, respectively. Using the dose distribution of RCT images as reference, the 3D gamma passing rate of the hybrid method was significantly better than the other methods. The effectiveness of CBCT-based sCT generated using CycleGAN with HU correction for adaptive radiotherapy of nasopharyngeal carcinoma was confirmed. The image quality and dose accuracy of SCT2 were outperform the simple CycleGAN method. This finding has great significance for the clinical application of adaptive radiotherapy for NPC.

Deep learning-based segmentation of prostatic urethra on computed tomography scans for treatment planning

Background and purpose

The intraprostatic urethra is an organ at risk in prostate cancer radiotherapy, but its segmentation in computed tomography (CT) is challenging. This work sought to: i) propose an automatic pipeline for intraprostatic urethra segmentation in CT, ii) analyze the dose to the urethra, iii) compare the predictions to magnetic resonance (MR) contours.

Materials and methods

First, we trained Deep Learning networks to segment the rectum, bladder, prostate, and seminal vesicles. Then, the proposed Deep Learning Urethra Segmentation model was trained with the bladder and prostate distance transforms and 44 labeled CT with visible catheters. The evaluation was performed on 11 datasets, calculating centerline distance (CLD) and percentage of centerline within 3.5 and 5 mm. We applied this method to a dataset of 32 patients treated with intensity-modulated radiation therapy (IMRT) to quantify the urethral dose. Finally, we compared predicted intraprostatic urethra contours to manual delineations in MR for 15 patients without catheter.

Results

A mean CLD of 1.6 ± 0.8 mm for the whole urethra and 1.7 ± 1.4, 1.5 ± 0.9, and 1.7 ± 0.9 mm for the top, middle, and bottom thirds were obtained in CT. On average, 94% and 97% of the segmented centerlines were within a 3.5 mm and 5 mm radius, respectively. In IMRT, the urethra received a higher dose than the overall prostate. We also found a slight deviation between the predicted and manual MR delineations.

Conclusion

A fully-automatic segmentation pipeline was validated to delineate the intraprostatic urethra in CT images.

FlowRAU-Net: Accelerated 4D Flow MRI of Aortic Valvular Flows With a Deep 2D Residual Attention Network

In this work, we propose a novel deep learning reconstruction framework for rapid and accurate reconstruction of 4D flow MRI data. Reconstruction is performed on a slice-by-slice basis by reducing artifacts in zero-filled reconstructed complex images obtained from undersampled k-space. A deep residual attention network FlowRAU-Net is proposed, trained separately for each encoding direction with 2D complex image slices extracted from complex 4D images at each temporal frame and slice position. The network was trained and tested on 4D flow MRI data of aortic valvular flow in 18 human subjects. Performance of the reconstructions was measured in terms of image quality, 3-D velocity vector accuracy, and accuracy in hemodynamic parameters. Reconstruction performance was measured for three different k-space undersamplings and compared with one state of the art compressed sensing reconstruction method and three deep learning-based reconstruction methods. The proposed method outperforms state of the art methods in all performance measures for all three different k-space undersamplings. Hemodynamic parameters such as blood flow rate and peak velocity from the proposed technique show good agreement with reference flow parameters. Visualization of the reconstructed image and velocity magnitude also shows excellent agreement with the fully sampled reference dataset. Moreover, the proposed method is computationally fast. Total 4D flow data (including all slices in space and time) for a subject can be reconstructed in 69 seconds on a single GPU. Although the proposed method has been applied to 4D flow MRI of aortic valvular flows, given a sufficient number of training samples, it should be applicable to other arterial flows.

A retrospective study of 3D deep learning approach incorporating coordinate information to improve the segmentation of pre- and post-operative abdominal aortic aneurysm

Abdominal aortic aneurysm (AAA) is one of the most common diseases worldwide. 3D segmentation of AAA provides useful information for surgical decisions and follow-up treatment. However, existing segmentation methods are time consuming and not practical in routine use. In this article, the segmentation task will be addressed automatically using a deep learning based approach which has been proved to successfully solve several medical imaging problems with excellent performances. This article therefore proposes a new solution of AAA segmentation using deep learning in a type of 3D convolutional neural network (CNN) architecture that also incorporates coordinate information. The tested CNNs are UNet, AG-DSV-UNet, VNet, ResNetMed and DenseVoxNet. The 3D-CNNs are trained with a dataset of high resolution (256 × 256) non-contrast and post-contrast CT images containing 64 slices from each of 200 patients. The dataset consists of contiguous CT slices without augmentation and no post-processing step. The experiments show that incorporation of coordinate information improves the segmentation results. The best accuracies on non-contrast and contrast-enhanced images have average dice scores of 97.13% and 96.74%, respectively. Transfer learning from a pre-trained network of a pre-operative dataset to post-operative endovascular aneurysm repair (EVAR) was also performed. The segmentation accuracy of post-operative EVAR using transfer learning on non-contrast and contrast-enhanced CT datasets achieved the best dice scores of 94.90% and 95.66%, respectively.

Segmentation of male pelvic organs on computed tomography with a deep neural network fine-tuned by a level-set method

Computed Tomography (CT) imaging is used in Radiation Therapy planning, where the treatment is carefully tailored to each patient in order to maximize radiation dose to the target while decreasing adverse effects to nearby healthy tissues. A crucial step in this process is manual organ contouring, which if performed automatically could considerably decrease the time to starting treatment and improve outcomes. Computerized segmentation of male pelvic organs has been studied for decades and deep learning models have brought considerable advances to the field, but improvements are still demanded. A two-step framework for automatic segmentation of the prostate, bladder and rectum is presented: a convolutional neural network enhanced with attention gates performs an initial segmentation, followed by a region-based active contour model to fine-tune the segmentations to each patient's specific anatomy. The framework was evaluated on a large collection of planning CTs of patients who had Radiation Therapy for prostate cancer. The Surface Dice Coefficient improved from 79.41 to 81.00% on segmentation of the prostate, 94.03-95.36% on the bladder and 82.17-83.68% on the rectum, comparing the proposed framework with the baseline convolutional neural network. This study shows that traditional image segmentation algorithms can help improve the immense gains that deep learning models have brought to the medical imaging segmentation field.

A 3D deep learning approach to epicardial fat segmentation in non-contrast and post-contrast cardiac CT images

Epicardial fat (ECF) is localized fat surrounding the heart muscle or myocardium and enclosed by the thin-layer pericardium membrane. Segmenting the ECF is one of the most difficult medical image segmentation tasks. Since the epicardial fat is infiltrated into the groove between cardiac chambers and is contiguous with cardiac muscle, segmentation requires location and voxel intensity. Recently, deep learning methods have been effectively used to solve medical image segmentation problems in several domains with state-of-the-art performance. This paper presents a novel approach to 3D segmentation of ECF by integrating attention gates and deep supervision into the 3D U-Net deep learning architecture. The proposed method shows significant improvement of the segmentation performance, when compared with standard 3D U-Net. The experiments show excellent performance on non-contrast CT datasets with average Dice scores of 90.06%. Transfer learning from a pre-trained model of a non-contrast CT to contrast-enhanced CT dataset was also performed. The segmentation accuracy on the contrast-enhanced CT dataset achieved a Dice score of 88.16%.

Magnetic Resonance Imaging Image Feature Analysis Algorithm under Convolutional Neural Network in the Diagnosis and Risk Stratification of Prostate Cancer

This work aimed to explore the accuracy of magnetic resonance imaging (MRI) images based on the convolutional neural network (CNN) algorithm in the diagnosis of prostate cancer patients and tumor risk grading. A total of 89 patients with prostate cancer and benign prostatic hyperplasia diagnosed by MRI examination and pathological examination in hospital were selected as the research objects in this study (they passed the exclusion criteria). The MRI images of these patients were collected in two groups and divided into two groups before and after treatment according to whether the CNN algorithm was used to process them. The number of diagnosed diseases and the number of cases of risk level inferred based on the tumor grading were compared to observe which group was closer to the diagnosis of pathological biopsy. Through comparative analysis, compared with the positive rate of pathological diagnosis (44%), the positive rate after the treatment of the CNN algorithm (42%) was more similar to that before the treatment (34%), and the comparison was statistically marked (P < 0.05). In terms of risk stratification, the grading results after treatment (37 cases) were closer to the results of pathological grading (39 cases) than those before treatment (30 cases), and the comparison was statistically obvious (P < 0.05). In addition, it was obvious that the MRT images would be clearer after treatment through the observation of the MRT images before and after treatment. In conclusion, MRI image segmentation algorithm based on CNN was more accurate in the diagnosis and risk stratification of prostate cancer than routine MRI. According to the evaluation of Dice similarity coefficient (DSC) and Hausdorff I distance (HD), the CNN segmentation method used in this study was more perfect than other segmentation methods.

A deep-learning method for generating synthetic kV-CT and improving tumor segmentation for helical tomotherapy of nasopharyngeal carcinoma

Objective:Megavoltage computed tomography (MV-CT) is used for setup verification and adaptive radiotherapy in tomotherapy. However, its low contrast and high noise lead to poor image quality. This study aimed to develop a deep-learning-based method to generate synthetic kilovoltage CT (skV-CT) and then evaluate its ability to improve image quality and tumor segmentation.Approach:The planning kV-CT and MV-CT images of 270 patients with nasopharyngeal carcinoma (NPC) treated on an Accuray TomoHD system were used. An improved cycle-consistent adversarial network which used residual blocks as its generator was adopted to learn the mapping between MV-CT and kV-CT and then generate skV-CT from MV-CT. A Catphan 700 phantom and 30 patients with NPC were used to evaluate image quality. The quantitative indices included contrast-to-noise ratio (CNR), uniformity and signal-to-noise ratio (SNR) for the phantom and the structural similarity index measure (SSIM), mean absolute error (MAE), and peak signal-to-noise ratio (PSNR) for patients. Next, we trained three models for segmentation of the clinical target volume (CTV): MV-CT, skV-CT, and MV-CT combined with skV-CT. The segmentation accuracy was compared with indices of the dice similarity coefficient (DSC) and mean distance agreement (MDA).Mainresults:Compared with MV-CT, skV-CT showed significant improvement in CNR (184.0%), image uniformity (34.7%), and SNR (199.0%) in the phantom study and improved SSIM (1.7%), MAE (24.7%), and PSNR (7.5%) in the patient study. For CTV segmentation with only MV-CT, only skV-CT, and MV-CT combined with skV-CT, the DSCs were 0.75 ± 0.04, 0.78 ± 0.04, and 0.79 ± 0.03, respectively, and the MDAs (in mm) were 3.69 ± 0.81, 3.14 ± 0.80, and 2.90 ± 0.62, respectively.Significance:The proposed method improved the image quality of MV-CT and thus tumor segmentation in helical tomotherapy. The method potentially can benefit adaptive radiotherapy.

Automatic Segmentation of Pelvic Cancers Using Deep Learning: State-of-the-Art Approaches and Challenges

The recent rise of deep learning (DL) and its promising capabilities in capturing non-explicit detail from large datasets have attracted substantial research attention in the field of medical image processing. DL provides grounds for technological development of computer-aided diagnosis and segmentation in radiology and radiation oncology. Amongst the anatomical locations where recent auto-segmentation algorithms have been employed, the pelvis remains one of the most challenging due to large intra- and inter-patient soft-tissue variabilities. This review provides a comprehensive, non-systematic and clinically-oriented overview of 74 DL-based segmentation studies, published between January 2016 and December 2020, for bladder, prostate, cervical and rectal cancers on computed tomography (CT) and magnetic resonance imaging (MRI), highlighting the key findings, challenges and limitations.

DeepWL: Robust EPID based Winston-Lutz analysis using deep learning, synthetic image generation and optical path-tracing

Radiation therapy requires clinical linear accelerators to be mechanically and dosimetrically calibrated to a high standard. One important quality assurance test is the Winston-Lutz test which localises the radiation isocentre of the linac. In the current work we demonstrate a novel method of analysing EPID based Winston-Lutz QA images using a deep learning model trained only on synthetic image data. In addition, we propose a novel method of generating the synthetic WL images and associated 'ground-truth' masks using an optical path-tracing engine to 'fake' mega-voltage EPID images. The model called DeepWL was trained on 1500 synthetic WL images using data augmentation techniques for 180 epochs. The model was built using Keras with a TensorFlow backend on an Intel Core i5-6500T CPU and trained in approximately 15 h. DeepWL was shown to produce ball bearing and multi-leaf collimator field segmentations with a mean dice coefficient of 0.964 and 0.994 respectively on previously unseen synthetic testing data. When DeepWL was applied to WL data measured on an EPID, the predicted mean displacements were shown to be statistically similar to the Canny Edge detection method. However, the DeepWL predictions for the ball bearing locations were shown to correlate better with manual annotations compared with the Canny edge detection algorithm. DeepWL was demonstrated to analyse Winston-Lutz images with an accuracy suitable for routine linac quality assurance with some statistical evidence that it may outperform Canny Edge detection methods in terms of segmentation robustness and the resultant displacement predictions.

Deep learning for segmentation in radiation therapy planning: a review

Segmentation of organs and structures, as either targets or organs-at-risk, has a significant influence on the success of radiation therapy. Manual segmentation is a tedious and time-consuming task for clinicians, and inter-observer variability can affect the outcomes of radiation therapy. The recent hype over deep neural networks has added many powerful auto-segmentation methods as variations of convolutional neural networks (CNN). This paper presents a descriptive review of the literature on deep learning techniques for segmentation in radiation therapy planning. The most common CNN architecture across the four clinical sub sites considered was U-net, with the majority of deep learning segmentation articles focussed on head and neck normal tissue structures. The most common data sets were CT images from an inhouse source, along with some public data sets. N-fold cross-validation was commonly employed; however, not all work separated training, test and validation data sets. This area of research is expanding rapidly. To facilitate comparisons of proposed methods and benchmarking, consistent use of appropriate metrics and independent validation should be carefully considered.

Effects of sample size and data augmentation on U-Net-based automatic segmentation of various organs

Deep learning has demonstrated high efficacy for automatic segmentation in contour delineation, which is crucial in radiation therapy planning. However, the collection, labeling, and management of medical imaging data can be challenging. This study aims to elucidate the effects of sample size and data augmentation on the automatic segmentation of computed tomography images using U-Net, a deep learning method. For the chest and pelvic regions, 232 and 556 cases are evaluated, respectively. We investigate multiple conditions by changing the sum of the training and validation datasets across a broad range of values: 10-200 and 10-500 cases for the chest and pelvic regions, respectively. A U-Net is constructed, and horizontal-flip data augmentation, which produces left and right inverse images resulting in twice the number of images, is compared with no augmentation for each training session. All lung cases and more than 100 prostate, bladder, and rectum cases indicate that adding horizontal-flip data augmentation is almost as effective as doubling the number of cases. The slope of the Dice similarity coefficient (DSC) in all organs decreases rapidly until approximately 100 cases, stabilizes after 200 cases, and shows minimal changes as the number of cases is increased further. The DSCs stabilize at a smaller sample size with the incorporation of data augmentation in all organs except the heart. This finding is applicable to the automation of radiation therapy for rare cancers, where large datasets may be difficult to obtain.

A deep learning-based framework for segmenting invisible clinical target volumes with estimated uncertainties for post-operative prostate cancer radiotherapy

In post-operative radiotherapy for prostate cancer, precisely contouring the clinical target volume (CTV) to be irradiated is challenging, because the cancerous prostate gland has been surgically removed, so the CTV encompasses the microscopic spread of tumor cells, which cannot be visualized in clinical images like computed tomography or magnetic resonance imaging. In current clinical practice, physicians' segment CTVs manually based on their relationship with nearby organs and other clinical information, but this allows large inter-physician variability. Automating post-operative prostate CTV segmentation with traditional image segmentation methods has yielded suboptimal results. We propose using deep learning to accurately segment post-operative prostate CTVs. The model proposed is trained using labels that were clinically approved and used for patient treatment. To segment the CTV, we segment nearby organs first, then use their relationship with the CTV to assist CTV segmentation. To ease the encoding of distance-based features, which are important for learning both the CTV contours' overlap with the surrounding OARs and the distance from their borders, we add distance prediction as an auxiliary task to the CTV network. To make the DL model practical for clinical use, we use Monte Carlo dropout (MCDO) to estimate model uncertainty. Using MCDO, we estimate and visualize the 95% upper and lower confidence bounds for each prediction which informs the physicians of areas that might require correction. The model proposed achieves an average Dice similarity coefficient (DSC) of 0.87 on a holdout test dataset, much better than established methods, such as atlas-based methods (DSC<0.7). The predicted contours agree with physician contours better than medical resident contours do. A reader study showed that the clinical acceptability of the automatically segmented CTV contours is equal to that of approved clinical contours manually drawn by physicians. Our deep learning model can accurately segment CTVs with the help of surrounding organ masks. Because the DL framework can outperform residents, it can be implemented practically in a clinical workflow to generate initial CTV contours or to guide residents in generating these contours for physicians to review and revise. Providing physicians with the 95% confidence bounds could streamline the review process for an efficient clinical workflow as this would enable physicians to concentrate their inspecting and editing efforts on the large uncertain areas.

Improved contrast and noise of megavoltage computed tomography (MVCT) through cycle-consistent generative machine learning

PURPOSE

Megavoltage computed tomography (MVCT) has been implemented on many radiation therapy treatment machines as a tomographic imaging modality that allows for three-dimensional visualization and localization of patient anatomy. Yet MVCT images exhibit lower contrast and greater noise than its kilovoltage CT (kVCT) counterpart. In this work, we sought to improve these disadvantages of MVCT images through an image-to-image-based machine learning transformation of MVCT and kVCT images. We demonstrated that by learning the style of kVCT images, MVCT images can be converted into high-quality synthetic kVCT (skVCT) images with higher contrast and lower noise, when compared to the original MVCT.

METHODS

Kilovoltage CT and MVCT images of 120 head and neck (H&N) cancer patients treated on an Accuray TomoHD system were retrospectively analyzed in this study. A cycle-consistent generative adversarial network (CycleGAN) machine learning, a variant of the generative adversarial network (GAN), was used to learn Hounsfield Unit (HU) transformations from MVCT to kVCT images, creating skVCT images. A formal mathematical proof is given describing the interplay between function sensitivity and input noise and how it applies to the error variance of a high-capacity function trained with noisy input data. Finally, we show how skVCT shares distributional similarity to kVCT for various macro-structures found in the body.

RESULTS

Signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) were improved in skVCT images relative to the original MVCT images and were consistent with kVCT images. Specifically, skVCT CNR for muscle-fat, bone-fat, and bone-muscle improved to 14.8 ± 0.4, 122.7 ± 22.6, and 107.9 ± 22.4 compared with 1.6 ± 0.3, 7.6 ± 1.9, and 6.0 ± 1.7, respectively, in the original MVCT images and was more consistent with kVCT CNR values of 15.2 ± 0.8, 124.9 ± 27.0, and 109.7 ± 26.5, respectively. Noise was significantly reduced in skVCT images with SNR values improving by roughly an order of magnitude and consistent with kVCT SNR values. Axial slice mean (S-ME) and mean absolute error (S-MAE) agreement between kVCT and MVCT/skVCT improved, on average, from -16.0 and 109.1 HU to 8.4 and 76.9 HU, respectively.

CONCLUSIONS

A kVCT-like qualitative aid was generated from input MVCT data through a CycleGAN instance. This qualitative aid, skVCT, was robust toward embedded metallic material, dramatically improves HU alignment from MVCT, and appears perceptually similar to kVCT with SNR and CNR values equivalent to that of kVCT images.

Automatic segmentation of three clinical target volumes in radiotherapy using lifelong learning

BACKGROUND AND PURPOSE

Convolutional neural networks (CNNs) have comparable human level performance in automatic segmentation. An important challenge that CNNs face in segmentation is catastrophic forgetting. They lose performance on tasks that were previously learned when trained on task. In this study, we propose a lifelong learning method to learn multiple segmentation tasks continuously without forgetting previous tasks.

MATERIALS AND METHODS

The cohort included three tumors, 800 patients of which had nasopharyngeal cancer (NPC), 800 patients had breast cancer, and 800 patients had rectal cancer. The tasks included segmentation of the clinical target volume (CTV) of these three cancers. The proposed lifelong learning network adopted dilation adapter to learn three segmentation tasks one by one. Only the newly added dilation adapter (seven layers) was fine tuning for incoming new task, whereas all the other learned layers were frozen.

RESULTS

Compared with single-task, multi-task or transfer learning, the proposed lifelong learning can achieve better or comparable segmentation accuracy with a DSC of 0.86 for NPC, 0.89 for breast cancer, and 0.87 for rectal cancer. Lifelong learning can avoid forgetting in sequential learning and yield good performance with less training data. Furthermore, it is more efficient than single-task or transfer learning, which reduced the number of parameters, size of model, and training time by ~58.8%, ~55.6%, and ~25.0%, respectively.

CONCLUSION

The proposed method preserved the knowledge of previous tasks while learning a new one using a dilation adapter. It could yield comparable performance with much less training data, model parameters, and training time.

A Deep Learning-Based Automated CT Segmentation of Prostate Cancer Anatomy for Radiation Therapy Planning-A Retrospective Multicenter Study

A commercial deep learning (DL)-based automated segmentation tool (AST) for computed tomography (CT) is evaluated for accuracy and efficiency gain within prostate cancer patients. Thirty patients from six clinics were reviewed with manual- (MC), automated- (AC) and automated and edited (AEC) contouring methods. In the AEC group, created contours (prostate, seminal vesicles, bladder, rectum, femoral heads and penile bulb) were edited, whereas the MC group included empty datasets for MC. In one clinic, lymph node CTV delineations were evaluated for interobserver variability. Compared to MC, the mean time saved using the AST was 12 min for the whole data set (46%) and 12 min for the lymph node CTV (60%), respectively. The delineation consistency between MC and AEC groups according to the Dice similarity coefficient (DSC) improved from 0.78 to 0.94 for the whole data set and from 0.76 to 0.91 for the lymph nodes. The mean DSCs between MC and AC for all six clinics were 0.82 for prostate, 0.72 for seminal vesicles, 0.93 for bladder, 0.84 for rectum, 0.69 for femoral heads and 0.51 for penile bulb. This study proves that using a general DL-based AST for CT images saves time and improves consistency.

Automatic intraprostatic lesion segmentation in multiparametric magnetic resonance images with proposed multiple branch UNet

PURPOSE

Contouring intraprostatic lesions is a prerequisite for dose-escalating these lesions in radiotherapy to improve the local cancer control. In this study, a deep learning-based approach was developed for automatic intraprostatic lesion segmentation in multiparametric magnetic resonance imaging (mpMRI) images contributing to clinical practice.

METHODS

Multiparametric magnetic resonance imaging images from 136 patient cases were collected from our institution, and all these cases contained suspicious lesions with Prostate Imaging Reporting and Data System (PI-RADS) score ≥ 4. The contours of the lesion and prostate were manually created on axial T2-weighted (T2W), apparent diffusion coefficient (ADC) and high b-value diffusion-weighted imaging (DWI) images to provide the ground truth data. Then a multiple branch UNet (MB-UNet) was proposed for the segmentation of an indistinct target in multi-modality MRI images. An encoder module was designed with three branches for the three MRI modalities separately, to fully extract the high-level features provided by different MRI modalities; an input module was added by using three sub-branches for three consecutive image slices, to consider the contour consistency among different image slices; deep supervision strategy was also integrated into the network to speed up the convergency of the network and improve the performance. The probability maps of the background, normal prostate and lesion were output by the network to generate the segmentation of the lesion, and the performance was evaluated using the dice similarity coefficient (DSC) as the main metric.

RESULTS

A total of 162 lesions were contoured on 652 image slices, with 119 lesions in the peripheral zone, 38 in the transition zone, four in the central zone and one in the anterior fibromuscular stroma. All prostates were also contoured on 1,264 image slices. As for the segmentation of lesions in the testing set, MB-UNet achieved a per case DSC of 0.6333, specificity of 0.9993, sensitivity of 0.7056; and global DSC of 0.7205, specificity of 0.9993, sensitivity of 0.7409. All the three deep learning strategies adopted in this study contributed to the performance promotion of the MB-UNet. Missing the DWI modality would degrade the segmentation performance more markedly compared with the other two modalities.

CONCLUSIONS

A deep learning-based approach with proposed MB-UNet was developed to automatically segment suspicious lesions in mpMRI images. This study makes it feasible to adopt boosting intraprostatic lesions in clinical practice to achieve better outcomes.

Improving CT Image Tumor Segmentation Through Deep Supervision and Attentional Gates

Computer Tomography (CT) is an imaging procedure that combines many X-ray measurements taken from different angles. The segmentation of areas in the CT images provides a valuable aid to physicians and radiologists in order to better provide a patient diagnose. The CT scans of a body torso usually include different neighboring internal body organs. Deep learning has become the state-of-the-art in medical image segmentation. For such techniques, in order to perform a successful segmentation, it is of great importance that the network learns to focus on the organ of interest and surrounding structures and also that the network can detect target regions of different sizes. In this paper, we propose the extension of a popular deep learning methodology, Convolutional Neural Networks (CNN), by including deep supervision and attention gates. Our experimental evaluation shows that the inclusion of attention and deep supervision results in consistent improvement of the tumor prediction accuracy across the different datasets and training sizes while adding minimal computational overhead.

DoseGAN: a generative adversarial network for synthetic dose prediction using attention-gated discrimination and generation

Deep learning algorithms have recently been developed that utilize patient anatomy and raw imaging information to predict radiation dose, as a means to increase treatment planning efficiency and improve radiotherapy plan quality. Current state-of-the-art techniques rely on convolutional neural networks (CNNs) that use pixel-to-pixel loss to update network parameters. However, stereotactic body radiotherapy (SBRT) dose is often heterogeneous, making it difficult to model using pixel-level loss. Generative adversarial networks (GANs) utilize adversarial learning that incorporates image-level loss and is better suited to learn from heterogeneous labels. However, GANs are difficult to train and rely on compromised architectures to facilitate convergence. This study suggests an attention-gated generative adversarial network (DoseGAN) to improve learning, increase model complexity, and reduce network redundancy by focusing on relevant anatomy. DoseGAN was compared to alternative state-of-the-art dose prediction algorithms using heterogeneity index, conformity index, and various dosimetric parameters. All algorithms were trained, validated, and tested using 141 prostate SBRT patients. DoseGAN was able to predict more realistic volumetric dosimetry compared to all other algorithms and achieved statistically significant improvement compared to all alternative algorithms for the V100 and V120 of the PTV, V60 of the rectum, and heterogeneity index.

Sequential vessel segmentation via deep channel attention network

Accurately segmenting contrast-filled vessels from X-ray coronary angiography (XCA) image sequence is an essential step for the diagnosis and therapy of coronary artery disease. However, developing automatic vessel segmentation is particularly challenging due to the overlapping structures, low contrast and the presence of complex and dynamic background artifacts in XCA images. This paper develops a novel encoder-decoder deep network architecture which exploits the several contextual frames of 2D+t sequential images in a sliding window centered at current frame to segment 2D vessel masks from the current frame. The architecture is equipped with temporal-spatial feature extraction in encoder stage, feature fusion in skip connection layers and channel attention mechanism in decoder stage. In the encoder stage, a series of 3D convolutional layers are employed to hierarchically extract temporal-spatial features. Skip connection layers subsequently fuse the temporal-spatial feature maps and deliver them to the corresponding decoder stages. To efficiently discriminate vessel features from the complex and noisy backgrounds in the XCA images, the decoder stage effectively utilizes channel attention blocks to refine the intermediate feature maps from skip connection layers for subsequently decoding the refined features in 2D ways to produce the segmented vessel masks. Furthermore, Dice loss function is implemented to train the proposed deep network in order to tackle the class imbalance problem in the XCA data due to the wide distribution of complex background artifacts. Extensive experiments by comparing our method with other state-of-the-art algorithms demonstrate the proposed method's superior performance over other methods in terms of the quantitative metrics and visual validation. To facilitate the reproductive research in XCA community, we publicly release our dataset and source codes at https://github.com/Binjie-Qin/SVS-net.

Attention-Aware Discrimination for MR-to-CT Image Translation Using Cycle-Consistent Generative Adversarial Networks

Purpose

To suggest an attention-aware, cycle-consistent generative adversarial network (A-CycleGAN) enhanced with variational autoencoding (VAE) as a superior alternative to current state-of-the-art MR-to-CT image translation methods.

Materials and Methods

An attention-gating mechanism is incorporated into a discriminator network to encourage a more parsimonious use of network parameters, whereas VAE enhancement enables deeper discrimination architectures without inhibiting model convergence. Findings from 60 patients with head, neck, and brain cancer were used to train and validate A-CycleGAN, and findings from 30 patients were used for the holdout test set and were used to report final evaluation metric results using mean absolute error (MAE) and peak signal-to-noise ratio (PSNR).

Results

A-CycleGAN achieved superior results compared with U-Net, a generative adversarial network (GAN), and a cycle-consistent GAN. The A-CycleGAN averages, 95% confidence intervals (CIs), and Wilcoxon signed-rank two-sided test statistics are shown for MAE (19.61 [95% CI: 18.83, 20.39], P = .0104), structure similarity index metric (0.778 [95% CI: 0.758, 0.798], P = .0495), and PSNR (62.35 [95% CI: 61.80, 62.90], P = .0571).

Conclusion

A-CycleGANs were a superior alternative to state-of-the-art MR-to-CT image translation methods.© RSNA, 2020.

Synthetic CT generation from CBCT images via deep learning

PURPOSE

Cone-beam computed tomography (CBCT) scanning is used daily or weekly (i.e., on-treatment CBCT) for accurate patient setup in image-guided radiotherapy. However, inaccuracy of CT numbers prevents CBCT from performing advanced tasks such as dose calculation and treatment planning. Motivated by the promising performance of deep learning in medical imaging, we propose a deep U-net-based approach that synthesizes CT-like images with accurate numbers from planning CT, while keeping the same anatomical structure as on-treatment CBCT.

METHODS

We formulated the CT synthesis problem under a deep learning framework, where a deep U-net architecture was used to take advantage of the anatomical structure of on-treatment CBCT and image intensity information of planning CT. U-net was chosen because it exploits both global and local features in the image spatial domain, matching our task to suppress global scattering artifacts and local artifacts such as noise in CBCT. To train the synthetic CT generation U-net (sCTU-net), we include on-treatment CBCT and initial planning CT of 37 patients (30 for training, seven for validation) as the input. Additional replanning CT images acquired on the same day as CBCT after deformable registration are utilized as the corresponding reference. To demonstrate the effectiveness of the proposed sCTU-net, we use another seven independent patient cases (560 slices) for testing.

RESULTS

We quantitatively compared the resulting synthetic CT (sCT) with the original CBCT image using deformed same-day pCT images as reference. The averaged accuracy measured by mean absolute error (MAE) between sCT and reference CT (rCT) on testing data is 18.98 HU, while MAE between CBCT and rCT is 44.38 HU.

CONCLUSIONS

The proposed sCTU-net can synthesize CT-quality images with accurate CT numbers from on-treatment CBCT and planning CT. This potentially enables advanced CBCT applications for adaptive treatment planning.

Artificial Intelligence: reshaping the practice of radiological sciences in the 21st century

Advances in computing hardware and software platforms have led to the recent resurgence in artificial intelligence (AI) touching almost every aspect of our daily lives by its capability for automating complex tasks or providing superior predictive analytics. AI applications are currently spanning many diverse fields from economics to entertainment, to manufacturing, as well as medicine. Since modern AI's inception decades ago, practitioners in radiological sciences have been pioneering its development and implementation in medicine, particularly in areas related to diagnostic imaging and therapy. In this anniversary article, we embark on a journey to reflect on the learned lessons from past AI's chequered history. We further summarize the current status of AI in radiological sciences, highlighting, with examples, its impressive achievements and effect on re-shaping the practice of medical imaging and radiotherapy in the areas of computer-aided detection, diagnosis, prognosis, and decision support. Moving beyond the commercial hype of AI into reality, we discuss the current challenges to overcome, for AI to achieve its promised hope of providing better precision healthcare for each patient while reducing cost burden on their families and the society at large.

Prediction of major torso organs in low-contrast micro-CT images of mice using a two-stage deeply supervised fully convolutional network

Delineation of major torso organs is a key step of mouse micro-CT image analysis. This task is challenging due to low soft tissue contrast and high image noise, therefore anatomical prior knowledge is needed for accurate prediction of organ regions. In this work, we develop a deeply supervised fully convolutional network which uses the organ anatomy prior learned from independently acquired contrast-enhanced micro-CT images to assist the segmentation of non-enhanced images. The network is designed with a two-stage workflow which firstly predicts the rough regions of multiple organs and then refines the accuracy of each organ in local regions. The network is trained and evaluated with 40 mouse micro-CT images. The volumetric prediction accuracy (Dice score) varies from 0.57 for the spleen to 0.95 for the heart. Compared to a conventional atlas registration method, our method dramatically improves the Dice of the abdominal organs by 18%-26%. Moreover, the incorporation of anatomical prior leads to more accurate results for small-sized low-contrast organs (e.g. the spleen and kidneys). We also find that the localized stage of the network has better accuracy than the global stage, indicating that localized single organ prediction is more accurate than global multiple organ prediction. With this work, the accuracy and efficiency of mouse micro-CT image analysis are greatly improved and the need for using contrast agent and high x-ray dose is potentially reduced.