Cascaded MultiTask 3-D Fully Convolutional Networks for Pancreas Segmentation

Beyond CNNs: Exploiting Further Inherent Symmetries in Medical Image Segmentation

Automatic tumor or lesion segmentation is a crucial step in medical image analysis for computer-aided diagnosis. Although the existing methods based on convolutional neural networks (CNNs) have achieved the state-of-the-art performance, many challenges still remain in medical tumor segmentation. This is because, although the human visual system can detect symmetries in 2-D images effectively, regular CNNs can only exploit translation invariance, overlooking further inherent symmetries existing in medical images, such as rotations and reflections. To solve this problem, we propose a novel group equivariant segmentation framework by encoding those inherent symmetries for learning more precise representations. First, kernel-based equivariant operations are devised on each orientation, which allows it to effectively address the gaps of learning symmetries in existing approaches. Then, to keep segmentation networks globally equivariant, we design distinctive group layers with layer-wise symmetry constraints. Finally, based on our novel framework, extensive experiments conducted on real-world clinical data demonstrate that a group equivariant Res-UNet (called GER-UNet) outperforms its regular CNN-based counterpart and the state-of-the-art segmentation methods in the tasks of hepatic tumor segmentation, COVID-19 lung infection segmentation, and retinal vessel detection. More importantly, the newly built GER-UNet also shows potential in reducing the sample complexity and the redundancy of filters, upgrading current segmentation CNNs, and delineating organs on other medical imaging modalities.

Clinical capability of modern brain tumor segmentation models

PURPOSE

State-of-the-art automated segmentation methods achieve exceptionally high performance on the Brain Tumor Segmentation (BraTS) challenge, a dataset of uniformly processed and standardized magnetic resonance generated images (MRIs) of gliomas. However, a reasonable concern is that these models may not fare well on clinical MRIs that do not belong to the specially curated BraTS dataset. Research using the previous generation of deep learning models indicates significant performance loss on cross-institutional predictions. Here, we evaluate the cross-institutional applicability and generalzsability of state-of-the-art deep learning models on new clinical data.

METHODS

We train a state-of-the-art 3D U-Net model on the conventional BraTS dataset comprising low- and high-grade gliomas. We then evaluate the performance of this model for automatic tumor segmentation of brain tumors on in-house clinical data. This dataset contains MRIs of different tumor types, resolutions, and standardization than those found in the BraTS dataset. Ground truth segmentations to validate the automated segmentation for in-house clinical data were obtained from expert radiation oncologists.

RESULTS

We report average Dice scores of 0.764, 0.648, and 0.61 for the whole tumor, tumor core, and enhancing tumor, respectively, in the clinical MRIs. These means are higher than numbers reported previously on same institution and cross-institution datasets of different origin using different methods. There is no statistically significant difference when comparing the dice scores to the inter-annotation variability between two expert clinical radiation oncologists. Although performance on the clinical data is lower than on the BraTS data, these numbers indicate that models trained on the BraTS dataset have impressive segmentation performance on previously unseen images obtained at a separate clinical institution. These images differ in the imaging resolutions, standardization pipelines, and tumor types from the BraTS data.

CONCLUSIONS

State-of-the-art deep learning models demonstrate promising performance on cross-institutional predictions. They considerably improve on previous models and can transfer knowledge to new types of brain tumors without additional modeling.

PolypSeg+: A Lightweight Context-Aware Network for Real-Time Polyp Segmentation

Automatic polyp segmentation from colonoscopy videos is a prerequisite for the development of a computer-assisted colon cancer examination and diagnosis system. However, it remains a very challenging task owing to the large variation of polyps, the low contrast between polyps and background, and the blurring boundaries of polyps. More importantly, real-time performance is a necessity of this task, as it is anticipated that the segmented results can be immediately presented to the doctor during the colonoscopy intervention for his/her prompt decision and action. It is difficult to develop a model with powerful representation capability, yielding satisfactory segmentation results and, simultaneously, maintaining real-time performance. In this article, we present a novel lightweight context-aware network, namely, PolypSeg+, attempting to capture distinguishable features of polyps without increasing network complexity and sacrificing time performance. To achieve this, a set of novel lightweight techniques is developed and integrated into the proposed PolypSeg+, including an adaptive scale context (ASC) module equipped with a lightweight attention mechanism to tackle the large-scale variation of polyps, an efficient global context (EGC) module to promote the fusion of low-level and high-level features by excluding background noise and preserving boundary details, and a lightweight feature pyramid fusion (FPF) module to further refine the features extracted from the ASC and EGC. We extensively evaluate the proposed PolypSeg+ on two famous public available datasets for the polyp segmentation task: 1) Kvasir-SEG and 2) CVC-Endoscenestill. The experimental results demonstrate that our PolypSeg+ consistently outperforms other state-of-the-art networks by achieving better segmentation accuracy in much less running time. The code is available at https://github.com/szu-zzb/polypsegplus.

Multi-task deep learning for medical image computing and analysis: A review

The renaissance of deep learning has provided promising solutions to various tasks. While conventional deep learning models are constructed for a single specific task, multi-task deep learning (MTDL) that is capable to simultaneously accomplish at least two tasks has attracted research attention. MTDL is a joint learning paradigm that harnesses the inherent correlation of multiple related tasks to achieve reciprocal benefits in improving performance, enhancing generalizability, and reducing the overall computational cost. This review focuses on the advanced applications of MTDL for medical image computing and analysis. We first summarize four popular MTDL network architectures (i.e., cascaded, parallel, interacted, and hybrid). Then, we review the representative MTDL-based networks for eight application areas, including the brain, eye, chest, cardiac, abdomen, musculoskeletal, pathology, and other human body regions. While MTDL-based medical image processing has been flourishing and demonstrating outstanding performance in many tasks, in the meanwhile, there are performance gaps in some tasks, and accordingly we perceive the open challenges and the perspective trends. For instance, in the 2018 Ischemic Stroke Lesion Segmentation challenge, the reported top dice score of 0.51 and top recall of 0.55 achieved by the cascaded MTDL model indicate further research efforts in high demand to escalate the performance of current models.

Boundary-Sensitive Loss Function With Location Constraint for Hard Region Segmentation

In computer-aided diagnosis and treatment planning, accurate segmentation of medical images plays an essential role, especially for some hard regions including boundaries, small objects and background interference. However, existing segmentation loss functions including distribution-, region- and boundary-based losses cannot achieve satisfactory performances on these hard regions. In this paper, a boundary-sensitive loss function with location constraint is proposed for hard region segmentation in medical images, which provides three advantages: i) our Boundary-Sensitive loss (BS-loss) can automatically pay more attention to the hard-to-segment boundaries (e.g., thin structures and blurred boundaries), thus obtaining finer object boundaries; ii) BS-loss also can adjust its attention to small objects during training to segment them more accurately; and iii) our location constraint can alleviate the negative impact of the background interference, through the distribution matching of pixels between prediction and Ground Truth (GT) along each axis. By resorting to the proposed BS-loss and location constraint, the hard regions in both foreground and background are considered. Experimental results on three public datasets demonstrate the superiority of our method. Specifically, compared to the second-best method tested in this study, our method improves performance on hard regions in terms of Dice similarity coefficient (DSC) and 95% Hausdorff distance (95%HD) of up to 4.17% and 73% respectively. In addition, it also achieves the best overall segmentation performance. Hence, we can conclude that our method can accurately segment these hard regions and improve the overall segmentation performance in medical images.

Extension-contraction transformation network for pancreas segmentation in abdominal CT scans

Accurate and automatic pancreas segmentation from abdominal computed tomography (CT) scans is crucial for the diagnosis and prognosis of pancreatic diseases. However, the pancreas accounts for a relatively small portion of the scan and presents high anatomical variability and low contrast, making traditional automated segmentation methods fail to generate satisfactory results. In this paper, we propose an extension-contraction transformation network (ECTN) and deploy it into a cascaded two-stage segmentation framework for accurate pancreas segmenting. This model can enhance the perception of 3D context by distinguishing and exploiting the extension and contraction transformation of the pancreas between slices. It consists of an encoder, a segmentation decoder, and an extension-contraction (EC) decoder. The EC decoder is responsible for predicting the inter-slice extension and contraction transformation of the pancreas by feeding the extension and contraction information generated by the segmentation decoder; meanwhile, its output is combined with the output of the segmentation decoder to reconstruct and refine the segmentation results. Quantitative evaluation is performed on NIH Pancreas Segmentation (Pancreas-CT) dataset using 4-fold cross-validation. We obtained average Precision of 86.59±6.14% , Recall of 85.11±5.96%, Dice similarity coefficient (DSC) of 85.58±3.98%. and Jaccard Index (JI) of 74.99±5.86%. The performance of our method outperforms several baseline and state-of-the-art methods.

Segmentation and differentiation of periventricular and deep white matter hyperintensities in 2D T2-FLAIR MRI based on a cascade U-net

Background

White matter hyperintensities (WMHs) are a subtype of cerebral small vessel disease and can be divided into periventricular WMHs (pvWMHs) and deep WMHs (dWMHs). pvWMHs and dWMHs were proved to be determined by different etiologies. This study aimed to develop a 2D Cascade U-net (Cascade U) for the segmentation and differentiation of pvWMHs and dWMHs on 2D T2-FLAIR images.

Methods

A total of 253 subjects were recruited in the present study. All subjects underwent 2D T2-FLAIR scan on a 3.0 Tesla MR scanner. Both contours of pvWMHs and dWMHs were manually delineated by the observers and considered as the gold standard. Fazekas scale was used to evaluate the burdens of pvWMHs and dWMHs, respectively. Cascade U consisted of a segmentation U-net and a differentiation U-net and was trained with a combined loss function. The performance of Cascade U was compared with two other U-net models (Pipeline U and Separate U). Dice similarity coefficient (DSC), Matthews correlation coefficient (MCC), precision, and recall were used to evaluate the performances of all models. The linear correlations between WMHs volume (WMHV) measured by all models and the gold standard were also conducted.

Results

Compared with other models, Cascade U exhibited a better performance on WMHs segmentation and pvWMHs identification. Cascade U achieved DSC values of 0.605 ± 0.135, 0.517 ± 0.263, and 0.510 ± 0.241 and MCC values of 0.617 ± 0.122, 0.526 ± 0.263, and 0.522 ± 0.243 on the segmentation of total WMHs, pvWMHs, and dWMHs, respectively. Cascade U exhibited strong correlations with the gold standard on measuring WMHV (R2 = 0.954, p < 0.001), pvWMHV (R2 = 0.933, p < 0.001), and dWMHV (R2 = 0.918, p < 0.001). A significant correlation was found on lesion volume between Cascade U and gold standard (r > 0.510, p < 0.001).

Conclusion

Cascade U showed competitive results in segmentation and differentiation of pvWMHs and dWMHs on 2D T2-FLAIR images, indicating potential feasibility in precisely evaluating the burdens of WMHs.

AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem?

With the unprecedented developments in deep learning, automatic segmentation of main abdominal organs seems to be a solved problem as state-of-the-art (SOTA) methods have achieved comparable results with inter-rater variability on many benchmark datasets. However, most of the existing abdominal datasets only contain single-center, single-phase, single-vendor, or single-disease cases, and it is unclear whether the excellent performance can generalize on diverse datasets. This paper presents a large and diverse abdominal CT organ segmentation dataset, termed AbdomenCT-1K, with more than 1000 (1K) CT scans from 12 medical centers, including multi-phase, multi-vendor, and multi-disease cases. Furthermore, we conduct a large-scale study for liver, kidney, spleen, and pancreas segmentation and reveal the unsolved segmentation problems of the SOTA methods, such as the limited generalization ability on distinct medical centers, phases, and unseen diseases. To advance the unsolved problems, we further build four organ segmentation benchmarks for fully supervised, semi-supervised, weakly supervised, and continual learning, which are currently challenging and active research topics. Accordingly, we develop a simple and effective method for each benchmark, which can be used as out-of-the-box methods and strong baselines. We believe the AbdomenCT-1K dataset will promote future in-depth research towards clinical applicable abdominal organ segmentation methods.

Artificial Intelligence Applied to Pancreatic Imaging: A Narrative Review

The diagnosis, evaluation, and treatment planning of pancreatic pathologies usually require the combined use of different imaging modalities, mainly, computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). Artificial intelligence (AI) has the potential to transform the clinical practice of medical imaging and has been applied to various radiological techniques for different purposes, such as segmentation, lesion detection, characterization, risk stratification, or prediction of response to treatments. The aim of the present narrative review is to assess the available literature on the role of AI applied to pancreatic imaging. Up to now, the use of computer-aided diagnosis (CAD) and radiomics in pancreatic imaging has proven to be useful for both non-oncological and oncological purposes and represents a promising tool for personalized approaches to patients. Although great developments have occurred in recent years, it is important to address the obstacles that still need to be overcome before these technologies can be implemented into our clinical routine, mainly considering the heterogeneity among studies.

AX-Unet: A Deep Learning Framework for Image Segmentation to Assist Pancreatic Tumor Diagnosis

Image segmentation plays an essential role in medical imaging analysis such as tumor boundary extraction. Recently, deep learning techniques have dramatically improved performance for image segmentation. However, an important factor preventing deep neural networks from going further is the information loss during the information propagation process. In this article, we present AX-Unet, a deep learning framework incorporating a modified atrous spatial pyramid pooling module to learn the location information and to extract multi-level contextual information to reduce information loss during downsampling. We also introduce a special group convolution operation on the feature map at each level to achieve information decoupling between channels. In addition, we propose an explicit boundary-aware loss function to tackle the blurry boundary problem. We evaluate our model on two public Pancreas-CT datasets, NIH Pancreas-CT dataset, and the pancreas part in medical segmentation decathlon (MSD) medical dataset. The experimental results validate that our model can outperform the state-of-the-art methods in pancreas CT image segmentation. By comparing the extracted feature output of our model, we find that the pancreatic region of normal people and patients with pancreatic tumors shows significant differences. This could provide a promising and reliable way to assist physicians for the screening of pancreatic tumors.

Subset selection strategy-based pancreas segmentation in CT

BACKGROUND

Although convolutional neural network (CNN)-based methods have been widely used in medical image analysis and have achieved great success in many medical segmentation tasks, these methods suffer from various imbalance problems, which reduce the accuracy and validity of segmentation results.

METHODS

We proposed two simple but effective sample balancing methods, positive-negative subset selection (PNSS) and hard-easy subset selection (HESS) for foreground-to-background imbalance and hard-to-easy imbalance problems in medical segmentation tasks. The PNSS method gradually reduces negative-easy slices to enhance the contribution of positive pixels, and the HESS method enhances the iteration of hard slices to assist the model in paying greater attention to the feature extraction of hard samples.

RESULTS

The proposed methods greatly improved the segmentation accuracy of the worst case (samples with the worst segmentation results) on the public National Institutes of Health (NIH) clinical center pancreatic segmentation dataset, and the minimum dice similarity coefficient (DSC) was improved by nearly 5%. Furthermore, performance gains were also observed with the proposed methods in liver segmentation (the minimum DSC increased from 75.03% to 84.29%), liver tumor segmentation (the minimum DSC increased from 20.92% to 35.73%), and brain tumor segmentation (the minimum DSC increased from 21.97% to 30.38%) on different neural networks. These results indicate that the proposed methods are effective and robust.

CONCLUSIONS

Our proposed method can effectively alleviate foreground-to-background imbalance and hard-to-easy imbalance problems, and can improve segmentation accuracy, especially for the worst case, which guarantees the reliability of the proposed methods in clinical applications.

Cascade Path Augmentation Unet for Bladder Cancer Segmentation in MRI

BACKGROUND

Treatment choices for patients with bladder cancer are determined by the presence of muscular invasion. The precise segmentation of the inner and outer walls (IW and OW), as well as the bladder tumor (BT), is crucial for improving computer-aided diagnosis of muscle-invasive bladder cancer.

PURPOSE

To propose a novel deep learning-based model to improve the segmentation accuracy of the IW, OW, and BT, which can be useful in clinical practice.

METHODS

We proposed a Cascade Path Augmentation Unet (CPA-Unet) network to conduct multi-regional segmentation of the bladder using 1545 T2 weighted MRI scans. The model employs a cascade strategy to eliminate the redundant information in the background. Unet is used to segment the bladder from the background in the rough segmentation. The path augmentation structure is used in the fine segmentation to mine multi-scale features. Additionally, the partial dense connection is adopted as the skip connection module to concatenate the low and high-level sematic features.

RESULTS

The CPA-Unet is trained using 1391 T2WI slices and tested using 154 T2WI slices. In comparison to previous deep learning-based methods, the CPA-Unet achieves superior segmentation results in terms of dice similarity coefficient (DSC) and Hausdorff distance (HD). (IW: DSC = 98.19%, HD = 2.07mm; OW: DSC = 82.24%, HD = 2.62mm; BT: DSC = 87.40%, HD = 0.76mm).

CONCLUSIONS

Our proposed CPA-Unet network is capable of segmenting the bladder into its IW and OW, as well as tumors. The segmentation results provide a reliable and effective foundation for computer-assisted clinical diagnosis of muscle-invasive bladder cancer. This article is protected by copyright. All rights reserved.

Automatic Pancreas Segmentation in CT Images With Distance-Based Saliency-Aware DenseASPP Network

Pancreas identification and segmentation is an essential task in the diagnosis and prognosis of pancreas disease. Although deep neural networks have been widely applied in abdominal organ segmentation, it is still challenging for small organs (e.g. pancreas) that present low contrast, highly flexible anatomical structure and relatively small region. In recent years, coarse-to-fine methods have improved pancreas segmentation accuracy by using coarse predictions in the fine stage, but only object location is utilized and rich image context is neglected. In this paper, we propose a novel distance-based saliency-aware model, namely DSD-ASPP-Net, to fully use coarse segmentation to highlight the pancreas feature and boost accuracy in the fine segmentation stage. Specifically, a DenseASPP (Dense Atrous Spatial Pyramid Pooling) model is trained to learn the pancreas location and probability map, which is then transformed into saliency map through geodesic distance-based saliency transformation. In the fine stage, saliency-aware modules that combine saliency map and image context are introduced into DenseASPP to develop the DSD-ASPP-Net. The architecture of DenseASPP brings multi-scale feature representation and achieves larger receptive field in a denser way, which overcome the difficulties brought by variable object sizes and locations. Our method was evaluated on both public NIH pancreas dataset and local hospital dataset, and achieved an average Dice-Sørensen Coefficient (DSC) value of 85.49±4.77% on the NIH dataset, outperforming former coarse-to-fine methods.

Application of deep learning in image recognition and diagnosis of gastric cancer

In recent years, artificial intelligence has been extensively applied in the diagnosis of gastric cancer based on medical imaging. In particular, using deep learning as one of the mainstream approaches in image processing has made remarkable progress. In this paper, we also provide a comprehensive literature survey using four electronic databases, PubMed, EMBASE, Web of Science, and Cochrane. The literature search is performed until November 2020. This article provides a summary of the existing algorithm of image recognition, reviews the available datasets used in gastric cancer diagnosis and the current trends in applications of deep learning theory in image recognition of gastric cancer. covers the theory of deep learning on endoscopic image recognition. We further evaluate the advantages and disadvantages of the current algorithms and summarize the characteristics of the existing image datasets, then combined with the latest progress in deep learning theory, and propose suggestions on the applications of optimization algorithms. Based on the existing research and application, the label, quantity, size, resolutions, and other aspects of the image dataset are also discussed. The future developments of this field are analyzed from two perspectives including algorithm optimization and data support, aiming to improve the diagnosis accuracy and reduce the risk of misdiagnosis.

Deep learning-based detection and segmentation-assisted management of brain metastases

BACKGROUND

Three-dimensional T1 magnetization prepared rapid acquisition gradient echo (3D-T1-MPRAGE) is preferred in detecting brain metastases (BM) among MRI. We developed an automatic deep learning-based detection and segmentation method for BM (named BMDS net) on 3D-T1-MPRAGE images and evaluated its performance.

METHODS

The BMDS net is a cascaded 3D fully convolution network (FCN) to automatically detect and segment BM. In total, 1652 patients with 3D-T1-MPRAGE images from 3 hospitals (n = 1201, 231, and 220, respectively) were retrospectively included. Manual segmentations were obtained by a neuroradiologist and a radiation oncologist in a consensus reading in 3D-T1-MPRAGE images. Sensitivity, specificity, and dice ratio of the segmentation were evaluated. Specificity and sensitivity measure the fractions of relevant segmented voxels. Dice ratio was used to quantitatively measure the overlap between automatic and manual segmentation results. Paired samples t-tests and analysis of variance were employed for statistical analysis.

RESULTS

The BMDS net can detect all BM, providing a detection result with an accuracy of 100%. Automatic segmentations correlated strongly with manual segmentations through 4-fold cross-validation of the dataset with 1201 patients: the sensitivity was 0.96 ± 0.03 (range, 0.84-0.99), the specificity was 0.99 ± 0.0002 (range, 0.99-1.00), and the dice ratio was 0.85 ± 0.08 (range, 0.62-0.95) for total tumor volume. Similar performances on the other 2 datasets also demonstrate the robustness of BMDS net in correctly detecting and segmenting BM in various settings.

CONCLUSIONS

The BMDS net yields accurate detection and segmentation of BM automatically and could assist stereotactic radiotherapy management for diagnosis, therapy planning, and follow-up.