Crossbar-Net: A Novel Convolutional Neural Network for Kidney Tumor Segmentation in CT Images

STC-UNet: renal tumor segmentation based on enhanced feature extraction at different network levels

Renal tumors are one of the common diseases of urology, and precise segmentation of these tumors plays a crucial role in aiding physicians to improve diagnostic accuracy and treatment effectiveness. Nevertheless, inherent challenges associated with renal tumors, such as indistinct boundaries, morphological variations, and uncertainties in size and location, segmenting renal tumors accurately remains a significant challenge in the field of medical image segmentation. With the development of deep learning, substantial achievements have been made in the domain of medical image segmentation. However, existing models lack specificity in extracting features of renal tumors across different network hierarchies, which results in insufficient extraction of renal tumor features and subsequently affects the accuracy of renal tumor segmentation. To address this issue, we propose the Selective Kernel, Vision Transformer, and Coordinate Attention Enhanced U-Net (STC-UNet). This model aims to enhance feature extraction, adapting to the distinctive characteristics of renal tumors across various network levels. Specifically, the Selective Kernel modules are introduced in the shallow layers of the U-Net, where detailed features are more abundant. By selectively employing convolutional kernels of different scales, the model enhances its capability to extract detailed features of renal tumors across multiple scales. Subsequently, in the deeper layers of the network, where feature maps are smaller yet contain rich semantic information, the Vision Transformer modules are integrated in a non-patch manner. These assist the model in capturing long-range contextual information globally. Their non-patch implementation facilitates the capture of fine-grained features, thereby achieving collaborative enhancement of global-local information and ultimately strengthening the model's extraction of semantic features of renal tumors. Finally, in the decoder segment, the Coordinate Attention modules embedding positional information are proposed aiming to enhance the model's feature recovery and tumor region localization capabilities. Our model is validated on the KiTS19 dataset, and experimental results indicate that compared to the baseline model, STC-UNet shows improvements of 1.60%, 2.02%, 2.27%, 1.18%, 1.52%, and 1.35% in IoU, Dice, Accuracy, Precision, Recall, and F1-score, respectively. Furthermore, the experimental results demonstrate that the proposed STC-UNet method surpasses other advanced algorithms in both visual effectiveness and objective evaluation metrics.

Multi-phase features interaction transformer network for liver tumor segmentation and microvascular invasion assessment in contrast-enhanced CT

Precise segmentation of liver tumors from computed tomography (CT) scans is a prerequisite step in various clinical applications. Multi-phase CT imaging enhances tumor characterization, thereby assisting radiologists in accurate identification. However, existing automatic liver tumor segmentation models did not fully exploit multi-phase information and lacked the capability to capture global information. In this study, we developed a pioneering multi-phase feature interaction Transformer network (MI-TransSeg) for accurate liver tumor segmentation and a subsequent microvascular invasion (MVI) assessment in contrast-enhanced CT images. In the proposed network, an efficient multi-phase features interaction module was introduced to enable bi-directional feature interaction among multiple phases, thus maximally exploiting the available multi-phase information. To enhance the model's capability to extract global information, a hierarchical transformer-based encoder and decoder architecture was designed. Importantly, we devised a multi-resolution scales feature aggregation strategy (MSFA) to optimize the parameters and performance of the proposed model. Subsequent to segmentation, the liver tumor masks generated by MI-TransSeg were applied to extract radiomic features for the clinical applications of the MVI assessment. With Institutional Review Board (IRB) approval, a clinical multi-phase contrast-enhanced CT abdominal dataset was collected that included 164 patients with liver tumors. The experimental results demonstrated that the proposed MI-TransSeg was superior to various state-of-the-art methods. Additionally, we found that the tumor mask predicted by our method showed promising potential in the assessment of microvascular invasion. In conclusion, MI-TransSeg presents an innovative paradigm for the segmentation of complex liver tumors, thus underscoring the significance of multi-phase CT data exploitation. The proposed MI-TransSeg network has the potential to assist radiologists in diagnosing liver tumors and assessing microvascular invasion.

Ultrasound image segmentation of renal tumors based on UNet++ with fusion of multiscale residuals and dual attention

Objective. Laparoscopic renal unit-preserving resection is a routine and effective means of treating renal tumors. Image segmentation is an essential part before tumor resection. The current segmentation method mainly relies on doctors manual delineation, which is time-consuming, labor-intensive, and influenced by their personal experience and ability. And the image quality of segmentation is low, with problems such as blurred edges, unclear size and shape, which are not conducive to clinical diagnosis.Approach. To address these problems, we propose an automated segmentation method, i.e. the UNet++ algorithm fusing multiscale residuals and dual attention (MRDA_UNet++). It replaces two consecutive 3 × 3 convolutions in UNet++ with the 'MultiRes block' module, which incorporates coordinate attention to fuse features from different scales and suppress the impact of background noise. Furthermore, an attention gate is also added at the short connections to enhance the ability of the network to extract features from the target area.Main results. The experimental results show that MRDA_UNet++ achieves 93.18%, 92.87%, 93.66%, and 92.09% on the real-world dataset for MIoU, Dice, Precision, and Recall, respectively. Compared to the baseline model UNet++ on three public datasets, the MIoU, Dice, and Recall metrics improved by 6.00%, 7.90% and 18.09% respectively for BUSI, 0.39%, 0.27% and 1.03% for Dataset C, and 1.37%, 1.75% and 1.30% for DDTI.Significance. The proposed MRDA_UNet++ exhibits obvious advantages in feature extraction, which can not only significantly reduce the workload of doctors, but also further decrease the risk of misdiagnosis. It is of great value to assist doctors diagnosis in the clinic.

Rethinking automatic segmentation of gross target volume from a decoupling perspective

Accurate and reliable segmentation of Gross Target Volume (GTV) is critical in cancer Radiation Therapy (RT) planning, but manual delineation is time-consuming and subject to inter-observer variations. Recently, deep learning methods have achieved remarkable success in medical image segmentation. However, due to the low image contrast and extreme pixel imbalance between GTV and adjacent tissues, most existing methods usually obtained limited performance on automatic GTV segmentation. In this paper, we propose a Heterogeneous Cascade Framework (HCF) from a decoupling perspective, which decomposes the GTV segmentation into independent recognition and segmentation subtasks. The former aims to screen out the abnormal slices containing GTV, while the latter performs pixel-wise segmentation of these slices. With the decoupled two-stage framework, we can efficiently filter normal slices to reduce false positives. To further improve the segmentation performance, we design a multi-level Spatial Alignment Network (SANet) based on the feature pyramid structure, which introduces a spatial alignment module into the decoder to compensate for the information loss caused by downsampling. Moreover, we propose a Combined Regularization (CR) loss and Balance-Sampling Strategy (BSS) to alleviate the pixel imbalance problem and improve network convergence. Extensive experiments on two public datasets of StructSeg2019 challenge demonstrate that our method outperforms state-of-the-art methods, especially with significant advantages in reducing false positives and accurately segmenting small objects. The code is available at https://github.com/shijun18/GTV_AutoSeg.

Learning From Partially Labeled Data for Multi-Organ and Tumor Segmentation

Medical image benchmarks for the segmentation of organs and tumors suffer from the partially labeling issue due to its intensive cost of labor and expertise. Current mainstream approaches follow the practice of one network solving one task. With this pipeline, not only the performance is limited by the typically small dataset of a single task, but also the computation cost linearly increases with the number of tasks. To address this, we propose a Transformer based dynamic on-demand network (TransDoDNet) that learns to segment organs and tumors on multiple partially labeled datasets. Specifically, TransDoDNet has a hybrid backbone that is composed of the convolutional neural network and Transformer. A dynamic head enables the network to accomplish multiple segmentation tasks flexibly. Unlike existing approaches that fix kernels after training, the kernels in the dynamic head are generated adaptively by the Transformer, which employs the self-attention mechanism to model long-range organ-wise dependencies and decodes the organ embedding that can represent each organ. We create a large-scale partially labeled Multi-Organ and Tumor Segmentation benchmark, termed MOTS, and demonstrate the superior performance of our TransDoDNet over other competitors on seven organ and tumor segmentation tasks. This study also provides a general 3D medical image segmentation model, which has been pre-trained on the large-scale MOTS benchmark and has demonstrated advanced performance over current predominant self-supervised learning methods.

Self-Supervised Tumor Segmentation With Sim2Real Adaptation

This paper targets on self-supervised tumor segmentation. We make the following contributions: (i) we take inspiration from the observation that tumors are often characterised independently of their contexts, we propose a novel proxy task "layer-decomposition", that closely matches the goal of the downstream task, and design a scalable pipeline for generating synthetic tumor data for pre-training; (ii) we propose a two-stage Sim2Real training regime for unsupervised tumor segmentation, where we first pre-train a model with simulated tumors, and then adopt a self-training strategy for downstream data adaptation; (iii) when evaluating on different tumor segmentation benchmarks, e.g. BraTS2018 for brain tumor segmentation and LiTS2017 for liver tumor segmentation, our approach achieves state-of-the-art segmentation performance under the unsupervised setting. While transferring the model for tumor segmentation under a low-annotation regime, the proposed approach also outperforms all existing self-supervised approaches; (iv) we conduct extensive ablation studies to analyse the critical components in data simulation, and validate the necessity of different proxy tasks. We demonstrate that, with sufficient texture randomization in simulation, model trained on synthetic data can effortlessly generalise to datasets with real tumors.

USBDAN: Unsupervised Scale-aware and Boundary-aware Domain Adaptive Network for Gastric Tumor Segmentation

Accurate segmentation of gastric tumors from computed tomography (CT) images provides useful image information for guiding the diagnosis and treatment of gastric cancer. Researchers typically collect datasets from multiple medical centers to increase sample size and representation, but this raises the issue of data heterogeneity. To this end, we propose a new cross-center 3D tumor segmentation method named unsupervised scale-aware and boundary-aware domain adaptive network (USBDAN), which includes a new 3D neural network that efficiently bridges an Anisotropic neural network and a Transformer (AsTr) for extracting multi-scale features from the CT images with anisotropic resolution, and a scale-aware and boundary-aware domain alignment (SaBaDA) module for adaptively aligning multi-scale features between two domains and enhancing tumor boundary drawing based on location-related information drawn from each sample across all domains. We evaluate the proposed method on an in-house CT image dataset collected from four medical centers. Our results demonstrate that the proposed method outperforms several state-of-the-art methods.

Kidney Cancer Diagnosis and Surgery Selection by Machine Learning from CT Scans Combined with Clinical Metadata

Kidney cancers are one of the most common malignancies worldwide. Accurate diagnosis is a critical step in the management of kidney cancer patients and is influenced by multiple factors including tumor size or volume, cancer types and stages, etc. For malignant tumors, partial or radical surgery of the kidney might be required, but for clinicians, the basis for making this decision is often unclear. Partial nephrectomy could result in patient death due to cancer if kidney removal was necessary, whereas radical nephrectomy in less severe cases could resign patients to lifelong dialysis or need for future transplantation without sufficient cause. Using machine learning to consider clinical data alongside computed tomography images could potentially help resolve some of these surgical ambiguities, by enabling a more robust classification of kidney cancers and selection of optimal surgical approaches. In this study, we used the publicly available KiTS dataset of contrast-enhanced CT images and corresponding patient metadata to differentiate four major classes of kidney cancer: clear cell (ccRCC), chromophobe (chRCC), papillary (pRCC) renal cell carcinoma, and oncocytoma (ONC). We rationalized these data to overcome the high field of view (FoV), extract tumor regions of interest (ROIs), classify patients using deep machine-learning models, and extract/post-process CT image features for combination with clinical data. Regardless of marked data imbalance, our combined approach achieved a high level of performance (85.66% accuracy, 84.18% precision, 85.66% recall, and 84.92% F1-score). When selecting surgical procedures for malignant tumors (RCC), our method proved even more reliable (90.63% accuracy, 90.83% precision, 90.61% recall, and 90.50% F1-score). Using feature ranking, we confirmed that tumor volume and cancer stage are the most relevant clinical features for predicting surgical procedures. Once fully mature, the approach we propose could be used to assist surgeons in performing nephrectomies by guiding the choices of optimal procedures in individual patients with kidney cancer.

AI-Driven Robust Kidney and Renal Mass Segmentation and Classification on 3D CT Images

Early intervention in kidney cancer helps to improve survival rates. Abdominal computed tomography (CT) is often used to diagnose renal masses. In clinical practice, the manual segmentation and quantification of organs and tumors are expensive and time-consuming. Artificial intelligence (AI) has shown a significant advantage in assisting cancer diagnosis. To reduce the workload of manual segmentation and avoid unnecessary biopsies or surgeries, in this paper, we propose a novel end-to-end AI-driven automatic kidney and renal mass diagnosis framework to identify the abnormal areas of the kidney and diagnose the histological subtypes of renal cell carcinoma (RCC). The proposed framework first segments the kidney and renal mass regions by a 3D deep learning architecture (Res-UNet), followed by a dual-path classification network utilizing local and global features for the subtype prediction of the most common RCCs: clear cell, chromophobe, oncocytoma, papillary, and other RCC subtypes. To improve the robustness of the proposed framework on the dataset collected from various institutions, a weakly supervised learning schema is proposed to leverage the domain gap between various vendors via very few CT slice annotations. Our proposed diagnosis system can accurately segment the kidney and renal mass regions and predict tumor subtypes, outperforming existing methods on the KiTs19 dataset. Furthermore, cross-dataset validation results demonstrate the robustness of datasets collected from different institutions trained via the weakly supervised learning schema.

CAT-Net: A Cross-Slice Attention Transformer Model for Prostate Zonal Segmentation in MRI

Prostate cancer is the second leading cause of cancer death among men in the United States. The diagnosis of prostate MRI often relies on accurate prostate zonal segmentation. However, state-of-the-art automatic segmentation methods often fail to produce well-contained volumetric segmentation of the prostate zones since certain slices of prostate MRI, such as base and apex slices, are harder to segment than other slices. This difficulty can be overcome by leveraging important multi-scale image-based information from adjacent slices, but current methods do not fully learn and exploit such cross-slice information. In this paper, we propose a novel cross-slice attention mechanism, which we use in a Transformer module to systematically learn cross-slice information at multiple scales. The module can be utilized in any existing deep-learning-based segmentation framework with skip connections. Experiments show that our cross-slice attention is able to capture cross-slice information significant for prostate zonal segmentation in order to improve the performance of current state-of-the-art methods. Cross-slice attention improves segmentation accuracy in the peripheral zones, such that segmentation results are consistent across all the prostate slices (apex, mid-gland, and base). The code for the proposed model is available at https://bit.ly/CAT-Net.

Efficient Combination of CNN and Transformer for Dual-Teacher Uncertainty-guided Semi-supervised Medical Image Segmentation

BACKGROUND AND OBJECTIVE

Deep learning-based methods for fast target segmentation of magnetic resonance imaging (MRI) have become increasingly popular in recent years. Generally, the success of deep learning methods in medical image segmentation tasks relies on a large amount of labeled data. The time-consuming and labor-intensive problem of data annotation is a major challenge in medical image segmentation tasks. The aim of this work is to enhance the segmentation of MR images using a semi-supervised learning-based method using a small amount of labeled data and a large amount of unlabeled data.

METHODS

To utilize the effective information of the unlabeled data, we designed the method of guiding the Student segmentation model simultaneously by the Dual-Teacher structure of CNN and transformer forming the subject network. Both Teacher A and Student models are CNNs, and the TA-S module they form is a mean teacher structure with added data noise. In the TB-S module formed by the combination of Student and Teacher B models, their backbone networks CNN and transformer capture the local and global information of the image at the same time, respectively, to create pseudo labels for each other and perform cross-supervision. The Dual-Teacher guides the Student through synchronous training and performs knowledge rectification and communication with each other through consistent regular constraints, which better utilizes the valid information in the unlabeled data. In addition, the segmentation predictions of Teacher A and Student and Teacher A and Teacher B are screened for uncertainty assessment during the training process to enhance the prediction accuracy and generalization of the model. This method uses the mechanism of simultaneous training of the synthetic structure composed of TA-S and TB-S modules to jointly guide the optimization of the Student model to obtain better segmentation ability.

RESULTS

We evaluated the proposed method on a publicly available MRI dataset from a cardiac segmentation competition organized by MICCAI in 2017. Compared with several existing state-of-the-art semi-supervised segmentation methods, the method achieves better segmentation results in terms of Dice coefficient and HD distance evaluation metrics of 0.878 and 4.9 mm and 0.886 and 5.0 mm, respectively, using a training set containing only 10% and 20% of labeled data.

CONCLUSION

This method fuses CNN and transformer to design a new Teacher-Student semi-supervised learning optimization strategy, which greatly improves the utilization of a large number of unlabeled medical images and the effectiveness of model segmentation results.

Crosslink-Net: Double-Branch Encoder Network via Fusing Vertical and Horizontal Convolutions for Medical Image Segmentation

Accurate image segmentation plays a crucial role in medical image analysis, yet it faces great challenges caused by various shapes, diverse sizes, and blurry boundaries. To address these difficulties, square kernel-based encoder-decoder architectures have been proposed and widely used, but their performance remains unsatisfactory. To further address these challenges, we present a novel double-branch encoder architecture. Our architecture is inspired by two observations. (1) Since the discrimination of the features learned via square convolutional kernels needs to be further improved, we propose utilizing nonsquare vertical and horizontal convolutional kernels in a double-branch encoder so that the features learned by both branches can be expected to complement each other. (2) Considering that spatial attention can help models to better focus on the target region in a large-sized image, we develop an attention loss to further emphasize the segmentation of small-sized targets. With the above two schemes, we develop a novel double-branch encoder-based segmentation framework for medical image segmentation, namely, Crosslink-Net, and validate its effectiveness on five datasets with experiments. The code is released at https://github.com/Qianyu1226/Crosslink-Net.

Adaptive soft erasure with edge self-attention for weakly supervised semantic segmentation: Thyroid ultrasound image case study

[S U M M A R Y] Weakly supervised segmentation for medical images ease the reliance of models on pixel-level annotation while advancing the field of computer-aided diagnosis. However, the differences in nodule size in thyroid ultrasound images and the limitations of class activation maps in weakly supervised segmentation methods typically lead to under- and/or over-segmentation problems in real predictions. To alleviate this problem, we propose a weakly supervised segmentation neural network approach. This new method is based on a dual branch soft erase module that expands the foreground response region while constraining the erroneous expansion of the foreground region by the enhancement of background features. The sensitivity of this neural network to the nodule scale size is further enhanced by the scale feature adaptation module, which in turn generates integral and high-quality segmentation masks. In addition, while the nodule area can be significantly expanded through soft erase module and scale feature adaptation module, the activation effect in the nodule edge area is still not satisfactory, so that we further add an edge-based attention mechanism to strengthen the nodule edge segmentation effect. The results of experiments performed on the thyroid ultrasound image dataset showed that our new approach significantly outperformed existing weakly supervised semantic segmentation methods, e.g., 5.9% and 6.3% more accurate than the second-based results in terms of Jaccard and Dice coefficients, respectively.

Inconsistency-Aware Uncertainty Estimation for Semi-Supervised Medical Image Segmentation

In semi-supervised medical image segmentation, most previous works draw on the common assumption that higher entropy means higher uncertainty. In this paper, we investigate a novel method of estimating uncertainty. We observe that, when assigned different misclassification costs in a certain degree, if the segmentation result of a pixel becomes inconsistent, this pixel shows a relative uncertainty in its segmentation. Therefore, we present a new semi-supervised segmentation model, namely, conservative-radical network (CoraNet in short) based on our uncertainty estimation and separate self-training strategy. In particular, our CoraNet model consists of three major components: a conservative-radical module (CRM), a certain region segmentation network (C-SN), and an uncertain region segmentation network (UC-SN) that could be alternatively trained in an end-to-end manner. We have extensively evaluated our method on various segmentation tasks with publicly available benchmark datasets, including CT pancreas, MR endocardium, and MR multi-structures segmentation on the ACDC dataset. Compared with the current state of the art, our CoraNet has demonstrated superior performance. In addition, we have also analyzed its connection with and difference from conventional methods of uncertainty estimation in semi-supervised medical image segmentation.

Kidney Tumor Semantic Segmentation Using Deep Learning: A Survey of State-of-the-Art

Cure rates for kidney cancer vary according to stage and grade; hence, accurate diagnostic procedures for early detection and diagnosis are crucial. Some difficulties with manual segmentation have necessitated the use of deep learning models to assist clinicians in effectively recognizing and segmenting tumors. Deep learning (DL), particularly convolutional neural networks, has produced outstanding success in classifying and segmenting images. Simultaneously, researchers in the field of medical image segmentation employ DL approaches to solve problems such as tumor segmentation, cell segmentation, and organ segmentation. Segmentation of tumors semantically is critical in radiation and therapeutic practice. This article discusses current advances in kidney tumor segmentation systems based on DL. We discuss the various types of medical images and segmentation techniques and the assessment criteria for segmentation outcomes in kidney tumor segmentation, highlighting their building blocks and various strategies.

ECSU-Net: An Embedded Clustering Sliced U-Net Coupled With Fusing Strategy for Efficient Intervertebral Disc Segmentation and Classification

Automatic vertebra segmentation from computed tomography (CT) image is the very first and a decisive stage in vertebra analysis for computer-based spinal diagnosis and therapy support system. However, automatic segmentation of vertebra remains challenging due to several reasons, including anatomic complexity of spine, unclear boundaries of the vertebrae associated with spongy and soft bones. Based on 2D U-Net, we have proposed an Embedded Clustering Sliced U-Net (ECSU-Net). ECSU-Net comprises of three modules named segmentation, intervertebral disc extraction (IDE) and fusion. The segmentation module follows an instance embedding clustering approach, where our three sliced sub-nets use axis of CT images to generate a coarse 2D segmentation along with embedding space with the same size of the input slices. Our IDE module is designed to classify vertebra and find the inter-space between two slices of segmented spine. Our fusion module takes the coarse segmentation (2D) and outputs the refined 3D results of vertebra. A novel adaptive discriminative loss (ADL) function is introduced to train the embedding space for clustering. In the fusion strategy, three modules are integrated via a learnable weight control component, which adaptively sets their contribution. We have evaluated classical and deep learning methods on Spineweb dataset-2. ECSU-Net has provided comparable performance to previous neural network based algorithms achieving the best segmentation dice score of 95.60% and classification accuracy of 96.20%, while taking less time and computation resources.

An Interval Iteration Based Multilevel Thresholding Algorithm for Brain MR Image Segmentation

In this paper, we propose an interval iteration multilevel thresholding method (IIMT). This approach is based on the Otsu method but iteratively searches for sub-regions of the image to achieve segmentation, rather than processing the full image as a whole region. Then, a novel multilevel thresholding framework based on IIMT for brain MR image segmentation is proposed. In this framework, the original image is first decomposed using a hybrid L₁ − L₀ layer decomposition method to obtain the base layer. Second, we use IIMT to segment both the original image and its base layer. Finally, the two segmentation results are integrated by a fusion scheme to obtain a more refined and accurate segmentation result. Experimental results showed that our proposed algorithm is effective, and outperforms the standard Otsu-based and other optimization-based segmentation methods.

Artificial intelligence in assessment of hepatocellular carcinoma treatment response

Artificial Intelligence (AI) continues to shape the practice of radiology, with imaging of hepatocellular carcinoma (HCC) being of no exception. This article prepared by members of the LI-RADS Treatment Response (TR LI-RADS) work group and associates, presents recent trends in the utility of AI applications for the volumetric evaluation and assessment of HCC treatment response. Various topics including radiomics, prognostic imaging findings, and locoregional therapy (LRT) specific issues will be discussed in the framework of HCC treatment response classification systems with focus on the Liver Reporting and Data System treatment response algorithm (LI-RADS TRA).

Deep learning for end-to-end kidney cancer diagnosis on multi-phase abdominal computed tomography

In 2020, it is estimated that 73,750 kidney cancer cases were diagnosed, and 14,830 people died from cancer in the United States. Preoperative multi-phase abdominal computed tomography (CT) is often used for detecting lesions and classifying histologic subtypes of renal tumor to avoid unnecessary biopsy or surgery. However, there exists inter-observer variability due to subtle differences in the imaging features of tumor subtypes, which makes decisions on treatment challenging. While deep learning has been recently applied to the automated diagnosis of renal tumor, classification of a wide range of subtype classes has not been sufficiently studied yet. In this paper, we propose an end-to-end deep learning model for the differential diagnosis of five major histologic subtypes of renal tumors including both benign and malignant tumors on multi-phase CT. Our model is a unified framework to simultaneously identify lesions and classify subtypes for the diagnosis without manual intervention. We trained and tested the model using CT data from 308 patients who underwent nephrectomy for renal tumors. The model achieved an area under the curve (AUC) of 0.889, and outperformed radiologists for most subtypes. We further validated the model on an independent dataset of 184 patients from The Cancer Imaging Archive (TCIA). The AUC for this dataset was 0.855, and the model performed comparably to the radiologists. These results indicate that our model can achieve similar or better diagnostic performance than radiologists in differentiating a wide range of renal tumors on multi-phase CT.

3D Multi-Attention Guided Multi-Task Learning Network for Automatic Gastric Tumor Segmentation and Lymph Node Classification

Automatic gastric tumor segmentation and lymph node (LN) classification not only can assist radiologists in reading images, but also provide image-guided clinical diagnosis and improve diagnosis accuracy. However, due to the inhomogeneous intensity distribution of gastric tumor and LN in CT scans, the ambiguous/missing boundaries, and highly variable shapes of gastric tumor, it is quite challenging to develop an automatic solution. To comprehensively address these challenges, we propose a novel 3D multi-attention guided multi-task learning network for simultaneous gastric tumor segmentation and LN classification, which makes full use of the complementary information extracted from different dimensions, scales, and tasks. Specifically, we tackle task correlation and heterogeneity with the convolutional neural network consisting of scale-aware attention-guided shared feature learning for refined and universal multi-scale features, and task-aware attention-guided feature learning for task-specific discriminative features. This shared feature learning is equipped with two types of scale-aware attention (visual attention and adaptive spatial attention) and two stage-wise deep supervision paths. The task-aware attention-guided feature learning comprises a segmentation-aware attention module and a classification-aware attention module. The proposed 3D multi-task learning network can balance all tasks by combining segmentation and classification loss functions with weight uncertainty. We evaluate our model on an in-house CT images dataset collected from three medical centers. Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, and obtains promising performance for tumor segmentation and LN classification. Moreover, to explore the generalization for other segmentation tasks, we also extend the proposed network to liver tumor segmentation in CT images of the MICCAI 2017 Liver Tumor Segmentation Challenge. Our implementation is released at https://github.com/infinite-tao/MA-MTLN.

2D medical image segmentation via learning multi-scale contextual dependencies

Automatic medical image segmentation plays an important role as a diagnostic aid in the identification of diseases and their treatment in clinical settings. Recently proposed methods based on Convolutional Neural Networks (CNNs) have demonstrated their potential in image processing tasks, including some medical image analysis tasks. Those methods can learn various feature representations with numerous weight-shared convolutional kernels, however, the missed diagnosis rate of regions of interest (ROIs) is still high in medical image segmentation. Two crucial factors behind this shortcoming, which have been overlooked, are small ROIs from medical images and the limited context information from the existing network models. In order to reduce the missed diagnosis rate of ROIs from medical images, we propose a new segmentation framework which enhances the representative capability of small ROIs (particularly in deep layers) and explicitly learns global contextual dependencies in multi-scale feature spaces. In particular, the local features and their global dependencies from each feature space are adaptively aggregated based on both the spatial and the channel dimensions. Moreover, some visualization comparisons of the learned features from our framework further boost neural networks' interpretability. Experimental results show that, in comparison to some popular medical image segmentation and general image segmentation methods, our proposed framework achieves the state-of-the-art performance on the liver tumor segmentation task with 91.18% Sensitivity, the COVID-19 lung infection segmentation task with 75.73% Sensitivity and the retinal vessel detection task with 82.68% Sensitivity. Moreover, it is possible to integrate (parts of) the proposed framework into most of the recently proposed Fully CNN-based models, in order to improve their effectiveness in medical image segmentation tasks.

Tumor attention networks: Better feature selection, better tumor segmentation

Compared with the traditional analysis of computed tomography scans, automatic liver tumor segmentation can supply precise tumor volumes and reduce the inter-observer variability in estimating the tumor size and the tumor burden, which could further assist physicians to make better therapeutic choices for hepatic diseases and monitoring treatment. Among current mainstream segmentation approaches, multi-layer and multi-kernel convolutional neural networks (CNNs) have attracted much attention in diverse biomedical/medical image segmentation tasks with remarkable performance. However, an arbitrary stacking of feature maps makes CNNs quite inconsistent in imitating the cognition and the visual attention of human beings for a specific visual task. To mitigate the lack of a reasonable feature selection mechanism in CNNs, we exploit a novel and effective network architecture, called Tumor Attention Networks (TA-Net), for mining adaptive features by embedding Tumor Attention layers with multi-functional modules to assist the liver tumor segmentation task. In particular, each tumor attention layer can adaptively highlight valuable tumor features and suppress unrelated ones among feature maps from 3D and 2D perspectives. Moreover, an analysis of visualization results illustrates the effectiveness of our tumor attention modules and the interpretability of CNNs for liver tumor segmentation. Furthermore, we explore different arrangements of skip connections in information fusion. A deep ablation study is also conducted to illustrate the effects of different attention strategies for hepatic tumors. The results of extensive experiments demonstrate that the proposed TA-Net increases the liver tumor segmentation performance with a lower computation cost and a small parameter overhead over the state-of-the-art methods, under various evaluation metrics on clinical benchmark data. In addition, two additional medical image datasets are used to evaluate generalization capability of TA-Net, including the comparison with general semantic segmentation methods and a non-tumor segmentation task. All the program codes have been released at https://github.com/shuchao1212/TA-Net.

3D PBV-Net: An automated prostate MRI data segmentation method

Prostate cancer is one of the most common deadly diseases in men worldwide, which is seriously affecting people's life and health. Reliable and automated segmentation of the prostate gland in MRI data is exceptionally critical for diagnosis and treatment planning of prostate cancer. Although many automated segmentation methods have emerged, including deep learning based approaches, segmentation performance is still poor due to the large variability of image appearance, anisotropic spatial resolution, and imaging interference. This study proposes an automated prostate MRI data segmentation approach using bicubic interpolation with improved 3D V-Net (dubbed 3D PBV-Net). Considering the low-frequency components in the prostate gland, the bicubic interpolation is applied to preprocess the MRI data. On this basis, a 3D PBV-Net is developed to perform prostate MRI data segmentation. To illustrate the effectiveness of our approach, we evaluate the proposed 3D PBV-Net on two clinical prostate MRI data datasets, i.e., PROMISE 12 and TPHOH, with the manual delineations available as the ground truth. Our approach generates promising segmentation results, which have achieved 97.65% and 98.29% of average accuracy, 0.9613 and 0.9765 of Dice metric, 3.120 mm and 0.9382 mm of Hausdorff distance, and average boundary distance of 1.708, 0.7950 on PROMISE 12 and TPHOH datasets, respectively. Our method has effectively improved the accuracy of automated segmentation of the prostate MRI data and is promising to meet the accuracy requirements for telehealth applications.

A Single-Shot Region-Adaptive Network for Myotendinous Junction Segmentation in Muscular Ultrasound Images

Tracking the myotendinous junction (MTJ) in consecutive ultrasound images is crucial for understanding the mechanics and pathological conditions of the muscle-tendon unit. However, the lack of reliable and efficient identification of MTJ due to poor image quality and boundary ambiguity restricts its application in motion analysis. In recent years, with the rapid development of deep learning, the region-based convolution neural network (RCNN) has shown great potential in the field of simultaneous objection detection and instance segmentation in medical images. This article proposes a region-adaptive network (RAN) to localize MTJ region and to segment it in a single shot. Our model learns about the salient information of MTJ with the help of a composite architecture. Herein, a region-based multitask learning network explores the region containing MTJ, while a parallel end-to-end U-shaped path extracts the MTJ structure from the adaptively selected region for combating data imbalance and boundary ambiguity. By demonstrating the ultrasound images of the gastrocnemius, we showed that the RAN achieves superior segmentation performance when compared with the state-of-the-art Mask RCNN method with an average Dice score of 80.1%. Our proposed method is robust and reliable for advanced muscle and tendon function examinations obtained by ultrasound imaging.

Kidney and Renal Tumor Segmentation Using a Hybrid V-Net-Based Model

Kidney tumors represent a type of cancer that people of advanced age are more likely to develop. For this reason, it is important to exercise caution and provide diagnostic tests in the later stages of life. Medical imaging and deep learning methods are becoming increasingly attractive in this sense. Developing deep learning models to help physicians identify tumors with successful segmentation is of great importance. However, not many successful systems exist for soft tissue organs, such as the kidneys and the prostate, of which segmentation is relatively difficult. In such cases where segmentation is difficult, V-Net-based models are mostly used. This paper proposes a new hybrid model using the superior features of existing V-Net models. The model represents a more successful system with improvements in the encoder and decoder phases not previously applied. We believe that this new hybrid V-Net model could help the majority of physicians, particularly those focused on kidney and kidney tumor segmentation. The proposed model showed better performance in segmentation than existing imaging models and can be easily integrated into all systems due to its flexible structure and applicability. The hybrid V-Net model exhibited average Dice coefficients of 97.7% and 86.5% for kidney and tumor segmentation, respectively, and, therefore, could be used as a reliable method for soft tissue organ segmentation.

Construction of a convolutional neural network classifier developed by computed tomography images for pancreatic cancer diagnosis

BACKGROUND

Efforts should be made to develop a deep-learning diagnosis system to distinguish pancreatic cancer from benign tissue due to the high morbidity of pancreatic cancer.

AIM

To identify pancreatic cancer in computed tomography (CT) images automatically by constructing a convolutional neural network (CNN) classifier.

METHODS

A CNN model was constructed using a dataset of 3494 CT images obtained from 222 patients with pathologically confirmed pancreatic cancer and 3751 CT images from 190 patients with normal pancreas from June 2017 to June 2018. We established three datasets from these images according to the image phases, evaluated the approach in terms of binary classification (i.e., cancer or not) and ternary classification (i.e., no cancer, cancer at tail/body, cancer at head/neck of the pancreas) using 10-fold cross validation, and measured the effectiveness of the model with regard to the accuracy, sensitivity, and specificity.

RESULTS

The overall diagnostic accuracy of the trained binary classifier was 95.47%, 95.76%, 95.15% on the plain scan, arterial phase, and venous phase, respectively. The sensitivity was 91.58%, 94.08%, 92.28% on three phases, with no significant differences (χ² = 0.914, P = 0.633). Considering that the plain phase had same sensitivity, easier access, and lower radiation compared with arterial phase and venous phase , it is more sufficient for the binary classifier. Its accuracy on plain scans was 95.47%, sensitivity was 91.58%, and specificity was 98.27%. The CNN and board-certified gastroenterologists achieved higher accuracies than trainees on plain scan diagnosis (χ² = 21.534, P < 0.001; χ² = 9.524, P < 0.05; respectively). However, the difference between CNN and gastroenterologists was not significant (χ² = 0.759, P = 0.384). In the trained ternary classifier, the overall diagnostic accuracy of the ternary classifier CNN was 82.06%, 79.06%, and 78.80% on plain phase, arterial phase, and venous phase, respectively. The sensitivity scores for detecting cancers in the tail were 52.51%, 41.10% and, 36.03%, while sensitivity for cancers in the head was 46.21%, 85.24% and 72.87% on three phases, respectively. Difference in sensitivity for cancers in the head among the three phases was significant (χ² = 16.651, P < 0.001), with arterial phase having the highest sensitivity.

CONCLUSION

We proposed a deep learning-based pancreatic cancer classifier trained on medium-sized datasets of CT images. It was suitable for screening purposes in pancreatic cancer detection.

Machine Learning in Radiomic Renal Mass Characterization: Fundamentals, Applications, Challenges, and Future Directions

OBJECTIVE. The purpose of this study is to provide an overview of the traditional machine learning (ML)-based and deep learning-based radiomic approaches, with focus placed on renal mass characterization. CONCLUSION. ML currently has a very low barrier to entry into general medical practice because of the availability of many open-source, free, and easy-to-use toolboxes. Therefore, it should not be surprising to see its related applications in renal mass characterization. A wider picture of the previous works might be beneficial to move this field forward.

MB-FSGAN: Joint segmentation and quantification of kidney tumor on CT by the multi-branch feature sharing generative adversarial network

The segmentation of the kidney tumor and the quantification of its tumor indices (i.e., the center point coordinates, diameter, circumference, and cross-sectional area of the tumor) are important steps in tumor therapy. These quantifies the tumor morphometrical details to monitor disease progression and accurately compare decisions regarding the kidney tumor treatment. However, manual segmentation and quantification is a challenging and time-consuming process in practice and exhibit a high degree of variability within and between operators. In this paper, MB-FSGAN (multi-branch feature sharing generative adversarial network) is proposed for simultaneous segmentation and quantification of kidney tumor on CT. MB-FSGAN consists of multi-scale feature extractor (MSFE), locator of the area of interest (LROI), and feature sharing generative adversarial network (FSGAN). MSFE makes strong semantic information on different scale feature maps, which is particularly effective in detecting small tumor targets. The LROI extracts the region of interest of the tumor, greatly reducing the time complexity of the network. FSGAN correctly segments and quantifies kidney tumors through joint learning and adversarial learning, which effectively exploited the commonalities and differences between the two related tasks. Experiments are performed on CT of 113 kidney tumor patients. For segmentation, MB-FSGAN achieves a pixel accuracy of 95.7%. For the quantification of five tumor indices, the R² coefficient of tumor circumference is 0.9465. The results show that the network has reliable performance and shows its effectiveness and potential as a clinical tool.

Feasibility of Automated Volumetric Assessment of Large Hepatocellular Carcinomas' Responses to Transarterial Chemoembolization

Background: Hepatocellular carcinoma (HCC) is the most common liver malignancy and the leading cause of death in patients with cirrhosis. Various treatments for HCC are available, including transarterial chemoembolization (TACE), which is the commonest intervention performed in HCC. Radiologic tumor response following TACE is an important prognostic factor for patients with HCC. We hypothesized that, for large HCC tumors, assessment of treatment response made with automated volumetric response evaluation criteria in solid tumors (RECIST) might correlate with the assessment made with the more time- and labor-intensive unidimensional modified RECIST (mRECIST) and manual volumetric RECIST (M-vRECIST) criteria. Accordingly, we undertook this retrospective study to compare automated volumetric RECIST (A-vRECIST) with M-vRECIST and mRESIST for the assessment of large HCC tumors' responses to TACE. Methods:We selected 42 pairs of contrast-enhanced computed tomography (CT) images of large HCCs. Images were taken before and after TACE, and in each of the images, the HCC was segmented using both a manual contouring tool and a convolutional neural network. Three experienced radiologists assessed tumor response to TACE using mRECIST criteria. The intra-class correlation coefficient was used to assess inter-reader reliability in the mRECIST measurements, while the Pearson correlation coefficient was used to assess correlation between the volumetric and mRECIST measurements. Results:Volumetric tumor assessment using automated and manual segmentation tools showed good correlation with mRECIST measurements. For A-vRECIST and M-vRECIST, respectively, r = 0.597 vs. 0.622 in the baseline studies; 0.648 vs. 0.748 in the follow-up studies; and 0.774 vs. 0.766 in the response assessment (P < 0.001 for all). The A-vRECIST evaluation showed high correlation with the M-vRECIST evaluation (r = 0.967, 0.937, and 0.826 in baseline studies, follow-up studies, and response assessment, respectively, P < 0.001 for all). Conclusion:Volumetric RECIST measurements are likely to provide an early marker for TACE monitoring, and automated measurements made with a convolutional neural network may be good substitutes for manual volumetric measurements.

Learning from Synthetic Images via Active Pseudo-Labeling

Synthetic visual data refers to the data automatically rendered by the mature computer graphic algorithms. With the rapid development of these techniques, we can now collect photo-realistic synthetic images with accurate pixel-level annotations without much effort. However, due to the domain gaps between synthetic data and real data, in terms of not only visual appearance but also label distribution, directly applying models trained on synthetic images to real ones can hardly yield satisfactory performance. Since the collection of accurate labels for real images is very laborious and time-consuming, developing algorithms which can learn from synthetic images is of great significance. In this paper, we propose a novel framework, namely Active Pseudo-Labeling (APL), to reduce the domain gaps between synthetic images and real images. In APL framework, we first predict pseudo-labels for the unlabeled real images in the target domain by actively adapting the style of the real images to source domain. Specifically, the style of real images is adjusted via a novel task guided generative model, and then pseudo-labels are predicted for these actively adapted images. Lastly, we fine-tune the source-trained model in the pseudo-labeled target domain, which helps to fit the distribution of the real data. Experiments on both semantic segmentation and object detection tasks with several challenging benchmark data sets demonstrate the priority of our proposed method compared to the existing state-of-the-art approaches.

Weakly-supervised convolutional neural networks of renal tumor segmentation in abdominal CTA images

Background

Renal cancer is one of the 10 most common cancers in human beings. The laparoscopic partial nephrectomy (LPN) is an effective way to treat renal cancer. Localization and delineation of the renal tumor from pre-operative CT Angiography (CTA) is an important step for LPN surgery planning. Recently, with the development of the technique of deep learning, deep neural networks can be trained to provide accurate pixel-wise renal tumor segmentation in CTA images. However, constructing the training dataset with a large amount of pixel-wise annotations is a time-consuming task for the radiologists. Therefore, weakly-supervised approaches attract more interest in research.

Methods

In this paper, we proposed a novel weakly-supervised convolutional neural network (CNN) for renal tumor segmentation. A three-stage framework was introduced to train the CNN with the weak annotations of renal tumors, i.e. the bounding boxes of renal tumors. The framework includes pseudo masks generation, group and weighted training phases. Clinical abdominal CT angiographic images of 200 patients were applied to perform the evaluation.

Results

Extensive experimental results show that the proposed method achieves a higher dice coefficient (DSC) of 0.826 than the other two existing weakly-supervised deep neural networks. Furthermore, the segmentation performance is close to the fully supervised deep CNN.

Conclusions

The proposed strategy improves not only the efficiency of network training but also the precision of the segmentation.