H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes

Accurate and Efficient Intracranial Hemorrhage Detection and Subtype Classification in 3D CT Scans with Convolutional and Long Short-Term Memory Neural Networks

In this paper, we present our system for the RSNA Intracranial Hemorrhage Detection challenge, which is based on the RSNA 2019 Brain CT Hemorrhage dataset. The proposed system is based on a lightweight deep neural network architecture composed of a convolutional neural network (CNN) that takes as input individual CT slices, and a Long Short-Term Memory (LSTM) network that takes as input multiple feature embeddings provided by the CNN. For efficient processing, we consider various feature selection methods to produce a subset of useful CNN features for the LSTM. Furthermore, we reduce the CT slices by a factor of 2×, which enables us to train the model faster. Even if our model is designed to balance speed and accuracy, we report a weighted mean log loss of 0.04989 on the final test set, which places us in the top 30 ranking (2%) from a total of 1345 participants. While our computing infrastructure does not allow it, processing CT slices at their original scale is likely to improve performance. In order to enable others to reproduce our results, we provide our code as open source. After the challenge, we conducted a subjective intracranial hemorrhage detection assessment by radiologists, indicating that the performance of our deep model is on par with that of doctors specialized in reading CT scans. Another contribution of our work is to integrate Grad-CAM visualizations in our system, providing useful explanations for its predictions. We therefore consider our system as a viable option when a fast diagnosis or a second opinion on intracranial hemorrhage detection are needed.

Pyramid graph cut: Integrating intensity and gradient information for grayscale medical image segmentation

Segmentation of grayscale medical images is challenging because of the similarity of pixel intensities and poor gradient strength between adjacent regions. The existing image segmentation approaches based on either intensity or gradient information alone often fail to produce accurate segmentation results. Previous approaches in the literature have approached the problem by embedded or sequential integration of different information types to improve the performance of the image segmentation on specific tasks. However, an effective combination or integration of such information is difficult to implement and not sufficiently generic for closely related tasks. Integration of the two information sources in a single graph structure is a potentially more effective way to solve the problem. In this paper we introduce a novel technique for grayscale medical image segmentation called pyramid graph cut, which combines intensity and gradient sources of information in a pyramid-shaped graph structure using a single source node and multiple sink nodes. The source node, which is the top of the pyramid graph, embeds intensity information into its linked edges. The sink nodes, which are the base of the pyramid graph, embed gradient information into their linked edges. The min-cut uses intensity information and gradient information, depending on which one is more useful or has a higher influence in each cutting location of each iteration. The experimental results demonstrate the effectiveness of the proposed method over intensity-based segmentation alone (i.e. Gaussian mixture model) and gradient-based segmentation alone (i.e. distance regularized level set evolution) on grayscale medical image datasets, including the public 3DIRCADb-01 dataset. The proposed method archives excellent segmentation results on the sample CT of abdominal aortic aneurysm, MRI of liver tumor and US of liver tumor, with dice scores of 90.49±5.23%, 88.86±11.77%, 90.68±2.45%, respectively.

Deep learning algorithms to isolate and quantify the structures of the anterior segment in optical coherence tomography images

BACKGROUND/AIMS

Accurate isolation and quantification of intraocular dimensions in the anterior segment (AS) of the eye using optical coherence tomography (OCT) images is important in the diagnosis and treatment of many eye diseases, especially angle-closure glaucoma.

METHOD

In this study, we developed a deep convolutional neural network (DCNN) for the localisation of the scleral spur; moreover, we introduced an information-rich segmentation approach for this localisation problem. An ensemble of DCNNs for the segmentation of AS structures (iris, corneosclera shell adn anterior chamber) was developed. Based on the results of two previous processes, an algorithm to automatically quantify clinically important measurements were created. 200 images from 58 patients (100 eyes) were used for testing.

RESULTS

With limited training data, the DCNN was able to detect the scleral spur on unseen anterior segment optical coherence tomography (ASOCT) images as accurately as an experienced ophthalmologist on the given test dataset and simultaneously isolated the AS structures with a Dice coefficient of 95.7%. We then automatically extracted eight clinically relevant ASOCT measurements and proposed an automated quality check process that asserts the reliability of these measurements. When combined with an OCT machine capable of imaging multiple radial sections, the algorithms can provide a more complete objective assessment. The total segmentation and measurement time for a single scan is less than 2 s.

CONCLUSION

This is an essential step towards providing a robust automated framework for reliable quantification of ASOCT scans, for applications in the diagnosis and management of angle-closure glaucoma.

CDA-Net for Automatic Prostate Segmentation in MR Images

Automatic and accurate prostate segmentation is an essential prerequisite for assisting diagnosis and treatment, such as guiding biopsy procedures and radiation therapy. Therefore, this paper proposes a cascaded dual attention network (CDA-Net) for automatic prostate segmentation in MRI scans. The network includes two stages of RAS-FasterRCNN and RAU-Net. Firstly, RAS-FasterRCNN uses improved FasterRCNN and sequence correlation processing to extract regions of interest (ROI) of organs. This ROI extraction serves as a hard attention mechanism to focus the segmentation of the subsequent network on a certain area. Secondly, the addition of residual convolution block and self-attention mechanism in RAU-Net enables the network to gradually focus on the area where the organ exists while making full use of multiscale features. The algorithm was evaluated on the PROMISE12 and ASPS13 datasets and presents the dice similarity coefficient of 92.88% and 92.65%, respectively, surpassing the state-of-the-art algorithms. In a variety of complex slice images, especially for the base and apex of slice sequences, the algorithm also achieved credible segmentation performance.

Road Extraction from Very-High-Resolution Remote Sensing Images via a Nested SE-Deeplab Model

Automatic road extraction from very-high-resolution remote sensing images has become a popular topic in a wide range of fields. Convolutional neural networks are often used for this purpose. However, many network models do not achieve satisfactory extraction results because of the elongated nature and varying sizes of roads in images. To improve the accuracy of road extraction, this paper proposes a deep learning model based on the structure of Deeplab v3. It incorporates squeeze-and-excitation (SE) module to apply weights to different feature channels, and performs multi-scale upsampling to preserve and fuse shallow and deep information. To solve the problems associated with unbalanced road samples in images, different loss functions and backbone network modules are tested in the model’s training process. Compared with cross entropy, dice loss can improve the performance of the model during training and prediction. The SE module is superior to ResNext and ResNet in improving the integrity of the extracted roads. Experimental results obtained using the Massachusetts Roads Dataset show that the proposed model (Nested SE-Deeplab) improves F1-Score by 2.4% and Intersection over Union by 2.0% compared with FC-DenseNet. The proposed model also achieves better segmentation accuracy in road extraction compared with other mainstream deep-learning models including Deeplab v3, SegNet, and UNet.

[Multi-scale 3D convolutional neural network-based segmentation of head and neck organs at risk]

OBJECTIVE

To establish an algorithm based on 3D convolution neural network to segment the organs at risk (OARs) in the head and neck on CT images.

METHODS

We propose an automatic segmentation algorithm of head and neck OARs based on V-Net. To enhance the feature expression ability of the 3D neural network, we combined the squeeze and exception (SE) module with the residual convolution module in V-Net to increase the weight of the features that has greater contributions to the segmentation task. Using a multi-scale strategy, we completed organ segmentation using two cascade models for location and fine segmentation, and the input image was resampled to different resolutions during preprocessing to allow the two models to focus on the extraction of global location information and local detail features respectively.

RESULTS

Our experiments on segmentation of 22 OARs in the head and neck indicated that compared with the existing methods, the proposed method achieved better segmentation accuracy and efficiency, and the average segmentation accuracy was improved by 9%. At the same time, the average test time was reduced from 33.82 s to 2.79 s.

CONCLUSIONS

The 3D convolution neural network based on multi-scale strategy can effectively and efficiently improve the accuracy of organ segmentation and can be potentially used in clinical setting for segmentation of other organs to improve the efficiency of clinical treatment.

Generation of Virtual Non-Contrast CT From Intravenous Enhanced CT in Radiotherapy Using Convolutional Neural Networks

Objective: To generate virtual non-contrast (VNC) computed tomography (CT) from intravenous enhanced CT through convolutional neural networks (CNN) and compare calculated dose among enhanced CT, VNC, and real non-contrast scanning.

Method: 50 patients who accepted non-contrast and enhanced CT scanning before and after intravenous contrast agent injections were selected, and two sets of CT images were registered. A total of 40 and 10 groups were used as training and test datasets, respectively. The U-Net architecture was applied to learn the relationship between the enhanced and non-contrast CT. VNC images were generated in the test through the trained U-Net. The CT values of non-contrast, enhanced and VNC CT images were compared. The radiotherapy treatment plans for esophageal cancer were designed, and dose calculation was performed. Dose distributions in the three image sets were compared.

Results: The mean absolute error of CT values between enhanced and non-contrast CT reached 32.3 ± 2.6 HU, and that between VNC and non-contrast CT totaled 6.7 ± 1.3 HU. The average CT values in enhanced CT of great vessels, heart, lungs, liver, and spinal cord were all significantly higher than those of non-contrast CT (p < 0.05), with the differences reaching 97, 83, 42, 40, and 10 HU, respectively. The average CT values of the organs in VNC CT showed no significant differences from those in non-contrast CT. The relative dose differences of the enhanced and non-contrast CT were −1.2, −1.3, −2.1, and −1.5% in the comparison of mean doses of planned target volume, heart, great vessels, and lungs, respectively. The mean dose calculated by VNC CT showed no significant difference from that by non-contrast CT. The average γ passing rate (2%, 2 mm) of VNC CT image was significantly higher than that of enhanced CT image (0.996 vs. 0.973, p < 0.05).

Conclusion: Designing a treatment plan based on enhanced CT will enlarge the dose calculation uncertainty in radiotherapy. This paper proposed the generation of VNC CT images from enhanced CT images based on U-Net architecture. The dose calculated through VNC CT images was identical with that obtained through real non-contrast CT.

3D Deep Learning on Medical Images: A Review

The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) CNNs have been employed for the analysis of medical images. In this paper, we trace the history of how the 3D CNN was developed from its machine learning roots, we provide a brief mathematical description of 3D CNN and provide the preprocessing steps required for medical images before feeding them to 3D CNNs. We review the significant research in the field of 3D medical imaging analysis using 3D CNNs (and its variants) in different medical areas such as classification, segmentation, detection and localization. We conclude by discussing the challenges associated with the use of 3D CNNs in the medical imaging domain (and the use of deep learning models in general) and possible future trends in the field.

Automatic multi-organ segmentation in computed tomography images using hierarchical convolutional neural network

Purpose: Accurate segmentation of treatment planning computed tomography (CT) images is important for radiation therapy (RT) planning. However, low soft tissue contrast in CT makes the segmentation task challenging. We propose a two-step hierarchical convolutional neural network (CNN) segmentation strategy to automatically segment multiple organs from CT. Approach: The first step generates a coarse segmentation from which organ-specific regions of interest (ROIs) are produced. The second step produces detailed segmentation of each organ. The ROIs are generated using UNet, which automatically identifies the area of each organ and improves computational efficiency by eliminating irrelevant background information. For the fine segmentation step, we combined UNet with a generative adversarial network. The generator is designed as a UNet that is trained to segment organ structures and the discriminator is a fully convolutional network, which distinguishes whether the segmentation is real or generator-predicted, thus improving the segmentation accuracy. We validated the proposed method on male pelvic and head and neck (H&N) CTs used for RT planning of prostate and H&N cancer, respectively. For the pelvic structure segmentation, the network was trained to segment the prostate, bladder, and rectum. For H&N, the network was trained to segment the parotid glands (PG) and submandibular glands (SMG). Results: The trained segmentation networks were tested on 15 pelvic and 20 H&N independent datasets. The H&N segmentation network was also tested on a public domain dataset ( N = 38 ) and showed similar performance. The average dice similarity coefficients ( mean ± SD ) of pelvic structures are 0.91 ± 0.05 (prostate), 0.95 ± 0.06 (bladder), 0.90 ± 0.09 (rectum), and H&N structures are 0.87 ± 0.04 (PG) and 0.86 ± 0.05 (SMG). The segmentation for each CT takes < 10    s on average. Conclusions: Experimental results demonstrate that the proposed method can produce fast, accurate, and reproducible segmentation of multiple organs of different sizes and shapes and show its potential to be applicable to different disease sites.

Block Level Skip Connections Across Cascaded V-Net for Multi-Organ Segmentation

Multi-organ segmentation is a challenging task due to the label imbalance and structural differences between different organs. In this work, we propose an efficient cascaded V-Net model to improve the performance of multi-organ segmentation by establishing dense Block Level Skip Connections (BLSC) across cascaded V-Net. Our model can take full advantage of features from the first stage network and make the cascaded structure more efficient. We also combine stacked small and large kernels with an inception-like structure to help our model to learn more patterns, which produces superior results for multi-organ segmentation. In addition, some small organs are commonly occluded by large organs and have unclear boundaries with other surrounding tissues, which makes them hard to be segmented. We therefore first locate the small organs through a multi-class network and crop them randomly with the surrounding region, then segment them with a single-class network. We evaluated our model on SegTHOR 2019 challenge unseen testing set and Multi-Atlas Labeling Beyond the Cranial Vault challenge validation set. Our model has achieved an average dice score gain of 1.62 percents and 3.90 percents compared to traditional cascaded networks on these two datasets, respectively. For hard-to-segment small organs, such as the esophagus in SegTHOR 2019 challenge, our technique has achieved a gain of 5.63 percents on dice score, and four organs in Multi-Atlas Labeling Beyond the Cranial Vault challenge have achieved a gain of 5.27 percents on average dice score.

Diffuse domain method for needle insertion simulations

We present a new strategy for needle insertion simulations without the necessity of meshing. A diffuse domain approach on a regular grid is applied to overcome the need for an explicit representation of organ boundaries. A phase field function captures the transition of tissue parameters and boundary conditions are imposed implicitly. Uncertainties of a volume segmentation are translated in the width of the phase field, an approach that is novel and overcomes the problem of defining an accurate segmentation boundary. We perform a convergence analysis of the diffuse elastic equation for decreasing phase field width, compare our results to deformation fields received from conforming mesh simulations and analyze the diffuse linear elastic equation for different widths of material interfaces. Then, the approach is applied to computed tomography data of a patient with liver tumors. A three-class U-Net is used to automatically generate tissue probability maps serving as phase field functions for the transition of elastic parameters between different tissues. The needle tissue interaction forces are approximated by the absolute gradient of a phase field function, which eliminates the need for explicit boundary parameterization and collision detection at the needle-tissue interface. The results show that the deformation field of the diffuse domain approach is comparable to the deformation of a conforming mesh simulation. Uncertainties of tissue boundaries are included in the model and the simulation can be directly performed on the automatically generated voxel-based probability maps. Thus, it is possible to perform easily implementable patient-specific elastomechanical simulations directly on voxel data.

Weakly Supervised Deep Learning for Whole Slide Lung Cancer Image Analysis

Histopathology image analysis serves as the gold standard for cancer diagnosis. Efficient and precise diagnosis is quite critical for the subsequent therapeutic treatment of patients. So far, computer-aided diagnosis has not been widely applied in pathological field yet as currently well-addressed tasks are only the tip of the iceberg. Whole slide image (WSI) classification is a quite challenging problem. First, the scarcity of annotations heavily impedes the pace of developing effective approaches. Pixelwise delineated annotations on WSIs are time consuming and tedious, which poses difficulties in building a large-scale training dataset. In addition, a variety of heterogeneous patterns of tumor existing in high magnification field are actually the major obstacle. Furthermore, a gigapixel scale WSI cannot be directly analyzed due to the immeasurable computational cost. How to design the weakly supervised learning methods to maximize the use of available WSI-level labels that can be readily obtained in clinical practice is quite appealing. To overcome these challenges, we present a weakly supervised approach in this article for fast and effective classification on the whole slide lung cancer images. Our method first takes advantage of a patch-based fully convolutional network (FCN) to retrieve discriminative blocks and provides representative deep features with high efficiency. Then, different context-aware block selection and feature aggregation strategies are explored to generate globally holistic WSI descriptor which is ultimately fed into a random forest (RF) classifier for the image-level prediction. To the best of our knowledge, this is the first study to exploit the potential of image-level labels along with some coarse annotations for weakly supervised learning. A large-scale lung cancer WSI dataset is constructed in this article for evaluation, which validates the effectiveness and feasibility of the proposed method. Extensive experiments demonstrate the superior performance of our method that surpasses the state-of-the-art approaches by a significant margin with an accuracy of 97.3%. In addition, our method also achieves the best performance on the public lung cancer WSIs dataset from The Cancer Genome Atlas (TCGA). We highlight that a small number of coarse annotations can contribute to further accuracy improvement. We believe that weakly supervised learning methods have great potential to assist pathologists in histology image diagnosis in the near future.

Automatic prostate and prostate zones segmentation of magnetic resonance images using DenseNet-like U-net

Magnetic resonance imaging (MRI) provides detailed anatomical images of the prostate and its zones. It has a crucial role for many diagnostic applications. Automatic segmentation such as that of the prostate and prostate zones from MR images facilitates many diagnostic and therapeutic applications. However, the lack of a clear prostate boundary, prostate tissue heterogeneity, and the wide interindividual variety of prostate shapes make this a very challenging task. To address this problem, we propose a new neural network to automatically segment the prostate and its zones. We term this algorithm Dense U-net as it is inspired by the two existing state-of-the-art tools-DenseNet and U-net. We trained the algorithm on 141 patient datasets and tested it on 47 patient datasets using axial T2-weighted images in a four-fold cross-validation fashion. The networks were trained and tested on weakly and accurately annotated masks separately to test the hypothesis that the network can learn even when the labels are not accurate. The network successfully detects the prostate region and segments the gland and its zones. Compared with U-net, the second version of our algorithm, Dense-2 U-net, achieved an average Dice score for the whole prostate of 92.1± 0.8% vs. 90.7 ± 2%, for the central zone of [Formula: see text]% vs. [Formula: see text] %, and for the peripheral zone of 78.1± 2.5% vs. [Formula: see text]%. Our initial results show Dense-2 U-net to be more accurate than state-of-the-art U-net for automatic segmentation of the prostate and prostate zones.

Fully automated multiorgan segmentation in abdominal magnetic resonance imaging with deep neural networks

PURPOSE

Segmentation of multiple organs-at-risk (OARs) is essential for magnetic resonance (MR)-only radiation therapy treatment planning and MR-guided adaptive radiotherapy of abdominal cancers. Current practice requires manual delineation that is labor-intensive, time-consuming, and prone to intra- and interobserver variations. We developed a deep learning (DL) technique for fully automated segmentation of multiple OARs on clinical abdominal MR images with high accuracy, reliability, and efficiency.

METHODS

We developed Automated deep Learning-based abdominal multiorgan segmentation (ALAMO) technique based on two-dimensional U-net and a densely connected network structure with tailored design in data augmentation and training procedures such as deep connection, auxiliary supervision, and multiview. The model takes in multislice MR images and generates the output of segmentation results. 3.0-Tesla T1 VIBE (Volumetric Interpolated Breath-hold Examination) images of 102 subjects were used in our study and split into 66 for training, 16 for validation, and 20 for testing. Ten OARs were studied, including the liver, spleen, pancreas, left/right kidneys, stomach, duodenum, small intestine, spinal cord, and vertebral bodies. An experienced radiologist manually labeled each OAR, followed by reediting, if necessary, by a senior radiologist, to create the ground-truth. The performance was measured using volume overlapping and surface distance.

RESULTS

The ALAMO technique generated segmentation labels in good agreement with the manual results. Specifically, among the ten OARs, nine achieved high dice similarity coefficients (DSCs) in the range of 0.87-0.96, except for the duodenum with a DSC of 0.80. The inference completed within 1 min for a three-dimensional volume of 320 × 288 × 180. Overall, the ALAMO model matched the state-of-the-art techniques in performance.

CONCLUSION

The proposed ALAMO technique allows for fully automated abdominal MR segmentation with high accuracy and practical memory and computation time demands.

Automatic Segmentation of Macular Edema in Retinal OCT Images Using Improved U-Net++

The number and volume of retinal macular edemas are important indicators for screening and diagnosing retinopathy. Aiming at the problem that the segmentation method of macular edemas in a retinal optical coherence tomography (OCT) image is not ideal in segmentation of diverse edemas, this paper proposes a new method of automatic segmentation of macular edema regions in retinal OCT images using the improved U-Net++. The proposed method makes full use of the U-Net++ re-designed skip pathways and dense convolution block; reduces the semantic gap of the feature maps in the encoder/decoder sub-network; and adds the improved Resnet network as the backbone, which make the extraction of features in the edema regions more accurate and improves the segmentation effect. The proposed method was trained and validated on the public dataset of Duke University, and the experiments demonstrated the proposed method can not only improve the overall segmentation effect, but also can significantly improve the segmented precision for diverse edema in multi-regions, as well as reducing the error of the number of edema regions.

s-LWSR: Super Lightweight Super-Resolution Network

In recent years, deep-based models have achieved great success in the field of single image super-resolution (SISR), where tremendous parameters are always needed to obtain a satisfying performance. However, the high computational complexity extremely limits its applications to some mobile devices that possess less computing and storage resources. To address this problem, in this paper, we propose a flexibly adjustable super lightweight SR network: s-LWSR. Firstly, in order to efficiently abstract features from the low resolution image, we design a high-efficient U-shape based block, where an information pool is constructed to mix multi-level information from the first half part of the pipeline. Secondly, a compression mechanism based on depth-wise separable convolution is employed to further reduce the numbers of parameters with negligible performance degradation. Thirdly, by revealing the specific role of activation in deep models, we remove several activation layers in our SR model to retain more information, thus leading to the final performance improvement. Extensive experiments show that our s-LWSR, with limited parameters and operations, can achieve similar performance compared with other cumbersome DL-SR methods.

Semantic Segmentation of White Matter in FDG-PET Using Generative Adversarial Network

In the diagnosis of neurodegenerative disorders, F-18 fluorodeoxyglucose positron emission tomography/computed tomography (18F-FDG PET/CT) is used for its ability to detect functional changes at early stages of disease process. However, anatomical information from another modality (CT or MRI) is still needed to properly interpret and localize the radiotracer uptake due to its low spatial resolution. Lack of structural information limits segmentation and accurate quantification of the 18F-FDG PET/CT. The correct segmentation of the brain compartment in 18F-FDG PET/CT will enable the quantitative analysis of the 18F-FDG PET/CT scan alone. In this paper, we propose a method to segment white matter in 18F-FDG PET/CT images using generative adversarial network (GAN). The segmentation result of GAN model was evaluated using evaluation parameters such as dice, AUC-PR, precision, and recall. It was also compared with other deep learning methods. As a result, the proposed method achieves superior segmentation accuracy and reliability compared with other deep learning methods.

Inf-Net: Automatic COVID-19 Lung Infection Segmentation From CT Images

Coronavirus Disease 2019 (COVID-19) spread globally in early 2020, causing the world to face an existential health crisis. Automated detection of lung infections from computed tomography (CT) images offers a great potential to augment the traditional healthcare strategy for tackling COVID-19. However, segmenting infected regions from CT slices faces several challenges, including high variation in infection characteristics, and low intensity contrast between infections and normal tissues. Further, collecting a large amount of data is impractical within a short time period, inhibiting the training of a deep model. To address these challenges, a novel COVID-19 Lung Infection Segmentation Deep Network (Inf-Net) is proposed to automatically identify infected regions from chest CT slices. In our Inf-Net, a parallel partial decoder is used to aggregate the high-level features and generate a global map. Then, the implicit reverse attention and explicit edge-attention are utilized to model the boundaries and enhance the representations. Moreover, to alleviate the shortage of labeled data, we present a semi-supervised segmentation framework based on a randomly selected propagation strategy, which only requires a few labeled images and leverages primarily unlabeled data. Our semi-supervised framework can improve the learning ability and achieve a higher performance. Extensive experiments on our COVID-SemiSeg and real CT volumes demonstrate that the proposed Inf-Net outperforms most cutting-edge segmentation models and advances the state-of-the-art performance.

Inf-Net: Automatic COVID-19 Lung Infection Segmentation From CT Images

Coronavirus Disease 2019 (COVID-19) spread globally in early 2020, causing the world to face an existential health crisis. Automated detection of lung infections from computed tomography (CT) images offers a great potential to augment the traditional healthcare strategy for tackling COVID-19. However, segmenting infected regions from CT slices faces several challenges, including high variation in infection characteristics, and low intensity contrast between infections and normal tissues. Further, collecting a large amount of data is impractical within a short time period, inhibiting the training of a deep model. To address these challenges, a novel COVID-19 Lung Infection Segmentation Deep Network (Inf-Net) is proposed to automatically identify infected regions from chest CT slices. In our Inf-Net, a parallel partial decoder is used to aggregate the high-level features and generate a global map. Then, the implicit reverse attention and explicit edge-attention are utilized to model the boundaries and enhance the representations. Moreover, to alleviate the shortage of labeled data, we present a semi-supervised segmentation framework based on a randomly selected propagation strategy, which only requires a few labeled images and leverages primarily unlabeled data. Our semi-supervised framework can improve the learning ability and achieve a higher performance. Extensive experiments on our COVID-SemiSeg and real CT volumes demonstrate that the proposed Inf-Net outperforms most cutting-edge segmentation models and advances the state-of-the-art performance.

Towards Efficient U-Nets: A Coupled and Quantized Approach

In this paper, we propose to couple stacked U-Nets for efficient visual landmark localization. The key idea is to globally reuse features of the same semantic meanings across the stacked U-Nets. The feature reuse makes each U-Net light-weighted. Specially, we propose an order- K coupling design to trim off long-distance shortcuts, together with an iterative refinement and memory sharing mechanism. To further improve the efficiency, we quantize the parameters, intermediate features, and gradients of the coupled U-Nets to low bit-width numbers. We validate our approach in two tasks: human pose estimation and facial landmark localization. The results show that our approach achieves state-of-the-art localization accuracy but using  ∼ 70% fewer parameters,  ∼ 30% less inference time,  ∼ 98% less model size, and saving  ∼ 75% training memory compared with benchmark localizers.

A Rapid, Accurate and Machine-Agnostic Segmentation and Quantification Method for CT-Based COVID-19 Diagnosis

COVID-19 has caused a global pandemic and become the most urgent threat to the entire world. Tremendous efforts and resources have been invested in developing diagnosis, prognosis and treatment strategies to combat the disease. Although nucleic acid detection has been mainly used as the gold standard to confirm this RNA virus-based disease, it has been shown that such a strategy has a high false negative rate, especially for patients in the early stage, and thus CT imaging has been applied as a major diagnostic modality in confirming positive COVID-19. Despite the various, urgent advances in developing artificial intelligence (AI)-based computer-aided systems for CT-based COVID-19 diagnosis, most of the existing methods can only perform classification, whereas the state-of-the-art segmentation method requires a high level of human intervention. In this paper, we propose a fully-automatic, rapid, accurate, and machine-agnostic method that can segment and quantify the infection regions on CT scans from different sources. Our method is founded upon two innovations: 1) the first CT scan simulator for COVID-19, by fitting the dynamic change of real patients' data measured at different time points, which greatly alleviates the data scarcity issue; and 2) a novel deep learning algorithm to solve the large-scene-small-object problem, which decomposes the 3D segmentation problem into three 2D ones, and thus reduces the model complexity by an order of magnitude and, at the same time, significantly improves the segmentation accuracy. Comprehensive experimental results over multi-country, multi-hospital, and multi-machine datasets demonstrate the superior performance of our method over the existing ones and suggest its important application value in combating the disease.

Cerebral Artery and Vein Segmentation in Four-dimensional CT Angiography Using Convolutional Neural Networks

Purpose

To implement and test a deep learning approach for the segmentation of the arterial and venous cerebral vasculature with four-dimensional (4D) CT angiography.

Materials and Methods

Patients who had undergone 4D CT angiography for the suspicion of acute ischemic stroke were retrospectively identified. A total of 390 patients evaluated in 2014 (n = 113) or 2018 (n = 277) were included in this study, with each patient having undergone one 4D CT angiographic scan. One hundred patients from 2014 were randomly selected, and the arteries and veins on their CT scans were manually annotated by five experienced observers. The weighted temporal average and weighted temporal variance from 4D CT angiography were used as input for a three-dimensional Dense-U-Net. The network was trained with the fully annotated cerebral vessel artery-vein maps from 60 patients. Forty patients were used for quantitative evaluation. The relative absolute volume difference and the Dice similarity coefficient are reported. The neural network segmentations from 277 patients who underwent scanning in 2018 were qualitatively evaluated by an experienced neuroradiologist using a five-point scale.

Results

The average time for processing arterial and venous cerebral vasculature with the network was less than 90 seconds. The mean Dice similarity coefficient in the test set was 0.80 ± 0.04 (standard deviation) for the arteries and 0.88 ± 0.03 for the veins. The mean relative absolute volume difference was 7.3% ± 5.7 for the arteries and 8.5% ± 4.8 for the veins. Most of the segmentations (n = 273, 99.3%) were rated as very good to perfect.

Conclusion

The proposed convolutional neural network enables accurate artery and vein segmentation with 4D CT angiography with a processing time of less than 90 seconds.© RSNA, 2020.

Artificial intelligence in radiotherapy: a technological review

Radiation therapy (RT) is widely used to treat cancer. Technological advances in RT have occurred in the past 30 years. These advances, such as three-dimensional image guidance, intensity modulation, and robotics, created challenges and opportunities for the next breakthrough, in which artificial intelligence (AI) will possibly play important roles. AI will replace certain repetitive and labor-intensive tasks and improve the accuracy and consistency of others, particularly those with increased complexity because of technological advances. The improvement in efficiency and consistency is important to manage the increasing cancer patient burden to the society. Furthermore, AI may provide new functionalities that facilitate satisfactory RT. The functionalities include superior images for real-time intervention and adaptive and personalized RT. AI may effectively synthesize and analyze big data for such purposes. This review describes the RT workflow and identifies areas, including imaging, treatment planning, quality assurance, and outcome prediction, that benefit from AI. This review primarily focuses on deep-learning techniques, although conventional machine-learning techniques are also mentioned.

Three-Dimensional Neural Network to Automatically Assess Liver Tumor Burden Change on Consecutive Liver MRIs

BACKGROUND

Tumor response to therapy is often assessed by measuring change in liver lesion size between consecutive MRIs. However, these evaluations are both tedious and time-consuming for clinical radiologists.

PURPOSE

In this study, we sought to develop a convolutional neural network to detect liver metastases on MRI and applied this algorithm to assess change in tumor size on consecutive examinations.

METHODS

We annotated a data set of 64 patients with neuroendocrine tumors who underwent at least two consecutive liver MRIs with gadoxetic acid. We then developed a 3-D neural network using a U-Net architecture with ResNet-18 building blocks that first detected the liver and then lesions within the liver. Liver lesion labels for each examination were then matched in 3-D space using an iterative closest point algorithm followed by Kuhn-Munkres algorithm.

RESULTS

We developed a deep learning algorithm that detected liver metastases, co-registered the detected lesions, and then assessed the interval change in tumor burden between two multiparametric liver MRI examinations. Our deep learning algorithm was concordant in 91% with the radiologists' manual assessment about the interval change of disease burden. It had a sensitivity of 0.85 (95% confidence interval (95% CI): 0.77; 0.93) and specificity of 0.92 (95% CI: 0.87; 0.96) to classify liver segments as diseased or healthy. The mean DICE coefficient for individual lesions ranged between 0.73 and 0.81.

CONCLUSIONS

Our algorithm displayed high agreement with human readers for detecting change in liver lesions on MRI, offering evidence that artificial intelligence-based detectors may perform these tasks as part of routine clinical care in the future.

Triple U-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation

Nuclei segmentation is a vital step for pathological cancer research. It is still an open problem due to some difficulties, such as color inconsistency introduced by non-uniform manual operations, blurry tumor nucleus boundaries and overlapping tumor cells. In this paper, we aim to leverage the unique optical characteristic of H&E staining images that hematoxylin always stains cell nuclei blue, and eosin always stains the extracellular matrix and cytoplasm pink. Therefore, we extract the Hematoxylin component from RGB images by Beer-Lambert's Law. According to the optical attribute, the extracted Hematoxylin component is robust to color inconsistency. With the Hematoxylin component, we propose a Hematoxylin-aware CNN model for nuclei segmentation without the necessity of color normalization. Our proposed network is formulated as a Triple U-net structure which includes an RGB branch, a Hematoxylin branch and a Segmentation branch. Then we propose a novel feature aggregation strategy to allow the network to fuse features progressively and to learn better feature representations from different branches. Extensive experiments are performed to qualitatively and quantitatively evaluate the effectiveness of our proposed method. In the meanwhile, it outperforms state-of-the-art methods on three different nuclei segmentation datasets.

Computer-aided diagnosis systems for osteoporosis detection: a comprehensive survey

Computer-aided diagnosis (CAD) has revolutionized the field of medical diagnosis. They assist in improving the treatment potentials and intensify the survival frequency by early diagnosing the diseases in an efficient, timely, and cost-effective way. The automatic segmentation has led the radiologist to successfully segment the region of interest to improve the diagnosis of diseases from medical images which is not so efficiently possible by manual segmentation. The aim of this paper is to survey the vision-based CAD systems especially focusing on the segmentation techniques for the pathological bone disease known as osteoporosis. Osteoporosis is the state of the bones where the mineral density of bones decreases and they become porous, making the bones easily susceptible to fractures by small injury or a fall. The article covers the image acquisition techniques for acquiring the medical images for osteoporosis diagnosis. The article also discusses the advanced machine learning paradigms employed in segmentation for osteoporosis disease. Other image processing steps in osteoporosis like feature extraction and classification are also briefly described. Finally, the paper gives the future directions to improve the osteoporosis diagnosis and presents the proposed architecture. Graphical abstract.

Deep Learning-Based Detection of Pigment Signs for Analysis and Diagnosis of Retinitis Pigmentosa

Ophthalmological analysis plays a vital role in the diagnosis of various eye diseases, such as glaucoma, retinitis pigmentosa (RP), and diabetic and hypertensive retinopathy. RP is a genetic retinal disorder that leads to progressive vision degeneration and initially causes night blindness. Currently, the most commonly applied method for diagnosing retinal diseases is optical coherence tomography (OCT)-based disease analysis. In contrast, fundus imaging-based disease diagnosis is considered a low-cost diagnostic solution for retinal diseases. This study focuses on the detection of RP from the fundus image, which is a crucial task because of the low quality of fundus images and non-cooperative image acquisition conditions. Automatic detection of pigment signs in fundus images can help ophthalmologists and medical practitioners in diagnosing and analyzing RP disorders. To accurately segment pigment signs for diagnostic purposes, we present an automatic RP segmentation network (RPS-Net), which is a specifically designed deep learning-based semantic segmentation network to accurately detect and segment the pigment signs with fewer trainable parameters. Compared with the conventional deep learning methods, the proposed method applies a feature enhancement policy through multiple dense connections between the convolutional layers, which enables the network to discriminate between normal and diseased eyes, and accurately segment the diseased area from the background. Because pigment spots can be very small and consist of very few pixels, the RPS-Net provides fine segmentation, even in the case of degraded images, by importing high-frequency information from the preceding layers through concatenation inside and outside the encoder-decoder. To evaluate the proposed RPS-Net, experiments were performed based on 4-fold cross-validation using the publicly available Retinal Images for Pigment Signs (RIPS) dataset for detection and segmentation of retinal pigments. Experimental results show that RPS-Net achieved superior segmentation performance for RP diagnosis, compared with the state-of-the-art methods.

Mini Review: Deep Learning for Atrial Segmentation From Late Gadolinium-Enhanced MRIs

Segmentation and 3D reconstruction of the human atria is of crucial importance for precise diagnosis and treatment of atrial fibrillation, the most common cardiac arrhythmia. However, the current manual segmentation of the atria from medical images is a time-consuming, labor-intensive, and error-prone process. The recent emergence of artificial intelligence, particularly deep learning, provides an alternative solution to the traditional methods that fail to accurately segment atrial structures from clinical images. This has been illustrated during the recent 2018 Atrial Segmentation Challenge for which most of the challengers developed deep learning approaches for atrial segmentation, reaching high accuracy (>90% Dice score). However, as significant discrepancies exist between the approaches developed, many important questions remain unanswered, such as which deep learning architectures and methods to ensure reliability while achieving the best performance. In this paper, we conduct an in-depth review of the current state-of-the-art of deep learning approaches for atrial segmentation from late gadolinium-enhanced MRIs, and provide critical insights for overcoming the main hindrances faced in this task.

T-Net: Nested encoder-decoder architecture for the main vessel segmentation in coronary angiography

In this paper, we proposed nested encoder-decoder architecture named T-Net. T-Net consists of several small encoder-decoders for each block constituting convolutional network. T-Net overcomes the limitation that U-Net can only have a single set of the concatenate layer between encoder and decoder block. To be more precise, the U-Net symmetrically forms the concatenate layers, so the low-level feature of the encoder is connected to the latter part of the decoder, and the high-level feature is connected to the beginning of the decoder. T-Net arranges the pooling and up-sampling appropriately during the encoding process, and likewise during the decoding process so that feature-maps of various sizes are obtained in a single block. As a result, all features from the low-level to the high-level extracted from the encoder are delivered from the beginning of the decoder to predict a more accurate mask. We evaluated T-Net for the problem of segmenting three main vessels in coronary angiography images. The experiment consisted of a comparison of U-Net and T-Nets under the same conditions, and an optimized T-Net for the main vessel segmentation. As a result, T-Net recorded a Dice Similarity Coefficient score (DSC) of 83.77%, 10.69% higher than that of U-Net, and the optimized T-Net recorded a DSC of 88.97% which was 15.89% higher than that of U-Net. In addition, we visualized the weight activation of the convolutional layer of T-Net and U-Net to show that T-Net actually predicts the mask from earlier decoders. Therefore, we expect that T-Net can be effectively applied to other similar medical image segmentation problems.

Deep Color Transfer for Color-Plus-Mono Dual Cameras

A few approaches have studied image fusion using color-plus-mono dual cameras to improve the image quality in low-light shooting. Among them, the color transfer approach, which transfers the color information of a color image to a mono image, is considered to be promising for obtaining improved images with less noise and more detail. However, the color transfer algorithms rely heavily on appropriate color hints from a given color image. Unreliable color hints caused by errors in stereo matching of a color-plus-mono image pair can generate various visual artifacts in the final fused image. This study proposes a novel color transfer method that seeks reliable color hints from a color image and colorizes a corresponding mono image with reliable color hints that are based on a deep learning model. Specifically, a color-hint-based mask generation algorithm is developed to obtain reliable color hints. It removes unreliable color pixels using a reliability map computed by the binocular just-noticeable-difference model. In addition, a deep colorization network that utilizes structural information is proposed for solving the color bleeding artifact problem. The experimental results demonstrate that the proposed method provides better results than the existing image fusion algorithms for dual cameras.

Feasibility of Automated Volumetric Assessment of Large Hepatocellular Carcinomas' Responses to Transarterial Chemoembolization

Background: Hepatocellular carcinoma (HCC) is the most common liver malignancy and the leading cause of death in patients with cirrhosis. Various treatments for HCC are available, including transarterial chemoembolization (TACE), which is the commonest intervention performed in HCC. Radiologic tumor response following TACE is an important prognostic factor for patients with HCC. We hypothesized that, for large HCC tumors, assessment of treatment response made with automated volumetric response evaluation criteria in solid tumors (RECIST) might correlate with the assessment made with the more time- and labor-intensive unidimensional modified RECIST (mRECIST) and manual volumetric RECIST (M-vRECIST) criteria. Accordingly, we undertook this retrospective study to compare automated volumetric RECIST (A-vRECIST) with M-vRECIST and mRESIST for the assessment of large HCC tumors' responses to TACE. Methods:We selected 42 pairs of contrast-enhanced computed tomography (CT) images of large HCCs. Images were taken before and after TACE, and in each of the images, the HCC was segmented using both a manual contouring tool and a convolutional neural network. Three experienced radiologists assessed tumor response to TACE using mRECIST criteria. The intra-class correlation coefficient was used to assess inter-reader reliability in the mRECIST measurements, while the Pearson correlation coefficient was used to assess correlation between the volumetric and mRECIST measurements. Results:Volumetric tumor assessment using automated and manual segmentation tools showed good correlation with mRECIST measurements. For A-vRECIST and M-vRECIST, respectively, r = 0.597 vs. 0.622 in the baseline studies; 0.648 vs. 0.748 in the follow-up studies; and 0.774 vs. 0.766 in the response assessment (P < 0.001 for all). The A-vRECIST evaluation showed high correlation with the M-vRECIST evaluation (r = 0.967, 0.937, and 0.826 in baseline studies, follow-up studies, and response assessment, respectively, P < 0.001 for all). Conclusion:Volumetric RECIST measurements are likely to provide an early marker for TACE monitoring, and automated measurements made with a convolutional neural network may be good substitutes for manual volumetric measurements.

Modified U-Net (mU-Net) With Incorporation of Object-Dependent High Level Features for Improved Liver and Liver-Tumor Segmentation in CT Images

Segmentation of livers and liver tumors is one of the most important steps in radiation therapy of hepatocellular carcinoma. The segmentation task is often done manually, making it tedious, labor intensive, and subject to intra-/inter- operator variations. While various algorithms for delineating organ-at-risks (OARs) and tumor targets have been proposed, automatic segmentation of livers and liver tumors remains intractable due to their low tissue contrast with respect to the surrounding organs and their deformable shape in CT images. The U-Net has gained increasing popularity recently for image analysis tasks and has shown promising results. Conventional U-Net architectures, however, suffer from three major drawbacks. First, skip connections allow for the duplicated transfer of low resolution information in feature maps to improve efficiency in learning, but this often leads to blurring of extracted image features. Secondly, high level features extracted by the network often do not contain enough high resolution edge information of the input, leading to greater uncertainty where high resolution edge dominantly affects the network's decisions such as liver and liver-tumor segmentation. Thirdly, it is generally difficult to optimize the number of pooling operations in order to extract high level global features, since the number of pooling operations used depends on the object size. To cope with these problems, we added a residual path with deconvolution and activation operations to the skip connection of the U-Net to avoid duplication of low resolution information of features. In the case of small object inputs, features in the skip connection are not incorporated with features in the residual path. Furthermore, the proposed architecture has additional convolution layers in the skip connection in order to extract high level global features of small object inputs as well as high level features of high resolution edge information of large object inputs. Efficacy of the modified U-Net (mU-Net) was demonstrated using the public dataset of Liver tumor segmentation (LiTS) challenge 2017. For liver-tumor segmentation, Dice similarity coefficient (DSC) of 89.72 %, volume of error (VOE) of 21.93 %, and relative volume difference (RVD) of - 0.49 % were obtained. For liver segmentation, DSC of 98.51 %, VOE of 3.07 %, and RVD of 0.26 % were calculated. For the public 3D Image Reconstruction for Comparison of Algorithm Database (3Dircadb), DSCs were 96.01 % for the liver and 68.14 % for liver-tumor segmentations, respectively. The proposed mU-Net outperformed existing state-of-art networks.

3D-BoxSup: Positive-Unlabeled Learning of Brain Tumor Segmentation Networks From 3D Bounding Boxes

Accurate segmentation is an essential task when working with medical images. Recently, deep convolutional neural networks achieved a state-of-the-art performance for many segmentation benchmarks. Regardless of the network architecture, the deep learning-based segmentation methods view the segmentation problem as a supervised task that requires a relatively large number of annotated images. Acquiring a large number of annotated medical images is time consuming, and high-quality segmented images (i.e., strong labels) crafted by human experts are expensive. In this paper, we have proposed a method that achieves competitive accuracy from a "weakly annotated" image where the weak annotation is obtained via a 3D bounding box denoting an object of interest. Our method, called "3D-BoxSup," employs a positive-unlabeled learning framework to learn segmentation masks from 3D bounding boxes. Specially, we consider the pixels outside of the bounding box as positively labeled data and the pixels inside the bounding box as unlabeled data. Our method can suppress the negative effects of pixels residing between the true segmentation mask and the 3D bounding box and produce accurate segmentation masks. We applied our method to segment a brain tumor. The experimental results on the BraTS 2017 dataset (Menze et al., 2015; Bakas et al., 2017a,b,c) have demonstrated the effectiveness of our method.

Deep learning-based liver segmentation for fusion-guided intervention

PURPOSE

Tumors often have different imaging properties, and there is no single imaging modality that can visualize all tumors. In CT-guided needle placement procedures, image fusion (e.g. with MRI, PET, or contrast CT) is often used as image guidance when the tumor is not directly visible in CT. In order to achieve image fusion, interventional CT image needs to be registered to an imaging modality, in which the tumor is visible. However, multi-modality image registration is a very challenging problem. In this work, we develop a deep learning-based liver segmentation algorithm and use the segmented surfaces to assist image fusion with the applications in guided needle placement procedures for diagnosing and treating liver tumors.

METHODS

The developed segmentation method integrates multi-scale input and multi-scale output features in one single network for context information abstraction. The automatic segmentation results are used to register an interventional CT with a diagnostic image. The registration helps visualize the target and guide the interventional operation.

RESULTS

The segmentation results demonstrated that the developed segmentation method is highly accurate with Dice of 96.1% on 70 CT scans provided by LiTS challenge. The segmentation algorithm is then applied to a set of images acquired for liver tumor intervention for surface-based image fusion. The effectiveness of the proposed methods is demonstrated through a number of clinical cases.

CONCLUSION

Our study shows that deep learning-based image segmentation can obtain useful results to help image fusion for interventional guidance. Such a technique may lead to a number of other potential applications.

MIRD-Net for Medical Image Segmentation

Medical image segmentation is a fundamental and challenging problem for analyzing medical images due to the approximate pixel values of adjacent tissues in boundary and the non-linear feature between pixels. Although fully convolutional neural networks such as U-Net has demonstrated impressive performance on medical image segmentation, distinguishing subtle features between different categories after pooling layers is still a difficult task, which affects the segmentation accuracy. In this paper, we propose a Mini-Inception-Residual-Dense (MIRD) network named MIRD-Net to deal with this problem. The key point of our proposed MIRD-Net is MIRD Block. It takes advantage of Inception, Residual Block (RB) and Dense Block (DB), aiming to make the network obtain more features to help improve the segmentation accuracy. There is no pooling layer in MIRD-Net. Such a design avoids loss of information during forward propagation. Experimental results show that our framework significantly outperforms U-Net in six different image segmentation tasks and its parameters are only about 1/50 of U-Net.

Multi-Task Joint Learning Model for Segmenting and Classifying Tongue Images Using a Deep Neural Network

Automatic tongue image segmentation and tongue image classification are two crucial tongue characterization tasks in traditional Chinese medicine (TCM). Due to the complexity of tongue segmentation and fine-grained traits of tongue image classification, both tasks are challenging. Fortunately, from the perspective of computer vision, these two tasks are highly interrelated, making them compatible with the idea of Multi-Task Joint learning (MTL). By sharing the underlying parameters and adding two different task loss functions, an MTL method for segmenting and classifying tongue images is proposed in this paper. Moreover, two state-of-the-art deep neural network variants (UNET and Discriminative Filter Learning (DFL)) are fused into the MTL to perform these two tasks. To the best of our knowledge, our method is the first attempt to manage both tasks simultaneously with MTL. We conducted extensive experiments with the proposed method. The experimental results show that our joint method outperforms the existing tongue characterization methods. Besides, visualizations and ablation studies are provided to aid in understanding our approach, which suggest that our method is highly consistent with human perception.

Abdominal multi-organ auto-segmentation using 3D-patch-based deep convolutional neural network

Segmentation of normal organs is a critical and time-consuming process in radiotherapy. Auto-segmentation of abdominal organs has been made possible by the advent of the convolutional neural network. We utilized the U-Net, a 3D-patch-based convolutional neural network, and added graph-cut algorithm-based post-processing. The inputs were 3D-patch-based CT images consisting of 64 × 64 × 64 voxels designed to produce 3D multi-label semantic images representing the liver, stomach, duodenum, and right/left kidneys. The datasets for training, validating, and testing consisted of 80, 20, and 20 CT simulation scans, respectively. For accuracy assessment, the predicted structures were compared with those produced from the atlas-based method and inter-observer segmentation using the Dice similarity coefficient, Hausdorff distance, and mean surface distance. The efficiency was quantified by measuring the time elapsed for segmentation with or without automation using the U-Net. The U-Net-based auto-segmentation outperformed the atlas-based auto-segmentation in all abdominal structures, and showed comparable results to the inter-observer segmentations especially for liver and kidney. The average segmentation time without automation was 22.6 minutes, which was reduced to 7.1 minutes with automation using the U-Net. Our proposed auto-segmentation framework using the 3D-patch-based U-Net for abdominal multi-organs demonstrated potential clinical usefulness in terms of accuracy and time-efficiency.

IOUC-3DSFCNN: Segmentation of Brain Tumors via IOU Constraint 3D Symmetric Full Convolution Network with Multimodal Auto-context

Accurate segmentation of brain tumors from magnetic resonance (MR) images play a pivot role in assisting diagnoses, treatments and postoperative evaluations. However, due to its structural complexities, e.g., fuzzy tumor boundaries with irregular shapes, accurate 3D brain tumor delineation is challenging. In this paper, an intersection over union (IOU) constraint 3D symmetric full convolutional neural network (IOUC-3DSFCNN) model fused with multimodal auto-context is proposed for the 3D brain tumor segmentation. IOUC-3DSFCNN incorporates 3D residual groups into the classic 3DU-Net to further deepen the network structure to obtain more abstract voxel features under a five-layer cohesion architecture to ensure the model stability. The IOU constraint is used to address the issue of extremely unbalanced tumor foreground and background regions in MR images. In addition, to obtain more comprehensive and stable 3D brain tumor profiles, the multimodal auto-context information is fused into the IOUC-3DSFCNN model to achieve end-to-end 3D brain tumor profiles. Extensive confirmatory and comparative experiments conducted on the benchmark BRATS 2017 dataset demonstrate that the proposed segmentation model is superior to classic 3DU-Net-relevant and other state-of-the-art segmentation models, which can achieve accurate 3D tumor profiles on multimodal MRI volumes even with blurred tumor boundaries and big noise.

Deep Reinforcement Learning for Data Association in Cell Tracking

Accurate target detection and association are vital for the development of reliable target tracking, especially for cell tracking based on microscopy images due to the similarity of cells. We propose a deep reinforcement learning method to associate the detected targets between frames. According to the dynamic model of each target, the cost matrix is produced by conjointly considering various features of targets and then used as the input of a neural network. The proposed neural network is trained using reinforcement learning to predict a distribution over the association solution. Furthermore, we design a residual convolutional neural network that results in more efficient learning. We validate our method on two applications: the multiple target tracking simulation and the ISBI cell tracking. The results demonstrate that our approach based on reinforcement learning techniques could effectively track targets following different motion patterns and show competitive results.

Optimisation of 2D U-Net Model Components for Automatic Prostate Segmentation on MRI

In this paper, we develop an optimised state-of-the-art 2D U-Net model by studying the effects of the individual deep learning model components in performing prostate segmentation. We found that for upsampling, the combination of interpolation and convolution is better than the use of transposed convolution. For combining feature maps in each convolution block, it is only beneficial if a skip connection with concatenation is used. With respect to pooling, average pooling is better than strided-convolution, max, RMS or L2 pooling. Introducing a batch normalisation layer before the activation layer gives further performance improvement. The optimisation is based on a private dataset as it has a fixed 2D resolution and voxel size for every image which mitigates the need of a resizing operation in the data preparation process. Non-enhancing data preprocessing was applied and five-fold cross-validation was used to evaluate the fully automatic segmentation approach. We show it outperforms the traditional methods that were previously applied on the private dataset, as well as outperforming other comparable state-of-the-art 2D models on the public dataset PROMISE12.

Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation

The medical imaging literature has witnessed remarkable progress in high-performing segmentation models based on convolutional neural networks. Despite the new performance highs, the recent advanced segmentation models still require large, representative, and high quality annotated datasets. However, rarely do we have a perfect training dataset, particularly in the field of medical imaging, where data and annotations are both expensive to acquire. Recently, a large body of research has studied the problem of medical image segmentation with imperfect datasets, tackling two major dataset limitations: scarce annotations where only limited annotated data is available for training, and weak annotations where the training data has only sparse annotations, noisy annotations, or image-level annotations. In this article, we provide a detailed review of the solutions above, summarizing both the technical novelties and empirical results. We further compare the benefits and requirements of the surveyed methodologies and provide our recommended solutions. We hope this survey article increases the community awareness of the techniques that are available to handle imperfect medical image segmentation datasets.

Use of Spectral Detector Computed Tomography to Improve Liver Segmentation and Volumetry

INTRODUCTION

Liver segmentation and volumetry have traditionally been performed using computed tomography (CT) attenuation to discriminate liver from other tissues. In this project, we evaluated if spectral detector CT (SDCT) can improve liver segmentation over conventional CT on 2 segmentation methods.

MATERIALS AND METHODS

In this Health Insurance Portability and Accountability Act-compliant institutional review board-approved retrospective study, 30 contrast-enhanced SDCT scans with healthy livers were selected. The first segmentation method is based on Gaussian mixture models of the SDCT data. The second method is a convolutional neural network-based technique called U-Net. Both methods were compared against equivalent algorithms, which used conventional CT attenuation, with hand segmentation as the reference standard. Agreement to the reference standard was assessed using Dice similarity coefficient.

RESULTS

Dice similarity coefficients to the reference standard are 0.93 ± 0.02 for the Gaussian mixture model method and 0.90 ± 0.04 for the CNN-based method (all 2 methods applied on SDCT). These were significantly higher compared with equivalent algorithms applied on conventional CT, with Dice coefficients of 0.90 ± 0.06 (P = 0.007) and 0.86 ± 0.06 (P < 0.001), respectively.

CONCLUSION

On both liver segmentation methods tested, we demonstrated higher segmentation performance when the algorithms are applied on SDCT data compared with equivalent algorithms applied on conventional CT data.

Deep learning in fracture detection: a narrative review

Artificial intelligence (AI) is a general term that implies the use of a computer to model intelligent behavior with minimal human intervention. AI, particularly deep learning, has recently made substantial strides in perception tasks allowing machines to better represent and interpret complex data. Deep learning is a subset of AI represented by the combination of artificial neuron layers. In the last years, deep learning has gained great momentum. In the field of orthopaedics and traumatology, some studies have been done using deep learning to detect fractures in radiographs. Deep learning studies to detect and classify fractures on computed tomography (CT) scans are even more limited. In this narrative review, we provide a brief overview of deep learning technology: we (1) describe the ways in which deep learning until now has been applied to fracture detection on radiographs and CT examinations; (2) discuss what value deep learning offers to this field; and finally (3) comment on future directions of this technology.