Classification of Lung Nodule Malignancy Risk on Computed Tomography Images Using Convolutional Neural Network: A Comparison Between 2D and 3D Strategies

MVCNet: Multiview Contrastive Network for Unsupervised Representation Learning for 3-D CT Lesions

With the renaissance of deep learning, automatic diagnostic algorithms for computed tomography (CT) have achieved many successful applications. However, they heavily rely on lesion-level annotations, which are often scarce due to the high cost of collecting pathological labels. On the other hand, the annotated CT data, especially the 3-D spatial information, may be underutilized by approaches that model a 3-D lesion with its 2-D slices, although such approaches have been proven effective and computationally efficient. This study presents a multiview contrastive network (MVCNet), which enhances the representations of 2-D views contrastively against other views of different spatial orientations. Specifically, MVCNet views each 3-D lesion from different orientations to collect multiple 2-D views; it learns to minimize a contrastive loss so that the 2-D views of the same 3-D lesion are aggregated, whereas those of different lesions are separated. To alleviate the issue of false negative examples, the uninformative negative samples are filtered out, which results in more discriminative features for downstream tasks. By linear evaluation, MVCNet achieves state-of-the-art accuracies on the lung image database consortium and image database resource initiative (LIDC-IDRI) (88.62%), lung nodule database (LNDb) (76.69%), and TianChi (84.33%) datasets for unsupervised representation learning. When fine-tuned on 10% of the labeled data, the accuracies are comparable to the supervised learning models (89.46% versus 85.03%, 73.85% versus 73.44%, 83.56% versus 83.34% on the three datasets, respectively), indicating the superiority of MVCNet in learning representations with limited annotations. Our findings suggest that contrasting multiple 2-D views is an effective approach to capturing the original 3-D information, which notably improves the utilization of the scarce and valuable annotated CT data.

A novel image deep learning-based sub-centimeter pulmonary nodule management algorithm to expedite resection of the malignant and avoid over-diagnosis of the benign

OBJECTIVES

With the popularization of chest computed tomography (CT) screening, there are more sub-centimeter (≤ 1 cm) pulmonary nodules (SCPNs) requiring further diagnostic workup. This area represents an important opportunity to optimize the SCPN management algorithm avoiding "one-size fits all" approach. One critical problem is how to learn the discriminative multi-view characteristics and the unique context of each SCPN.

METHODS

Here, we propose a multi-view coupled self-attention module (MVCS) to capture the global spatial context of the CT image through modeling the association order of space and dimension. Compared with existing self-attention methods, MVCS uses less memory consumption and computational complexity, unearths dimension correlations that previous methods have not found, and is easy to integrate with other frameworks.

RESULTS

In total, a public dataset LUNA16 from LIDC-IDRI, 1319 SCPNs from 1069 patients presenting to a major referral center, and 160 SCPNs from 137 patients from three other major centers were analyzed to pre-train, train, and validate the model. Experimental results showed that performance outperforms the state-of-the-art models in terms of accuracy and stability and is comparable to that of human experts in classifying precancerous lesions and invasive adenocarcinoma. We also provide a fusion MVCS network (MVCSN) by combining the CT image with the clinical characteristics and radiographic features of patients.

CONCLUSION

This tool may ultimately aid in expediting resection of the malignant SCPNs and avoid over-diagnosis of the benign ones, resulting in improved management outcomes.

CLINICAL RELEVANCE STATEMENT

In the diagnosis of sub-centimeter lung adenocarcinoma, fusion MVCSN can help doctors improve work efficiency and guide their treatment decisions to a certain extent.

KEY POINTS

• Advances in computed tomography (CT) not only increase the number of nodules detected, but also the nodules that are identified are smaller, such as sub-centimeter pulmonary nodules (SCPNs). • We propose a multi-view coupled self-attention module (MVCS), which could model spatial and dimensional correlations sequentially for learning global spatial contexts, which is better than other attention mechanisms. • MVCS uses fewer huge memory consumption and computational complexity than the existing self-attention methods when dealing with 3D medical image data. Additionally, it reaches promising accuracy for SCPNs' malignancy evaluation and has lower training cost than other models.

DBPNDNet: dual-branch networks using 3DCNN toward pulmonary nodule detection

With the advancement of artificial intelligence, CNNs have been successfully introduced into the discipline of medical data analyzing. Clinically, automatic pulmonary nodules detection remains an intractable issue since those nodules existing in the lung parenchyma or on the chest wall are tough to be visually distinguished from shadows, background noises, blood vessels, and bones. Thus, when making medical diagnosis, clinical doctors need to first pay attention to the intensity cue and contour characteristic of pulmonary nodules, so as to locate the specific spatial locations of nodules. To automate the detection process, we propose an efficient architecture of multi-task and dual-branch 3D convolution neural networks, called DBPNDNet, for automatic pulmonary nodule detection and segmentation. Among the dual-branch structure, one branch is designed for candidate region extraction of pulmonary nodule detection, while the other incorporated branch is exploited for lesion region semantic segmentation of pulmonary nodules. In addition, we develop a 3D attention weighted feature fusion module according to the doctor's diagnosis perspective, so that the captured information obtained by the designed segmentation branch can further promote the effect of the adopted detection branch mutually. The experiment has been implemented and assessed on the commonly used dataset for medical image analysis to evaluate our designed framework. On average, our framework achieved a sensitivity of 91.33% false positives per CT scan and reached 97.14% sensitivity with 8 FPs per scan. The results of the experiments indicate that our framework outperforms other mainstream approaches.

A novel fusion algorithm for benign-malignant lung nodule classification on CT images

The accurate recognition of malignant lung nodules on CT images is critical in lung cancer screening, which can offer patients the best chance of cure and significant reductions in mortality from lung cancer. Convolutional Neural Network (CNN) has been proven as a powerful method in medical image analysis. Radiomics which is believed to be of interest based on expert opinion can describe high-throughput extraction from CT images. Graph Convolutional Network explores the global context and makes the inference on both graph node features and relational structures. In this paper, we propose a novel fusion algorithm, RGD, for benign-malignant lung nodule classification by incorporating Radiomics study and Graph learning into the multiple Deep CNNs to form a more complete and distinctive feature representation, and ensemble the predictions for robust decision-making. The proposed method was conducted on the publicly available LIDC-IDRI dataset in a 10-fold cross-validation experiment and it obtained an average accuracy of 93.25%, a sensitivity of 89.22%, a specificity of 95.82%, precision of 92.46%, F1 Score of 0.9114 and AUC of 0.9629. Experimental results illustrate that the RGD model achieves superior performance compared with the state-of-the-art methods. Moreover, the effectiveness of the fusion strategy has been confirmed by extensive ablation studies. In the future, the proposed model which performs well on the pulmonary nodule classification on CT images will be applied to increase confidence in the clinical diagnosis of lung cancer.

An Improved Convolution Neural Network and Modified Regularized K-Means-Based Automatic Lung Nodule Detection and Classification

If lung cancer is not detected in its initial phases, it can be fatal. However, because of the quantity and structure of its nodules, lung cancer is difficult to detect early. For accurate detections, radiologists require assistance from automated tools. Numerous expert methods have been created over time to assist radiologists in the diagnosis of lung cancer. However, this requires accurate research. Therefore, in this article, we propose a framework to precisely detect lung cancer by categorizing it between benign and malignant nodules. To achieve this objective, an efficient deep-learning algorithm is presented. The presented technique consists of four stages, namely pre-processing, segmentation, classification, and severity stage analysis. Initially, the collected image is given to the pre-processing stage to eliminate the distortion present in the image. Then, the noise-free image is given to the segmentation stage. For segmentation, in this paper, modified regularized K-means (MRKM) clustering algorithm is presented. After the segmentation process, the segmented nodule image is fed to the classification stage to categorize the nodule as benign or malignant (risk nodule). For classification, an improved convolution neural network (ICNN) is presented. The proposed ICNN is designed by modifying CNN with the integration of the adaptive tree seed optimization (ATSO) algorithm. Finally, the stage identification is carried out based on the size of the nodule and we classify the malignant nodule as S1-S4. The presented technique attained the maximum accuracy of 96.5% and performance compared with existing state-of-art methods.

BIDL: a brain-inspired deep learning framework for spatiotemporal processing

Brain-inspired deep spiking neural network (DSNN) which emulates the function of the biological brain provides an effective approach for event-stream spatiotemporal perception (STP), especially for dynamic vision sensor (DVS) signals. However, there is a lack of generalized learning frameworks that can handle various spatiotemporal modalities beyond event-stream, such as video clips and 3D imaging data. To provide a unified design flow for generalized spatiotemporal processing (STP) and to investigate the capability of lightweight STP processing via brain-inspired neural dynamics, this study introduces a training platform called brain-inspired deep learning (BIDL). This framework constructs deep neural networks, which leverage neural dynamics for processing temporal information and ensures high-accuracy spatial processing via artificial neural network layers. We conducted experiments involving various types of data, including video information processing, DVS information processing, 3D medical imaging classification, and natural language processing. These experiments demonstrate the efficiency of the proposed method. Moreover, as a research framework for researchers in the fields of neuroscience and machine learning, BIDL facilitates the exploration of different neural models and enables global-local co-learning. For easily fitting to neuromorphic chips and GPUs, the framework incorporates several optimizations, including iteration representation, state-aware computational graph, and built-in neural functions. This study presents a user-friendly and efficient DSNN builder for lightweight STP applications and has the potential to drive future advancements in bio-inspired research.

Automatic classification of pulmonary nodules in computed tomography images using pre-trained networks and bag of features

Lung cancer has the highest incidence in the world. The standard tests for its diagnostics are medical imaging exams, sputum cytology, and lung biopsy. Computed Tomography (CT) of the chest plays an essential role in the early detection of nodules since it can allow for more treatment options and increases patient survival. However, the analysis of these exams is a tiring and error-prone process. Thus, computational methods can help the specialist in this analysis. This work addresses the classification of pulmonary nodules as benign or malignant on CT images. Our approach uses the pre-trained VGG16, VGG19, Inception, Resnet50, and Xception, to extract features from each 2D slice of the 3D nodules. Then, we use Principal Component Analysis to reduce the dimensionality of the feature vectors and make them all the same length. Then, we use Bag of Features (BoF) to combine the feature vectors of the different 2D slices and generate only one signature representing the 3D nodule. The classification step uses Random Forest. We evaluated the proposed method with 1,405 segmented nodules from the LIDC-IDRI database and obtained an accuracy of 95.34%, F1-Score of 91.73, kappa of 0.88, sensitivity of 90.53%, specificity of 97.26% and AUC of 0.99. The main conclusion was that the combination by BoF of features extracted from 2D slices using pre-trained architectures produced better results than training 2D and 3D CNNs in the nodules. In addition, the use of BoF also makes the creation of the nodule signature independent of the number of slices.

Pulmonary nodule detection based on IR-UNet + +

Lung cancer is one of the cancers with the highest incidence rate and death rate worldwide. An initial lesion of the lung appears as nodules in the lungs on CT images, and early and timely diagnosis can greatly improve the survival rate. Automatic detection of lung nodules can greatly improve work efficiency and accuracy rate. However, owing to the three-dimensional complex structure of lung CT data and the variation in shapes and appearances of lung nodules, high-precision detection of pulmonary nodules remains challenging. To address the problem, a new 3D framework IR-UNet +  + is proposed for automatic pulmonary nodule detection in this paper. First, the Inception Net and ResNet are combined as the building blocks. Second, the squeeze-and-excitation structure is introduced into building blocks for better feature extraction. Finally, two short skip pathways are redesigned based on the U-shaped network. To verify the effectiveness of our algorithm, systematic experiments are conducted on the LUNA16 dataset. Experimental results show that the proposed network performs better than several existing lung nodule detection methods with the sensitivity of 1 FP/scan, 4 FPs/scan, and 8 FPs/scan being 90.13%, 94.77%, and 95.78%, respectively. Therefore, it comes to the conclusion that our proposed model has achieved superior performance for lung nodule detection.

Self-Supervised Transfer Learning Based on Domain Adaptation for Benign-Malignant Lung Nodule Classification on Thoracic CT

The spatial heterogeneity is an important indicator of the malignancy of lung nodules in lung cancer diagnosis. Compared with 2D nodule CT images, the 3D volumes with entire nodule objects hold richer discriminative information. However, for deep learning methods driven by massive data, effectively capturing the 3D discriminative features of nodules in limited labeled samples is a challenging task. Different from previous models that proposed transfer learning models in a 2D pattern or learning from scratch 3D models, we develop a self-supervised transfer learning based on domain adaptation (SSTL-DA) 3D CNN framework for benign-malignant lung nodule classification. At first, a data pre-processing strategy termed adaptive slice selection (ASS) is developed to eliminate the redundant noise of the input samples with lung nodules. Then, the self-supervised learning network is constructed to learn robust image representation from CT images. Finally, a transfer learning method based on domain adaptation is designed to obtain discriminant features for classification. The proposed SSTL-DA method has been assessed on the LIDC-IDRI benchmark dataset, and it obtains an accuracy of 91.07% and an AUC of 95.84%. These results demonstrate that the SSTL-DA model achieves quite a competitive classification performance compared with some state-of-the-art approaches.

LungSeek: 3D Selective Kernel residual network for pulmonary nodule diagnosis

Early detection and diagnosis of pulmonary nodules is the most promising way to improve the survival chances of lung cancer patients. This paper proposes an automatic pulmonary cancer diagnosis system, LungSeek. LungSeek is mainly divided into two modules: (1) Nodule detection, which detects all suspicious nodules from computed tomography (CT) scan; (2) Nodule Classification, classifies nodules as benign or malignant. Specifically, a 3D Selective Kernel residual network (SK-ResNet) based on the Selective Kernel Network and 3D residual network is located. A deep 3D region proposal network with SK-ResNet is designed for detection of pulmonary nodules while a multi-scale feature fusion network is designed for the nodule classification. Both networks use the SK-Net module to obtain different receptive field information, thereby effectively learning nodule features and improving diagnostic performance. Our method has been verified on the luna16 data set, reaching 89.06, 94.53% and 97.72% when the average number of false positives is 1, 2 and 4, respectively. Meanwhile, its performance is better than the state-of-the-art method and other similar networks and experienced doctors. This method has the ability to adaptively adjust the receptive field according to multiple scales of the input information, so as to better detect nodules of various sizes. The framework of LungSeek based on 3D SK-ResNet is proposed for nodule detection and nodule classification from chest CT. Our experimental results demonstrate the effectiveness of the proposed method in the diagnosis of pulmonary nodules.

Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data

INTRODUCTION

Hepatocellular carcinoma is the prevalent primary liver cancer, a silent disease that killed 782,000 worldwide in 2018. Multimodal deep learning is the application of deep learning techniques, fusing more than one data modality as the model's input.

PURPOSE

A computer-aided diagnosis system for hepatocellular carcinoma developed with multimodal deep learning approaches could use multiple data modalities as recommended by clinical guidelines, and enhance the robustness and the value of the second-opinion given to physicians. This article describes the process of creation and evaluation of an algorithm for computer-aided diagnosis of hepatocellular carcinoma developed with multimodal deep learning techniques fusing preprocessed computed-tomography images with structured data from patient Electronic Health Records.

RESULTS

The classification performance achieved by the proposed algorithm in the test dataset was: accuracy = 86.9%, precision = 89.6%, recall = 86.9% and F-Score = 86.7%. These classification performance metrics are closer to the state-of-the-art in this area and were achieved with data modalities which are cheaper than traditional Magnetic Resonance Imaging approaches, enabling the use of the proposed algorithm by low and mid-sized healthcare institutions.

CONCLUSION

The classification performance achieved with the multimodal deep learning algorithm is higher than human specialists diagnostic performance using only CT for diagnosis. Even though the results are promising, the multimodal deep learning architecture used for hepatocellular carcinoma prediction needs more training and test processes using different datasets before the use of the proposed algorithm by physicians in real healthcare routines. The additional training aims to confirm the classification performance achieved and enhance the model's robustness.

Design of lung nodules segmentation and recognition algorithm based on deep learning

BACKGROUND

Accurate segmentation and recognition algorithm of lung nodules has great important value of reference for early diagnosis of lung cancer. An algorithm is proposed for 3D CT sequence images in this paper based on 3D Res U-Net segmentation network and 3D ResNet50 classification network. The common convolutional layers in encoding and decoding paths of U-Net are replaced by residual units while the loss function is changed to Dice loss after using cross entropy loss to accelerate network convergence. Since the lung nodules are small and rich in 3D information, the ResNet50 is improved by replacing the 2D convolutional layers with 3D convolutional layers and reducing the sizes of some convolution kernels, 3D ResNet50 network is obtained for the diagnosis of benign and malignant lung nodules.

RESULTS

3D Res U-Net was trained and tested on 1044 CT subcases in the LIDC-IDRI database. The segmentation result shows that the Dice coefficient of 3D Res U-Net is above 0.8 for the segmentation of lung nodules larger than 10 mm in diameter. 3D ResNet50 was trained and tested on 2960 lung nodules in the LIDC-IDRI database. The classification result shows that the diagnostic accuracy of 3D ResNet50 is 87.3% and AUC is 0.907.

CONCLUSION

The 3D Res U-Net module improves segmentation performance significantly with the comparison of 3D U-Net model based on residual learning mechanism. 3D Res U-Net can identify small nodules more effectively and improve its segmentation accuracy for large nodules. Compared with the original network, the classification performance of 3D ResNet50 is significantly improved, especially for small benign nodules.

3D gray density coding feature for benign-malignant pulmonary nodule classification on chest CT

PURPOSE

Early detection is significant to reduce lung cancer-related death. Computer-aided detection system (CADs) can help radiologists to make an early diagnosis. In this paper, we propose a novel 3D gray density coding feature (3D GDC) and fuse it with extracted geometric features. The fusion feature and random forest are used for benign-malignant pulmonary nodule classification on Chest CT.

METHODS

First, a dictionary model is created to acquire codebook. It is used to obtain feature descriptors and includes 3D block database (BD) and distance matrix clustering centers. 3D BD is balanced and randomly selecting from benign and malignant pulmonary nodules of training data. Clustering centers is got by clustering the distance matrix, which is the distance between every two blocks in 3D BD. Then, feature descriptor is obtained by coding the pulmonary nodule with codebook, and 3D GDC feature is the result of histogram statistics on feature descriptor. Second, geometric features are extracted for fusion feature. Finally, random forest is performed for benign-malignant pulmonary nodule classification with fusion feature of the 3D gray density coding feature and the geometric features.

RESULTS

We verify the effectiveness of our method on the public LIDC-IDRI dataset and the private ZSHD dataset. For LIDC-IDRI dataset, compared with other state-of-the-art methods, we achieve more satisfactory results with 93.17 ± 1.94% for accuracy and 97.53 ± 1.62% for AUC. As for private ZSHD dataset, it contains a total of 238 lung nodules from 203 patients. The accuracy and AUC achieved by our method are 90.0% and 93.15%.

CONCLUSIONS

The results show that our method can provide doctors with more accurate results of benign-malignant pulmonary nodule classification for auxiliary diagnosis, and our method is more interpretable than 3D CNN methods, which can provide doctors with more auxiliary information.

Lung Nodule Detection from Feature Engineering to Deep Learning in Thoracic CT Images: a Comprehensive Review

This paper presents a systematic review of the literature focused on the lung nodule detection in chest computed tomography (CT) images. Manual detection of lung nodules by the radiologist is a sequential and time-consuming process. The detection is subjective and depends on the radiologist's experiences. Owing to the variation in shapes and appearances of a lung nodule, it is very difficult to identify the proper location of the nodule from a huge number of slices generated by the CT scanner. Small nodules (< 10 mm in diameter) may be missed by this manual detection process. Therefore, computer-aided diagnosis (CAD) system acts as a "second opinion" for the radiologists, by making final decision quickly with higher accuracy and greater confidence. The goal of this survey work is to present the current state of the artworks and their progress towards lung nodule detection to the researchers and readers in this domain. This review paper has covered the published works from 2009 to April 2018. Different nodule detection approaches are described elaborately in this work. Recently, it is observed that deep learning (DL)-based approaches are applied extensively for nodule detection and characterization. Therefore, emphasis has been given to convolutional neural network (CNN)-based DL approaches by describing different CNN-based networks.

Pre-Training Autoencoder for Lung Nodule Malignancy Assessment Using CT Images

Lung cancer late diagnosis has a large impact on the mortality rate numbers, leading to a very low five-year survival rate of 5%. This issue emphasises the importance of developing systems to support a diagnostic at earlier stages. Clinicians use Computed Tomography (CT) scans to assess the nodules and the likelihood of malignancy. Automatic solutions can help to make a faster and more accurate diagnosis, which is crucial for the early detection of lung cancer. Convolutional neural networks (CNN) based approaches have shown to provide a reliable feature extraction ability to detect the malignancy risk associated with pulmonary nodules. This type of approach requires a massive amount of data to model training, which usually represents a limitation in the biomedical field due to medical data privacy and security issues. Transfer learning (TL) methods have been widely explored in medical imaging applications, offering a solution to overcome problems related to the lack of training data publicly available. For the clinical annotations experts with a deep understanding of the complex physiological phenomena represented in the data are required, which represents a huge investment. In this direction, this work explored a TL method based on unsupervised learning achieved when training a Convolutional Autoencoder (CAE) using images in the same domain. For this, lung nodules from the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) were extracted and used to train a CAE. Then, the encoder part was transferred, and the malignancy risk was assessed in a binary classification—benign and malignant lung nodules, achieving an Area Under the Curve (AUC) value of 0.936. To evaluate the reliability of this TL approach, the same architecture was trained from scratch and achieved an AUC value of 0.928. The results reported in this comparison suggested that the feature learning achieved when reconstructing the input with an encoder-decoder based architecture can be considered an useful knowledge that might allow overcoming labelling constraints.