Convolutional neural networks: an overview and application in radiology

Deep learning workflow in radiology: a primer

Interest for deep learning in radiology has increased tremendously in the past decade due to the high achievable performance for various computer vision tasks such as detection, segmentation, classification, monitoring, and prediction. This article provides step-by-step practical guidance for conducting a project that involves deep learning in radiology, from defining specifications, to deployment and scaling. Specifically, the objectives of this article are to provide an overview of clinical use cases of deep learning, describe the composition of multi-disciplinary team, and summarize current approaches to patient, data, model, and hardware selection. Key ideas will be illustrated by examples from a prototypical project on imaging of colorectal liver metastasis. This article illustrates the workflow for liver lesion detection, segmentation, classification, monitoring, and prediction of tumor recurrence and patient survival. Challenges are discussed, including ethical considerations, cohorting, data collection, anonymization, and availability of expert annotations. The practical guidance may be adapted to any project that requires automated medical image analysis.

CNN-based survival model for pancreatic ductal adenocarcinoma in medical imaging

Background

Cox proportional hazard model (CPH) is commonly used in clinical research for survival analysis. In quantitative medical imaging (radiomics) studies, CPH plays an important role in feature reduction and modeling. However, the underlying linear assumption of CPH model limits the prognostic performance. In this work, using transfer learning, a convolutional neural network (CNN) based survival model was built and tested on preoperative CT images of resectable Pancreatic Ductal Adenocarcinoma (PDAC) patients.

Results

The proposed CNN-based survival model outperformed the traditional CPH-based radiomics approach in terms of concordance index and index of prediction accuracy, providing a better fit for patients’ survival patterns.

Conclusions

The proposed CNN-based survival model outperforms CPH-based radiomics pipeline in PDAC prognosis. This approach offers a better fit for survival patterns based on CT images and overcomes the limitations of conventional survival models.

A Deep-Learning Approach for Diagnosis of Metastatic Breast Cancer in Bones from Whole-Body Scans

(1) Background: Bone metastasis is one of the most frequent diseases in breast, lung and prostate cancer; bone scintigraphy is the primary imaging method of screening that offers the highest sensitivity (95%) regarding metastases. To address the considerable problem of bone metastasis diagnosis, focused on breast cancer patients, artificial intelligence methods devoted to deep-learning algorithms for medical image analysis are investigated in this research work; (2) Methods: Deep learning is a powerful algorithm for automatic classification and diagnosis of medical images whereas its implementation is achieved by the use of convolutional neural networks (CNNs). The purpose of this study is to build a robust CNN model that will be able to classify images of whole-body scans in patients suffering from breast cancer, depending on whether or not they are infected by metastasis of breast cancer; (3) Results: A robust CNN architecture is selected based on CNN exploration performance for bone metastasis diagnosis using whole-body scan images, achieving a high classification accuracy of 92.50%. The best-performing CNN method is compared with other popular and well-known CNN architectures for medical imaging like ResNet50, VGG16, MobileNet, and DenseNet, reported in the literature, providing superior classification accuracy; and (4) Conclusions: Prediction results show the efficacy of the proposed deep learning approach in bone metastasis diagnosis for breast cancer patients in nuclear medicine.

Current applications and future directions of deep learning in musculoskeletal radiology

Deep learning with convolutional neural networks (CNN) is a rapidly advancing subset of artificial intelligence that is ideally suited to solving image-based problems. There are an increasing number of musculoskeletal applications of deep learning, which can be conceptually divided into the categories of lesion detection, classification, segmentation, and non-interpretive tasks. Numerous examples of deep learning achieving expert-level performance in specific tasks in all four categories have been demonstrated in the past few years, although comprehensive interpretation of imaging examinations has not yet been achieved. It is important for the practicing musculoskeletal radiologist to understand the current scope of deep learning as it relates to musculoskeletal radiology. Interest in deep learning from researchers, radiology leadership, and industry continues to increase, and it is likely that these developments will impact the daily practice of musculoskeletal radiology in the near future.

Beyond the Randomized Clinical Trial: Innovative Data Science to Close the Pediatric Evidence Gap

Despite the application of advanced statistical and pharmacometric approaches to pediatric trial data, a large pediatric evidence gap still remains. Here, we discuss how to collect more data from children by using real-world data from electronic health records, mobile applications, wearables, and social media. The large datasets collected with these approaches enable and may demand the use of artificial intelligence and machine learning to allow the data to be analyzed for decision making. Applications of this approach are presented, which include the prediction of future clinical complications, medical image analysis, identification of new pediatric end points and biomarkers, the prediction of treatment nonresponders, and the prediction of placebo-responders for trial enrichment. Finally, we discuss how to bring machine learning from science to pediatric clinical practice. We conclude that advantage should be taken of the current opportunities offered by innovations in data science and machine learning to close the pediatric evidence gap.

Deep convolutional neural network applied to the liver imaging reporting and data system (LI-RADS) version 2014 category classification: a pilot study

PURPOSE

To develop a deep convolutional neural network (CNN) model to categorize multiphase CT and MRI liver observations using the liver imaging reporting and data system (LI-RADS) (version 2014).

METHODS

A pre-existing dataset comprising 314 hepatic observations (163 CT, 151 MRI) with corresponding diameters and LI-RADS categories (LR-1-5) assigned in consensus by two LI-RADS steering committee members was used to develop two CNNs: pre-trained network with an input of triple-phase images (training with transfer learning) and custom-made network with an input of quadruple-phase images (training from scratch). The dataset was randomly split into training, validation, and internal test sets (70:15:15 split). The overall accuracy and area under receiver operating characteristic curve (AUROC) were assessed for categorizing LR-1/2, LR-3, LR-4, and LR-5. External validation was performed for the model with the better performance on the internal test set using two external datasets (EXT-CT and EXT-MR: 68 and 44 observations, respectively).

RESULTS

The transfer learning model outperformed the custom-made model: overall accuracy of 60.4% and AUROCs of 0.85, 0.90, 0.63, 0.82 for LR-1/2, LR-3, LR-4, LR-5, respectively. On EXT-CT, the model had an overall accuracy of 41.2% and AUROCs of 0.70, 0.66, 0.60, 0.76 for LR-1/2, LR-3, LR-4, LR-5, respectively. On EXT-MR, the model had an overall accuracy of 47.7% and AUROCs of 0.88, 0.74, 0.69, 0.79 for LR-1/2, LR-3, LR-4, LR-5, respectively.

CONCLUSION

Our study shows the feasibility of CNN for assigning LI-RADS categories from a relatively small dataset but highlights the challenges of model development and validation.

Deep learning-based CAD schemes for the detection and classification of lung nodules from CT images: A survey

BACKGROUND

Lung cancer is the most common cancer in the world. Computed tomography (CT) is the standard medical imaging modality for early lung nodule detection and diagnosis that improves patient's survival rate. Recently, deep learning algorithms, especially convolutional neural networks (CNNs), have become a preferred methodology for developing computer-aided detection and diagnosis (CAD) schemes of lung CT images.

OBJECTIVE

Several CNN-based research projects have been initiated to design robust and efficient CAD schemes for the detection and classification of lung nodules. This paper reviews the recent works in this area and gives an insight into technical progress.

METHODS

First, a brief overview of CNN models and their basic structures is presented in this investigation. Then, we provide an analytic comparison of the existing approaches to discover recent trend and upcoming challenges. We also introduce an objective description of both handcrafted and deep learning features, as well as the types of nodules, the medical imaging modalities, the widely used databases, and related works in the last three years. The articles presented in this work were selected from various databases. About 57% of reviewed articles published in the last year.

RESULTS

Our analysis reveals that several methods achieved promising performance with high sensitivity rates ranging from 66% to 100% under the false-positive rates ranging from 1 to 15 per CT scan. It can be noted that CNN models have contributed to the accurate detection and early diagnosis of lung nodules.

CONCLUSIONS

From the critical discussion and an outline for prospective directions, this survey provide researchers valuable information to master the deep learning concepts and to deepen their knowledge of the trend and latest techniques in developing CAD schemes of lung CT images.

Deep learning-based detection of seedling development

BACKGROUND

Monitoring the timing of seedling emergence and early development via high-throughput phenotyping with computer vision is a challenging topic of high interest in plant science. While most studies focus on the measurements of leaf area index or detection of specific events such as emergence, little attention has been put on the identification of kinetics of events of early seedling development on a seed to seed basis.

RESULT

Imaging systems screened the whole seedling growth process from the top view. Precise annotation of emergence out of the soil, cotyledon opening, and appearance of first leaf was conducted. This annotated data set served to train deep neural networks. Various strategies to incorporate in neural networks, the prior knowledge of the order of the developmental stages were investigated. Best results were obtained with a deep neural network followed with a long short term memory cell, which achieves more than 90% accuracy of correct detection.

CONCLUSION

This work provides a full pipeline of image processing and machine learning to classify three stages of plant growth plus soil on the different accessions of two species of red clover and alfalfa but which could easily be extended to other crops and other stages of development.

A Convolutional Neural Network for Lentigo Diagnosis

Using Reflectance Confocal Microscopy (RCM) for lentigo diagnosis is today considered essential. Indeed, RCM allows fast data acquisition with a high spatial resolution of the skin. In this paper, we use a deep convolutional neural network (CNN) to perform RCM image classification in order to detect lentigo. The proposed method relies on an InceptionV3 architecture combined with data augmentation and transfer learning. The method is validated on RCM data and shows very efficient detection performance with more than 98% of accuracy.

Data Augmentation Techniques for Deep Learning-Based Medical Image Analyses

영상처리 기반으로 의료영상을 분석하는 기법은 정상 환자와 비정상 환자를 분류, 병변 검출 및 장기나 병변의 분할 등에 사용되고 있다. 최근 인공지능 기술의 비약적 발전으로 의료영상 분석 연구들이 딥러닝 기술을 활용하여 시도되고 있다. 의료영상은 학습에 필요한 데이터를 충분히 모으기 어렵고 클래스별 데이터 수의 차이 때문에, 딥러닝 모델의 성능을 올리는데 어려움이 있다. 이러한 문제를 해결하기 위해 다양한 연구가 시도되고 있으며, 이 중 하나가 학습 데이터를 증강하는 것이다. 본 종설에서는 회전, 역상, 밝기 변화 등과 같은 영상처리 기반의 데이터 증강, 적대적생성네트워크를 활용한 데이터 증강, 그리고 기존 영상의 속성들을 섞는 등의 최신 데이터 증강 기법을 알아보고, 의료영상 연구에 적용된 사례들과 그 결과를 조사해 보고자 한다. 끝으로 데이터 증강의 필요성을 고찰하고 앞으로의 방향을 짚어본다.

Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers

Objective

To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images.

Materials and Methods

PubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journals.

Results

Of 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals.

Conclusion

Nearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms.

An artificial intelligence algorithm that differentiates anterior ethmoidal artery location on sinus computed tomography scans

OBJECTIVE

Deep learning using convolutional neural networks represents a form of artificial intelligence where computers recognise patterns and make predictions based upon provided datasets. This study aimed to determine if a convolutional neural network could be trained to differentiate the location of the anterior ethmoidal artery as either adhered to the skull base or within a bone 'mesentery' on sinus computed tomography scans.

METHODS

Coronal sinus computed tomography scans were reviewed by two otolaryngology residents for anterior ethmoidal artery location and used as data for the Google Inception-V3 convolutional neural network base. The classification layer of Inception-V3 was retrained in Python (programming language software) using a transfer learning method to interpret the computed tomography images.

RESULTS

A total of 675 images from 388 patients were used to train the convolutional neural network. A further 197 unique images were used to test the algorithm; this yielded a total accuracy of 82.7 per cent (95 per cent confidence interval = 77.7-87.8), kappa statistic of 0.62 and area under the curve of 0.86.

CONCLUSION

Convolutional neural networks demonstrate promise in identifying clinically important structures in functional endoscopic sinus surgery, such as anterior ethmoidal artery location on pre-operative sinus computed tomography.

Deep learning with convolutional neural networks for identification of liver masses and hepatocellular carcinoma: A systematic review

BACKGROUND

Artificial intelligence, such as convolutional neural networks (CNNs), has been used in the interpretation of images and the diagnosis of hepatocellular cancer (HCC) and liver masses. CNN, a machine-learning algorithm similar to deep learning, has demonstrated its capability to recognise specific features that can detect pathological lesions.

AIM

To assess the use of CNNs in examining HCC and liver masses images in the diagnosis of cancer and evaluating the accuracy level of CNNs and their performance.

METHODS

The databases PubMed, EMBASE, and the Web of Science and research books were systematically searched using related keywords. Studies analysing pathological anatomy, cellular, and radiological images on HCC or liver masses using CNNs were identified according to the study protocol to detect cancer, differentiating cancer from other lesions, or staging the lesion. The data were extracted as per a predefined extraction. The accuracy level and performance of the CNNs in detecting cancer or early stages of cancer were analysed. The primary outcomes of the study were analysing the type of cancer or liver mass and identifying the type of images that showed optimum accuracy in cancer detection.

RESULTS

A total of 11 studies that met the selection criteria and were consistent with the aims of the study were identified. The studies demonstrated the ability to differentiate liver masses or differentiate HCC from other lesions (n = 6), HCC from cirrhosis or development of new tumours (n = 3), and HCC nuclei grading or segmentation (n = 2). The CNNs showed satisfactory levels of accuracy. The studies aimed at detecting lesions (n = 4), classification (n = 5), and segmentation (n = 2). Several methods were used to assess the accuracy of CNN models used.

CONCLUSION

The role of CNNs in analysing images and as tools in early detection of HCC or liver masses has been demonstrated in these studies. While a few limitations have been identified in these studies, overall there was an optimal level of accuracy of the CNNs used in segmentation and classification of liver cancers images.

Deep learning and taphonomy: high accuracy in the classification of cut marks made on fleshed and defleshed bones using convolutional neural networks

Accurate identification of bone surface modifications (BSM) is crucial for the taphonomic understanding of archaeological and paleontological sites. Critical interpretations of when humans started eating meat and animal fat or when they started using stone tools, or when they occupied new continents or interacted with predatory guilds impinge on accurate identifications of BSM. Until now, interpretations of Plio-Pleistocene BSM have been contentious because of the high uncertainty in discriminating among taphonomic agents. Recently, the use of machine learning algorithms has yielded high accuracy in the identification of BSM. A branch of machine learning methods based on imaging, computer vision (CV), has opened the door to a more objective and accurate method of BSM identification. The present work has selected two extremely similar types of BSM (cut marks made on fleshed an defleshed bones) to test the immense potential of artificial intelligence methods. This CV approach not only produced the highest accuracy in the classification of these types of BSM until present (95% on complete images of BSM and 88.89% of images of only internal mark features), but it also has enabled a method for determining which inconspicuous microscopic features determine successful BSM discrimination. The potential of this method in other areas of taphonomy and paleobiology is enormous.

Convolutional Neural Network Detection of Axillary Lymph Node Metastasis Using Standard Clinical Breast MRI

BACKGROUND

Axillary lymph node status is important for breast cancer staging and treatment planning as the majority of breast cancer metastasis spreads through the axillary lymph nodes. There is currently no reliable noninvasive imaging method to detect nodal metastasis associated with breast cancer.

MATERIALS AND METHODS

Magnetic resonance imaging (MRI) data were those from the peak contrast dynamic image from 1.5 Tesla MRI scanners at the pre-neoadjuvant chemotherapy stage. Data consisted of 66 abnormal nodes from 38 patients and 193 normal nodes from 61 patients. Abnormal nodes were those determined by expert radiologist based on ¹⁸Fluorodeoxyglucose positron emission tomography images. Normal nodes were those with negative diagnosis of breast cancer. The convolutional neural network consisted of 5 convolutional layers with filters from 16 to 128. Receiver operating characteristic analysis was performed to evaluate prediction performance. For comparison, an expert radiologist also scored the same nodes as normal or abnormal.

RESULTS

The convolutional neural network model yielded a specificity of 79.3% ± 5.1%, sensitivity of 92.1% ± 2.9%, positive predictive value of 76.9% ± 4.0%, negative predictive value of 93.3% ± 1.9%, accuracy of 84.8% ± 2.4%, and receiver operating characteristic area under the curve of 0.91 ± 0.02 for the validation data set. These results compared favorably with scoring by radiologists (accuracy of 78%).

CONCLUSION

The results are encouraging and suggest that this approach may prove useful for classifying lymph node status on MRI in clinical settings in patients with breast cancer, although additional studies are needed before routine clinical use can be realized. This approach has the potential to ultimately be a noninvasive alternative to lymph node biopsy.

Fully Automated Segmentation Algorithm for Hematoma Volumetric Analysis in Spontaneous Intracerebral Hemorrhage

Background and Purpose—

Hematoma volume measurements influence prognosis and treatment decisions in patients with spontaneous intracerebral hemorrhage (ICH). The aims of this study are to derive and validate a fully automated segmentation algorithm for ICH volumetric analysis using deep learning methods.

Methods—

In-patient computed tomography scans of 300 consecutive adults (age ≥18 years) with spontaneous, supratentorial ICH who were enrolled in the ICHOP (Intracerebral Hemorrhage Outcomes Project; 2009–2018) were separated into training (n=260) and test (n=40) datasets. A fully automated segmentation algorithm was derived using convolutional neural networks, and it was trained on manual segmentations from the training dataset. The algorithm’s performance was assessed against manual and semiautomated segmentation methods in the test dataset.

Results—

The mean volumetric Dice similarity coefficients for the fully automated segmentation algorithm when tested against manual and semiautomated segmentation methods were 0.894±0.264 and 0.905±0.254, respectively. ICH volumes derived from fully automated versus manual ( R 2 =0.981; P <0.0001), fully automated versus semiautomated ( R 2 =0.978; P <0.0001), and semiautomated versus manual ( R 2 =0.990; P <0001) segmentation methods had strong between-group correlations. The fully automated segmentation algorithm (mean 12.0±2.7 s/scan) was significantly faster than both of the manual (mean 201.5±92.2 s/scan; P <0.001) and semiautomated (mean 288.58±160.3 s/scan; P <0.001) segmentation methods.

Conclusions—

The fully automated segmentation algorithm quantified hematoma volumes from computed tomography scans of supratentorial ICH patients with similar accuracy and substantially greater efficiency compared with manual and semiautomated segmentation methods. External validation of the fully automated segmentation algorithm is warranted.

Wearable-Sensor-based Detection and Prediction of Freezing of Gait in Parkinson's Disease: A Review

Freezing of gait (FOG) is a serious gait disturbance, common in mid- and late-stage Parkinson’s disease, that affects mobility and increases fall risk. Wearable sensors have been used to detect and predict FOG with the ultimate aim of preventing freezes or reducing their effect using gait monitoring and assistive devices. This review presents and assesses the state of the art of FOG detection and prediction using wearable sensors, with the intention of providing guidance on current knowledge, and identifying knowledge gaps that need to be filled and challenges to be considered in future studies. This review searched the Scopus, PubMed, and Web of Science databases to identify studies that used wearable sensors to detect or predict FOG episodes in Parkinson’s disease. Following screening, 74 publications were included, comprising 68 publications detecting FOG, seven predicting FOG, and one in both categories. Details were extracted regarding participants, walking task, sensor type and body location, detection or prediction approach, feature extraction and selection, classification method, and detection and prediction performance. The results showed that increasingly complex machine-learning algorithms combined with diverse feature sets improved FOG detection. The lack of large FOG datasets and highly person-specific FOG manifestation were common challenges. Transfer learning and semi-supervised learning were promising for FOG detection and prediction since they provided person-specific tuning while preserving model generalization.

Single-Sensor Source Localization Using Electromagnetic Time Reversal and Deep Transfer Learning: Application to Lightning

Electromagnetic Time Reversal (EMTR) has been used to locate different types of electromagnetic sources. We propose a novel technique based on the combination of EMTR and Machine Learning (ML) for source localization. We show for the first time that ML techniques can be used in conjunction with EMTR to reduce the required number of sensors to only one for the localization of electromagnetic sources in the presence of scatterers. In the EMTR part, we use 2D-FDTD method to generate 2D profiles of the vertical electric field as RGB images. Next, in the ML part, we take advantage of transfer learning techniques by using the pretrained VGG-19 Convolutional Neural Network (CNN) as the feature extractor tool. To the best of our knowledge, this is the first time that the knowledge of pretrained CNNs is applied to simulation-generated images. We demonstrate the skill of the developed methodology in localizing two kinds of electromagnetic sources, namely RF sources with a bandwidth of 0.1–10 MHz and lightning impulses. For the localization of lightning, based on the experimental recordings in the Säntis region, the new approach enables accurate 2D lightning localization using only one sensor, as opposed to current lightning location systems that need at least two sensors to operate.

Convolutional neural network for efficient estimation of regional brain strains

Head injury models are important tools to study concussion biomechanics but are impractical for real-world use because they are too slow. Here, we develop a convolutional neural network (CNN) to estimate regional brain strains instantly and accurately by conceptualizing head rotational velocity profiles as two-dimensional images for input. We use two impact datasets with augmentation to investigate the CNN prediction performances with a variety of training-testing configurations. Three strain measures are considered, including maximum principal strain (MPS) of the whole brain, MPS of the corpus callosum, and fiber strain of the corpus callosum. The CNN is further tested using an independent impact dataset (N = 314) measured in American football. Based on 2592 training samples, it achieves a testing R² of 0.916 and root mean squared error (RMSE) of 0.014 for MPS of the whole brain. Combining all impact-strain response data available (N = 3069), the CNN achieves an R² of 0.966 and RMSE of 0.013 in a 10-fold cross-validation. This technique may enable a clinical diagnostic capability to a sophisticated head injury model, such as facilitating head impact sensors in concussion detection via a mobile device. In addition, it may transform current acceleration-based injury studies into focusing on regional brain strains. The trained CNN is publicly available along with associated code and examples at https://github.com/Jilab-biomechanics/CNN-brain-strains. They will be updated as needed in the future.

Accuracy of computer-assisted image analysis in the diagnosis of maxillofacial radiolucent lesions: A systematic review and meta-analysis

OBJECTIVES

This study aimed to search for scientific evidence concerning the accuracy of computer-assisted analysis for diagnosing maxillofacial radiolucent lesions.

METHODS

A systematic review was conducted according to the statements of Preferred Reporting Items for Systematic Reviews and Meta-analyses Protocols and considering 10 databases, including the gray literature. Protocol was registered at the International Prospective Register of Systematic Reviews (CRD42018089945). The population, intervention, comparison and outcome strategy was used to define the eligibility criteria and only diagnostic test studies were included. Their risk of bias was assessed by the Joanna Briggs Institute Critical Appraisal tool. Random-effects model meta-analysis was performed and heterogeneity among the included studies was estimated using the I² statistic. The grade of recommendation, assessment, development, and evaluation (GRADE) tool assessed the quality of evidence and strength of recommendation across included studies.

RESULTS

Out of 715 identified citations, four papers, published between 2009 and 2017, fulfilled the criteria and were included in this systematic review. A total of 191 lesions, classified as periapical granuloma and cyst, dentigerous cyst or keratocystic odontogenic tumor, were analyzed. All selected articles scored low risk of bias. The pooled accuracy estimation, regardless of the classification method used, was 88.75% (95% CI = 85.19-92.30). Heterogeneity test reached moderate values (I² = 57.89%). According to the GRADE tool, the analyzed outcome was classified as having low level of certainty.

CONCLUSIONS

The overall evaluation showed all studies presented high accuracy rates of computer-aided diagnosis systems in classifying radiolucent maxillofacial lesions compared to histopathological biopsy. However, due to the moderate heterogeneity found among the studies included in this meta-analysis, a pragmatic recommendation about the use of computer-assisted analysis is not possible.

Dermatologist-level classification of malignant lip diseases using a deep convolutional neural network

BACKGROUND

Deep convolutional neural networks (DCNNs) can classify skin diseases at a level equivalent to a dermatologist, but their performance in specific areas requires further research.

OBJECTIVE

To evaluate the performance of a trained DCNN-based algorithm in classifying benign and malignant lip diseases.

METHODS

A training set of 1629 images (743 malignant, 886 benign) was used with Inception-Resnet-V2. Performance was evaluated using another set of 344 images and 281 images from other hospitals. Classifications by 44 participants (six board-certified dermatologists, 12 dermatology residents, nine medical doctors not specialized in dermatology and 17 medical students) were used for comparison.

RESULTS

The outcomes based on the area under curve, sensitivity and specificity were 0·827 [95% confidence interval (CI) 0·782-0·873], 0·755 (95% CI 0·673-0·827) and 0·803 (95% CI 0·752-0·855), respectively, for the set of 344 images; and 0·774 (95% CI 0·699-0·849), 0·702 (95% CI 0·579-0·808) and 0·759 (95% CI 0·701-0·813), respectively, for the set of 281 images. The DCNN was equivalent to the dermatologists and superior to the nondermatologists in classifying malignancy. After referencing the DCNN result, the mean ± SD Youden index increased significantly for nondermatologists, from 0·201 ± 0·156 to 0·322 ± 0·141 (P < 0·001).

CONCLUSIONS

DCNNs can classify lip diseases at a level similar to dermatologists. This will help unskilled physicians discriminate between benign and malignant lip diseases. What's already known about this topic? Deep convolutional neural networks (DCNNs) can classify malignant and benign skin diseases at a level equivalent to dermatologists. The lips are a unique feature in terms of histology and morphology. Previous studies of DCNNs have not investigated tumours on specific locations. What does this study add? This study shows that DCNNs can distinguish rare malignant and benign lip disorders at the same rate as dermatologists. DCNNs can help nondermatologists to distinguish malignant lip diseases. What are the clinical implications of this work? DCNNs can distinguish malignant and benign skin diseases even at specific locations such as the lips, as well as board-certified dermatologists. Malignant lip diseases are rare and difficult for less trained doctors to differentiate them from benign lesions. This study shows that in dermatology, DCNN can help improve decision-making processes for rare skin diseases in specific areas of the body.

Application of breast cancer diagnosis based on a combination of convolutional neural networks, ridge regression and linear discriminant analysis using invasive breast cancer images processed with autoencoders

Invasive ductal carcinoma cancer, which invades the breast tissues by destroying the milk channels, is the most common type of breast cancer in women. Approximately, 80% of breast cancer patients have invasive ductal carcinoma and roughly 66.6% of these patients are older than 55 years. This situation points out a powerful relationship between the type of breast cancer and progressed woman age. In this study, the classification of invasive ductal carcinoma breast cancer is performed by using deep learning models, which is the sub-branch of artificial intelligence. In this scope, convolutional neural network models and the autoencoder network model are combined. In the experiment, the dataset was reconstructed by processing with the autoencoder model. The discriminative features obtained from convolutional neural network models were utilized. As a result, the most efficient features were determined by using the ridge regression method, and classification was performed using linear discriminant analysis. The best success rate of classification was achieved as 98.59%. Consequently, the proposed approach can be admitted as a successful model in the classification.

MediMLP: Using Grad-CAM to Extract Crucial Variables for Lung Cancer Postoperative Complication Prediction

Lung cancer postoperative complication prediction (PCP) is significant for decreasing the perioperative mortality rate after lung cancer surgery. In this paper we concentrate on two PCP tasks: (1) the binary classification for predicting whether a patient will have postoperative complications; and (2) the three-class multi-label classification for predicting which postoperative complication a patient will experience. Furthermore, an important clinical requirement of PCP is the extraction of crucial variables from electronic medical records. We propose a novel multi-layer perceptron (MLP) model called medical MLP (MediMLP) together with the gradient-weighted class activation mapping (Grad-CAM) algorithm for lung cancer PCP. The proposed MediMLP, which involves one locally connected layer and fully connected layers with a shortcut connection, simultaneously extracts crucial variables and performs PCP tasks. The experimental results indicated that MediMLP outperformed normal MLP on two PCP tasks and had comparable performance with existing feature selection methods. Using MediMLP and further experimental analysis, we found that the variable of "time of indwelling drainage tube" was very relevant to lung cancer postoperative complications.

Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition

Importance

Deep learning convolutional neural networks (CNNs) have shown a performance at the level of dermatologists in the diagnosis of melanoma. Accordingly, further exploring the potential limitations of CNN technology before broadly applying it is of special interest.

Objective

To investigate the association between gentian violet surgical skin markings in dermoscopic images and the diagnostic performance of a CNN approved for use as a medical device in the European market.

Design and Setting

A cross-sectional analysis was conducted from August 1, 2018, to November 30, 2018, using a CNN architecture trained with more than 120 000 dermoscopic images of skin neoplasms and corresponding diagnoses. The association of gentian violet skin markings in dermoscopic images with the performance of the CNN was investigated in 3 image sets of 130 melanocytic lesions each (107 benign nevi, 23 melanomas).

Exposures

The same lesions were sequentially imaged with and without the application of a gentian violet surgical skin marker and then evaluated by the CNN for their probability of being a melanoma. In addition, the markings were removed by manually cropping the dermoscopic images to focus on the melanocytic lesion.

Main Outcomes and Measures

Sensitivity, specificity, and area under the curve (AUC) of the receiver operating characteristic (ROC) curve for the CNN's diagnostic classification in unmarked, marked, and cropped images.

Results

In all, 130 melanocytic lesions (107 benign nevi and 23 melanomas) were imaged. In unmarked lesions, the CNN achieved a sensitivity of 95.7% (95% CI, 79%-99.2%) and a specificity of 84.1% (95% CI, 76.0%-89.8%). The ROC AUC was 0.969. In marked lesions, an increase in melanoma probability scores was observed that resulted in a sensitivity of 100% (95% CI, 85.7%-100%) and a significantly reduced specificity of 45.8% (95% CI, 36.7%-55.2%, P < .001). The ROC AUC was 0.922. Cropping images led to the highest sensitivity of 100% (95% CI, 85.7%-100%), specificity of 97.2% (95% CI, 92.1%-99.0%), and ROC AUC of 0.993. Heat maps created by vanilla gradient descent backpropagation indicated that the blue markings were associated with the increased false-positive rate.

Conclusions and Relevance

This study's findings suggest that skin markings significantly interfered with the CNN's correct diagnosis of nevi by increasing the melanoma probability scores and consequently the false-positive rate. A predominance of skin markings in melanoma training images may have induced the CNN's association of markings with a melanoma diagnosis. Accordingly, these findings suggest that skin markings should be avoided in dermoscopic images intended for analysis by a CNN.

Trial Registration

German Clinical Trial Register (DRKS) Identifier: DRKS00013570.

Deep Learning-Based Stroke Volume Estimation Outperforms Conventional Arterial Contour Method in Patients with Hemodynamic Instability

Although the stroke volume (SV) estimation by arterial blood pressure has been widely used in clinical practice, its accuracy is questionable, especially during periods of hemodynamic instability. We aimed to create novel SV estimating model based on deep-learning (DL) method. A convolutional neural network was applied to estimate SV from arterial blood pressure waveform data recorded from liver transplantation (LT) surgeries. The model was trained using a gold standard referential SV measured via pulmonary artery thermodilution method. Merging a gold standard SV and corresponding 10.24 seconds of arterial blood pressure waveform as an input/output data set with 2-senconds of sliding overlap, 484,384 data sets from 34 LT surgeries were used for training and validation of DL model. The performance of DL model was evaluated by correlation and concordance analyses in another 491,353 data sets from 31 LT surgeries. We also evaluated the performance of pre-existing commercialized model (EV1000), and the performance results of DL model and EV1000 were compared. The DL model provided an acceptable performance throughout the surgery (r = 0.813, concordance rate = 74.15%). During the reperfusion phase, where the most severe hemodynamic instability occurred, DL model showed superior correlation (0.861; 95% Confidence Interval, (CI), 0.855-0.866 vs. 0.570; 95% CI, 0.556-0.584, P < 0.001) and higher concordance rate (90.6% vs. 75.8%) over EV1000. In conclusion, the DL-based model was superior for estimating intraoperative SV and thus might guide physicians to precise intraoperative hemodynamic management. Moreover, the DL model seems to be particularly promising because it outperformed EV1000 in circumstance of rapid hemodynamic changes where physicians need most help.

Radiomic features of glucose metabolism enable prediction of outcome in mantle cell lymphoma

PURPOSE

To determine whether [¹⁸F]FDG PET/CT-derived radiomic features alone or in combination with clinical, laboratory and biological parameters are predictive of 2-year progression-free survival (PFS) in patients with mantle cell lymphoma (MCL), and whether they enable outcome prognostication.

METHODS

Included in this retrospective study were 107 treatment-naive MCL patients scheduled to receive CD20 antibody-based immuno(chemo)therapy. Standardized uptake values (SUV), total lesion glycolysis, and 16 co-occurrence matrix radiomic features were extracted from metabolic tumour volumes on pretherapy [¹⁸F]FDG PET/CT scans. A multilayer perceptron neural network in combination with logistic regression analyses for feature selection was used for prediction of 2-year PFS. International prognostic indices for MCL (MIPI and MIPI-b) were calculated and combined with the radiomic data. Kaplan-Meier estimates with log-rank tests were used for PFS prognostication.

RESULTS

SUVmean (OR 1.272, P = 0.013) and Entropy (heterogeneity of glucose metabolism; OR 1.131, P = 0.027) were significantly predictive of 2-year PFS: median areas under the curve were 0.72 based on the two radiomic features alone, and 0.82 with the addition of clinical/laboratory/biological data. Higher SUVmean in combination with higher Entropy (SUVmean >3.55 and entropy >3.5), reflecting high "metabolic risk", was associated with a poorer prognosis (median PFS 20.3 vs. 39.4 months, HR 2.285, P = 0.005). The best PFS prognostication was achieved using the MIPI-bm (MIPI-b and metabolic risk combined): median PFS 43.2, 38.2 and 20.3 months in the low-risk, intermediate-risk and high-risk groups respectively (P = 0.005).

CONCLUSION

In MCL, the [¹⁸F]FDG PET/CT-derived radiomic features SUVmean and Entropy may improve prediction of 2-year PFS and PFS prognostication. The best results may be achieved using a combination of metabolic, clinical, laboratory and biological parameters.

Deep Convolutional Neural Network-Based Positron Emission Tomography Analysis Predicts Esophageal Cancer Outcome

In esophageal cancer, few prediction tools can be confidently used in current clinical practice. We developed a deep convolutional neural network (CNN) with 798 positron emission tomography (PET) scans of esophageal squamous cell carcinoma and 309 PET scans of stage I lung cancer. In the first stage, we pretrained a 3D-CNN with all PET scans for a task to classify the scans into esophageal cancer or lung cancer. Overall, 548 of 798 PET scans of esophageal cancer patients were included in the second stage with an aim to classify patients who expired within or survived more than one year after diagnosis. The area under the receiver operating characteristic curve (AUC) was used to evaluate model performance. In the pretrain model, the deep CNN attained an AUC of 0.738 in identifying patients who expired within one year after diagnosis. In the survival analysis, patients who were predicted to be expired but were alive at one year after diagnosis had a 5-year survival rate of 32.6%, which was significantly worse than the 5-year survival rate of the patients who were predicted to survive and were alive at one year after diagnosis (50.5%, p < 0.001). These results suggest that the prediction model could identify tumors with more aggressive behavior. In the multivariable analysis, the prediction result remained an independent prognostic factor (hazard ratio: 2.830; 95% confidence interval: 2.252–3.555, p < 0.001). We conclude that a 3D-CNN can be trained with PET image datasets to predict esophageal cancer outcome with acceptable accuracy.

Preliminary study on the application of deep learning system to diagnosis of Sjögren's syndrome on CT images

OBJECTIVES

This study estimated the diagnostic performance of a deep learning system for detection of Sjögren's syndrome (SjS) on CT, and compared it with the performance of radiologists.

METHODS

CT images were assessed from 25 patients confirmed to have SjS based on the both Japanese criteria and American-European Consensus Group criteria and 25 control subjects with no parotid gland abnormalities who were examined for other diseases. 10 CT slices were obtained for each patient. From among the total of 500 CT images, 400 images (200 from 20 SjS patients and 200 from 20 control subjects) were employed as the training data set and 100 images (50 from 5 SjS patients and 50 from 5 control subjects) were used as the test data set. The performance of a deep learning system for diagnosing SjS from the CT images was compared with the diagnoses made by six radiologists (three experienced and three inexperienced radiologists).

RESULTS

The accuracy, sensitivity, and specificity of the deep learning system were 96.0%, 100% and 92.0%, respectively. The corresponding values of experienced radiologists were 98.3%, 99.3% and 97.3% being equivalent to the deep learning, while those of inexperienced radiologists were 83.5%, 77.9% and 89.2%. The area under the curve of inexperienced radiologists were significantly different from those of the deep learning system and the experienced radiologists.

CONCLUSIONS

The deep learning system showed a high diagnostic performance for SjS, suggesting that it could possibly be used for diagnostic support when interpreting CT images.

Modeling and Experimental Study for Online Measurement of Hydraulic Cylinder Micro Leakage Based on Convolutional Neural Network

Internal leakage is the most common failure of hydraulic cylinder; when it increases, it decreases volumetric efficiency, pressure and speed of the hydraulic cylinder, and can seriously affect the normal operation of the hydraulic cylinder, so it is important to measure it, especially to measure it online. Firstly, the principle of internal leakage online measurement is proposed, including the online measurement system, the fixed mode of the strain gauge and the mathematical model of the flow-strain signal conversion. Secondly, an experimental system is established to collect internal leakages and strain values, and the data is processed. Finally, the convolutional neural network (CNN), BP neural network (BPNN), Radial Basis Function Network (RBF), and Support Vector Regression (SVR) are used to predict the hydraulic cylinder leakage; the comparison of experimental results show that the CNN has high accuracy and high efficiency. This study provides a new idea for online measurement of small flow on other hydraulic components.

Artificial intelligence in breast imaging

This article reviews current limitations and future opportunities for the application of computer-aided detection (CAD) systems and artificial intelligence in breast imaging. Traditional CAD systems in mammography screening have followed a rules-based approach, incorporating domain knowledge into hand-crafted features before using classical machine learning techniques as a classifier. The first commercial CAD system, ImageChecker M1000, relies on computer vision techniques for pattern recognition. Unfortunately, CAD systems have been shown to adversely affect some radiologists' performance and increase recall rates. The Digital Mammography DREAM Challenge was a multidisciplinary collaboration that provided 640,000 mammography images for teams to help decrease false-positive rates in breast cancer screening. Winning solutions leveraged deep learning's (DL) automatic hierarchical feature learning capabilities and used convolutional neural networks. Start-ups Therapixel and Kheiron Medical Technologies are using DL for breast cancer screening. With increasing use of digital breast tomosynthesis, specific artificial intelligence (AI)-CAD systems are emerging to include iCAD's PowerLook Tomo Detection and ScreenPoint Medical's Transpara. Other AI-CAD systems are focusing on breast diagnostic techniques such as ultrasound and magnetic resonance imaging (MRI). There is a gap in the market for contrast-enhanced spectral mammography AI-CAD tools. Clinical implementation of AI-CAD tools requires testing in scenarios mimicking real life to prove its usefulness in the clinical environment. This requires a large and representative dataset for testing and assessment of the reader's interaction with the tools. A cost-effectiveness assessment should be undertaken, with a large feasibility study carried out to ensure there are no unintended consequences. AI-CAD systems should incorporate explainable AI in accordance with the European Union General Data Protection Regulation (GDPR).

What the radiologist should know about artificial intelligence - an ESR white paper

This paper aims to provide a review of the basis for application of AI in radiology, to discuss the immediate ethical and professional impact in radiology, and to consider possible future evolution.Even if AI does add significant value to image interpretation, there are implications outside the traditional radiology activities of lesion detection and characterisation. In radiomics, AI can foster the analysis of the features and help in the correlation with other omics data. Imaging biobanks would become a necessary infrastructure to organise and share the image data from which AI models can be trained. AI can be used as an optimising tool to assist the technologist and radiologist in choosing a personalised patient's protocol, tracking the patient's dose parameters, providing an estimate of the radiation risks. AI can also aid the reporting workflow and help the linking between words, images, and quantitative data. Finally, AI coupled with CDS can improve the decision process and thereby optimise clinical and radiological workflow.

Spatial Signal Detection Using Continuous Shrinkage Priors

Motivated by the problem of detecting changes in two-dimensional X-ray diffraction data, we propose a Bayesian spatial model for sparse signal detection in image data. Our model places considerable mass near zero and has heavy tails to reflect the prior belief that the image signal is zero for most pixels and large for an important subset. We show that the spatial prior places mass on nearby locations simultaneously being zero, and also allows for nearby locations to simultaneously be large signals. The form of the prior also facilitates efficient computing for large images. We conduct a simulation study to evaluate the properties of the proposed prior and show that it outperforms other spatial models. We apply our method in the analysis of X-ray diffraction data from a two-dimensional area detector to detect changes in the pattern when the material is exposed to an electric field.

Artificial intelligence in medical imaging of the liver

Artificial intelligence (AI), particularly deep learning algorithms, is gaining extensive attention for its excellent performance in image-recognition tasks. They can automatically make a quantitative assessment of complex medical image characteristics and achieve an increased accuracy for diagnosis with higher efficiency. AI is widely used and getting increasingly popular in the medical imaging of the liver, including radiology, ultrasound, and nuclear medicine. AI can assist physicians to make more accurate and reproductive imaging diagnosis and also reduce the physicians’ workload. This article illustrates basic technical knowledge about AI, including traditional machine learning and deep learning algorithms, especially convolutional neural networks, and their clinical application in the medical imaging of liver diseases, such as detecting and evaluating focal liver lesions, facilitating treatment, and predicting liver treatment response. We conclude that machine-assisted medical services will be a promising solution for future liver medical care. Lastly, we discuss the challenges and future directions of clinical application of deep learning techniques.

Correlating exhaled aerosol images to small airway obstructive diseases: A study with dynamic mode decomposition and machine learning

Background

Exhaled aerosols from lungs have unique patterns, and their variation can be correlated to the underlying lung structure and associated abnormities. However, it is challenging to characterize such aerosol patterns and differentiate their difference because of their complexity. This challenge is even greater for small airway diseases, where the disturbance signals are weak.

Objectives and methods

The objective of this study is exploiting different feature extraction algorithms to develop a practical classifier to diagnose obstructive lung diseases using exhaled aerosol images. These include proper orthogonal decomposition (POD), principal component analysis (PCA), dynamic mode decomposition (DMD), and DMD with control (DMDC). Aerosol images were generated via physiology-based simulations in one normal and four diseased airway models in G7-9 bronchioles. The image data were classified using both the support vector machine (SVM) and random forest (RF) algorithms. The effectiveness of different features was evaluated by classification accuracy and misclassification rate.

Findings

Results show a significantly higher performance using dynamic feature extractions (DMD and DMDC) than static algorithms (POD and PCA). Adding the control variables to DMD further improved classification accuracy. Comparing the classification methods, RF persistently outperformed SVM for all types of features considered. While the performance of RF constantly increased with the number of features retained, the performance of SVM peaked at 50 and decreased thereafter. The 5-class classification accuracy was 94.8% using the DMDC-RF model and 93.0% using the DMD-RF model, both of which were higher than 87.0% in the previous study that used fractal dimension features.

Conclusion

Considering that disease progression is inherently a dynamic process, DMD(C)-based feature extraction preserves temporal information and is preferred over POD and PCA. Compared with hand-crafted features like fractals, feature extraction by DMD and DMDC is automatic and more accurate.

An overview of deep learning in medical imaging focusing on MRI

What has happened in machine learning lately, and what does it mean for the future of medical image analysis? Machine learning has witnessed a tremendous amount of attention over the last few years. The current boom started around 2009 when so-called deep artificial neural networks began outperforming other established models on a number of important benchmarks. Deep neural networks are now the state-of-the-art machine learning models across a variety of areas, from image analysis to natural language processing, and widely deployed in academia and industry. These developments have a huge potential for medical imaging technology, medical data analysis, medical diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis. As this has become a very broad and fast expanding field we will not survey the entire landscape of applications, but put particular focus on deep learning in MRI. Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core references; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point for people interested in experimenting and perhaps contributing to the field of deep learning for medical imaging by pointing out good educational resources, state-of-the-art open-source code, and interesting sources of data and problems related medical imaging.

Deep-learning classification using convolutional neural network for evaluation of maxillary sinusitis on panoramic radiography

OBJECTIVES

To apply a deep-learning system for diagnosis of maxillary sinusitis on panoramic radiography, and to clarify its diagnostic performance.

METHODS

Training data for 400 healthy and 400 inflamed maxillary sinuses were enhanced to 6000 samples in each category by data augmentation. Image patches were input into a deep-learning system, the learning process was repeated for 200 epochs, and a learning model was created. Newly-prepared testing image patches from 60 healthy and 60 inflamed sinuses were input into the learning model, and the diagnostic performance was calculated. Receiver-operating characteristic (ROC) curves were drawn, and the area under the curve (AUC) values were obtained. The results were compared with those of two experienced radiologists and two dental residents.

RESULTS

The diagnostic performance of the deep-learning system for maxillary sinusitis on panoramic radiographs was high, with accuracy of 87.5%, sensitivity of 86.7%, specificity of 88.3%, and AUC of 0.875. These values showed no significant differences compared with those of the radiologists and were higher than those of the dental residents.

CONCLUSIONS

The diagnostic performance of the deep-learning system for maxillary sinusitis on panoramic radiographs was sufficiently high. Results from the deep-learning system are expected to provide diagnostic support for inexperienced dentists.

Imaging Biomarkers for the Diagnosis and Prognosis of Neurodegenerative Diseases. The Example of Amyotrophic Lateral Sclerosis

The term amyotrophic lateral sclerosis (ALS) comprises a heterogeneous group of fatal neurodegenerative disorders of largely unknown etiology characterized by the upper motor neurons (UMN) and/or lower motor neurons (LMN) degeneration. The development of brain imaging biomarkers is essential to advance in the diagnosis, stratification and monitoring of ALS, both in the clinical practice and clinical trials. In this review, the characteristics of an optimal imaging biomarker and common pitfalls in biomarkers evaluation will be discussed. Moreover, the development and application of the most promising brain magnetic resonance (MR) imaging biomarkers will be reviewed. Finally, the integration of both qualitative and quantitative multimodal brain MR biomarkers in a structured report will be proposed as a support tool for ALS diagnosis and stratification.