The area under an ROC curve with limited information

Automatic detection of pleural line and lung sliding in lung ultrasonography using convolutional neural networks

Background

Lung ultrasonography (LUS) is a valuable diagnostic tool, but there is a shortage of LUS experts with extensive knowledge and significant experience in the field. Convolutional neural networks (CNNs) have the potential to mitigate this issue by facilitating computer-aided diagnosis.

Methods

We propose computer-aided system by a CNN-based method for LUS diagnosis. As the first consideration, we investigated pleural line and lung sliding. The pleural line indicates the position of pleura in an ultrasound image, and LUS is performed after first confirming the position of pleural line. Lung sliding defined as the movement of the pleural line, and the absence of this feature is associated with pneumothorax.

Results

Our proposed method accurately detected pleural line and lung sliding, demonstrating its potential to provide valuable diagnostic information on lung lesions.

Machine learning-assisted internal standard calibration label-free SERS strategy for colon cancer detection

Liquid biopsies have significance for early colon cancer screening and improving patient survival. Recently, several researchers have applied surface-enhanced Raman spectroscopy (SERS) for the label-free and non-invasive detection of serum. Most of these studies performed the assay using a mixture of noble metal nanoparticles (NMNPs) with serum. However, SERS analysis of serum remains a challenge in terms of reproducibility and stability, as NMNPs tend to aggregate when mixed with serum, resulting in a non-uniform distribution of hot spots. Here, we report on the non-invasive identification of colon cancer (CC) using an internal standard (IS)-calibrated label-free serum SERS assay in combination with machine learning. Serum SERS spectra of 50 CC patients and 50 health volunteers have been obtained using silver nanoparticle (Ag NP) colloid and mercaptopropionic acid-modified Ag NPs (Ag NPs-MPA) as the SERS substrates. Decision tree (DT), random forest (RF), and principal component and linear discriminant analysis (PCA-LDA) algorithms were utilized to establish the diagnosis model for SERS spectra data classifying. The results show that the RF model provides a high diagnostic accuracy compared to PCA-LDA. Following calibration with IS molecules, high diagnostic accuracy of over 90% and 100% specificity can be achieved with DT, RF, and PCA-LDA algorithms to differentiate between cancer and normal groups. The results from this exploratory work demonstrate that serum SERS detection combined with multivariate statistical methods and IS calibration has great potential for the non-invasive and label-free detection of CC.

ProtInteract: A deep learning framework for predicting protein-protein interactions

Proteins mainly perform their functions by interacting with other proteins. Protein-protein interactions underpin various biological activities such as metabolic cycles, signal transduction, and immune response. However, due to the sheer number of proteins, experimental methods for finding interacting and non-interacting protein pairs are time-consuming and costly. We therefore developed the ProtInteract framework to predict protein-protein interaction. ProtInteract comprises two components: first, a novel autoencoder architecture that encodes each protein's primary structure to a lower-dimensional vector while preserving its underlying sequence attributes. This leads to faster training of the second network, a deep convolutional neural network (CNN) that receives encoded proteins and predicts their interaction under three different scenarios. In each scenario, the deep CNN predicts the class of a given encoded protein pair. Each class indicates different ranges of confidence scores corresponding to the probability of whether a predicted interaction occurs or not. The proposed framework features significantly low computational complexity and relatively fast response. The contributions of this work are twofold. First, ProtInteract assimilates the protein's primary structure into a pseudo-time series. Therefore, we leverage the nature of the time series of proteins and their physicochemical properties to encode a protein's amino acid sequence into a lower-dimensional vector space. This approach enables extracting highly informative sequence attributes while reducing computational complexity. Second, the ProtInteract framework utilises this information to identify protein interactions with other proteins based on its amino acid configuration. Our results suggest that the proposed framework performs with high accuracy and efficiency in predicting protein-protein interactions.

A deep learning model incorporating spatial and temporal information successfully detects visual field worsening using a consensus based approach

Glaucoma is a leading cause of irreversible blindness, and its worsening is most often monitored with visual field (VF) testing. Deep learning models (DLM) may help identify VF worsening consistently and reproducibly. In this study, we developed and investigated the performance of a DLM on a large population of glaucoma patients. We included 5099 patients (8705 eyes) seen at one institute from June 1990 to June 2020 that had VF testing as well as clinician assessment of VF worsening. Since there is no gold standard to identify VF worsening, we used a consensus of six commonly used algorithmic methods which include global regressions as well as point-wise change in the VFs. We used the consensus decision as a reference standard to train/test the DLM and evaluate clinician performance. 80%, 10%, and 10% of patients were included in training, validation, and test sets, respectively. Of the 873 eyes in the test set, 309 [60.6%] were from females and the median age was 62.4; (IQR 54.8-68.9). The DLM achieved an AUC of 0.94 (95% CI 0.93-0.99). Even after removing the 6 most recent VFs, providing fewer data points to the model, the DLM successfully identified worsening with an AUC of 0.78 (95% CI 0.72-0.84). Clinician assessment of worsening (based on documentation from the health record at the time of the final VF in each eye) had an AUC of 0.64 (95% CI 0.63-0.66). Both the DLM and clinician performed worse when the initial disease was more severe. This data shows that a DLM trained on a consensus of methods to define worsening successfully identified VF worsening and could help guide clinicians during routine clinical care.

Automatic Diagnosis Labeling of Cardiovascular MRI by Using Semisupervised Natural Language Processing of Text Reports

PURPOSE

To assess whether the semisupervised natural language processing (NLP) of text from clinical radiology reports could provide useful automated diagnosis categorization for ground truth labeling to overcome manual labeling bottlenecks in the machine learning pipeline.

MATERIALS AND METHODS

In this retrospective study, 1503 text cardiac MRI reports from 2016 to 2019 were manually annotated for five diagnoses by clinicians: normal, dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy, myocardial infarction (MI), and myocarditis. A semisupervised method that uses bidirectional encoder representations from transformers (BERT) pretrained on 1.14 million scientific publications was fine-tuned by using the manually extracted labels, with a report dataset split into groups of 801 for training, 302 for validation, and 400 for testing. The model's performance was compared with two traditional NLP models: a rule-based model and a support vector machine (SVM) model. The models' F1 scores and receiver operating characteristic curves were used to analyze performance.

RESULTS

After 15 epochs, the F1 scores on the test set of 400 reports were as follows: normal, 84%; DCM, 79%; hypertrophic cardiomyopathy, 86%; MI, 91%; and myocarditis, 86%. The pooled F1 score and area under the receiver operating curve were 86% and 0.96, respectively. On the same test set, the BERT model had a higher performance than the rule-based model (F1 score, 42%) and SVM model (F1 score, 82%). Diagnosis categories classified by using the BERT model performed the labeling of 1000 MR images in 0.2 second.

CONCLUSION

The developed model used labels extracted from radiology reports to provide automated diagnosis categorization of MR images with a high level of performance.Keywords: Semisupervised Learning, Diagnosis/Classification/Application Domain, Named Entity Recognition, MRI Supplemental material is available for this article. © RSNA, 2021.

Label-free surface-enhanced Raman spectroscopy for diagnosis and analysis of serum samples with different types lung cancer

Screening and detection of early lung cancer is important for diagnosis and prognosis. Intervention in early stage of lung cancer can significantly improve the cure and survival of patients. Surface-enhanced Raman spectroscopy (SERS) is an increasingly popular method of diagnosing cancer. We used silver nanoparticles (AgNPs) as the Raman-enhanced substrate to increase Raman signals, which contributes to the subsequent classification of lung cancer and normal serum. SERS acquired from the serum indicated the difference in biochemical components between cancerous (n = 51) lung serum and normal (n = 18) serum. Principal component analysis (PCA) and partial least-squares discriminant analysis (PLS-DA) were utilized to establish the identification model, and the various indicators of PLS-DA were all superior to those of the PLS model. Our study offers a new proposal for the universal applicability of analysis and identification with SERS of serum samples in clinical diagnosis.

Designing deep learning studies in cancer diagnostics

The number of publications on deep learning for cancer diagnostics is rapidly increasing, and systems are frequently claimed to perform comparable with or better than clinicians. However, few systems have yet demonstrated real-world medical utility. In this Perspective, we discuss reasons for the moderate progress and describe remedies designed to facilitate transition to the clinic. Recent, presumably influential, deep learning studies in cancer diagnostics, of which the vast majority used images as input to the system, are evaluated to reveal the status of the field. By manipulating real data, we then exemplify that much and varied training data facilitate the generalizability of neural networks and thus the ability to use them clinically. To reduce the risk of biased performance estimation of deep learning systems, we advocate evaluation in external cohorts and strongly advise that the planned analyses, including a predefined primary analysis, are described in a protocol preferentially stored in an online repository. Recommended protocol items should be established for the field, and we present our suggestions.

A new deep learning approach integrated with clinical data for the dermoscopic differentiation of early melanomas from atypical nevi

BACKGROUND

Timely recognition of malignant melanoma (MM) is challenging for dermatologists worldwide and represents the main determinant for mortality. Dermoscopic examination is influenced by dermatologists' experience and fails to achieve adequate accuracy and reproducibility in discriminating atypical nevi (AN) from early melanomas (EM).

OBJECTIVE

We aimed to develop a Deep Convolutional Neural Network (DCNN) model able to support dermatologists in the classification and management of atypical melanocytic skin lesions (aMSL).

METHODS

A training set (630 images), a validation set (135) and a testing set (214) were derived from the idScore dataset of 979 challenging aMSL cases in which the dermoscopic image is integrated with clinical data (age, sex, body site and diameter) and associated with histological data. A DCNN_aMSL architecture was designed and then trained on both dermoscopic images of aMSL and the clinical/anamnestic data, resulting in the integrated "iDCNN_aMSL" model. Responses of 111 dermatologists with different experience levels on both aMSL classification (intuitive diagnosis) and management decisions (no/long follow-up; short follow-up; excision/preventive excision) were compared with the DCNNs models.

RESULTS

In the lesion classification study, the iDCNN_aMSL achieved the best accuracy, reaching an AUC = 90.3 %, SE = 86.5 % and SP = 73.6 %, compared to DCNN_aMSL (SE = 89.2 %, SP = 65.7 %) and intuitive diagnosis of dermatologists (SE = 77.0 %; SP = 61.4 %).

CONCLUSIONS

The iDCNN_aMSL proved to be the best support tool for management decisions reducing the ratio of inappropriate excision. The proposed iDCNN_aMSL model can represent a valid support for dermatologists in discriminating AN from EM with high accuracy and for medical decision making by reducing their rates of inappropriate excisions.

Keratinocytic Skin Cancer Detection on the Face Using Region-Based Convolutional Neural Network

Importance

Detection of cutaneous cancer on the face using deep-learning algorithms has been challenging because various anatomic structures create curves and shades that confuse the algorithm and can potentially lead to false-positive results.

Objective

To evaluate whether an algorithm can automatically locate suspected areas and predict the probability of a lesion being malignant.

Design, Setting, and Participants

Region-based convolutional neural network technology was used to create 924 538 possible lesions by extracting nodular benign lesions from 182 348 clinical photographs. After manually or automatically annotating these possible lesions based on image findings, convolutional neural networks were trained with 1 106 886 image crops to locate and diagnose cancer. Validation data sets (2844 images from 673 patients; mean [SD] age, 58.2 [19.9] years; 308 men [45.8%]; 185 patients with malignant tumors, 305 with benign tumors, and 183 free of tumor) were obtained from 3 hospitals between January 1, 2010, and September 30, 2018.

Main Outcomes and Measures

The area under the receiver operating characteristic curve, F1 score (mean of precision and recall; range, 0.000-1.000), and Youden index score (sensitivity + specificity -1; 0%-100%) were used to compare the performance of the algorithm with that of the participants.

Results

The algorithm analyzed a mean (SD) of 4.2 (2.4) photographs per patient and reported the malignancy score according to the highest malignancy output. The area under the receiver operating characteristic curve for the validation data set (673 patients) was 0.910. At a high-sensitivity cutoff threshold, the sensitivity and specificity of the model with the 673 patients were 76.8% and 90.6%, respectively. With the test partition (325 images; 80 patients), the performance of the algorithm was compared with the performance of 13 board-certified dermatologists, 34 dermatology residents, 20 nondermatologic physicians, and 52 members of the general public with no medical background. When the disease screening performance was evaluated at high sensitivity areas using the F1 score and Youden index score, the algorithm showed a higher F1 score (0.831 vs 0.653 [0.126], P < .001) and Youden index score (0.675 vs 0.417 [0.124], P < .001) than that of nondermatologic physicians. The accuracy of the algorithm was comparable with that of dermatologists (F1 score, 0.831 vs 0.835 [0.040]; Youden index score, 0.675 vs 0.671 [0.100]).

Conclusions and Relevance

The results of the study suggest that the algorithm could localize and diagnose skin cancer without preselection of suspicious lesions by dermatologists.

Use of Hy's law and a new composite algorithm to predict acute liver failure in patients with drug-induced liver injury

BACKGROUND & AIMS

Hy's Law, which states that hepatocellular drug-induced liver injury (DILI) with jaundice indicates a serious reaction, is used widely to determine risk for acute liver failure (ALF). We aimed to optimize the definition of Hy's Law and to develop a model for predicting ALF in patients with DILI.

METHODS

We collected data from 771 patients with DILI (805 episodes) from the Spanish DILI registry, from April 1994 through August 2012. We analyzed data collected at DILI recognition and at the time of peak levels of alanine aminotransferase (ALT) and total bilirubin (TBL).

RESULTS

Of the 771 patients with DILI, 32 developed ALF. Hepatocellular injury, female sex, high levels of TBL, and a high ratio of aspartate aminotransferase (AST):ALT were independent risk factors for ALF. We compared 3 ways to use Hy's Law to predict which patients would develop ALF; all included TBL greater than 2-fold the upper limit of normal (×ULN) and either ALT level greater than 3 × ULN, a ratio (R) value (ALT × ULN/alkaline phosphatase × ULN) of 5 or greater, or a new ratio (nR) value (ALT or AST, whichever produced the highest ×ULN/ alkaline phosphatase × ULN value) of 5 or greater. At recognition of DILI, the R- and nR-based models identified patients who developed ALF with 67% and 63% specificity, respectively, whereas use of only ALT level identified them with 44% specificity. However, the level of ALT and the nR model each identified patients who developed ALF with 90% sensitivity, whereas the R criteria identified them with 83% sensitivity. An equal number of patients who did and did not develop ALF had alkaline phosphatase levels greater than 2 × ULN. An algorithm based on AST level greater than 17.3 × ULN, TBL greater than 6.6 × ULN, and AST:ALT greater than 1.5 identified patients who developed ALF with 82% specificity and 80% sensitivity.

CONCLUSIONS

When applied at DILI recognition, the nR criteria for Hy's Law provides the best balance of sensitivity and specificity whereas our new composite algorithm provides additional specificity in predicting the ultimate development of ALF.

Comparison of different analytic algorithms for interpretation of the Swedish interactive threshold algorithm strategy

OBJECTIVE

To compare 4 analytic algorithms for interpretation of the Swedish Interactive Threshold Algorithm.

INTRODUCTION

Analytic algorithms were initially developed for interpretation of standard automated perimetry (using a full threshold strategy). The Swedish interactive threshold algorithm is a novel strategy that was developed to shorten test duration.

METHODS

One hundred forty-three printouts of normal and glaucomatous patients were analyzed using Caprioli's (strict, moderate and liberal) criteria and Anderson's modified criteria for perimetric defect. Areas under the receiver operator characteristics (ROC) curves, sensitivity, and specificity for each criteria were calculated.

RESULTS

Caprioli's strict and Anderson's modified criteria presented similar sensitivity (94.5% and 92.3%, respectively) and specificity (63.5% and 61.5%, respectively). Caprioli's liberal criteria were more sensitive (98.9%) and less specific (42.5%) than the other three criteria.

CONCLUSION

Both Caprioli's and Anderson's modified criteria can be used for interpretation of the Swedish interactive threshold algorithm.

Assessment of functional capacity of the musculoskeletal system in the context of work, daily living, and sport: a systematic review

The aim of this systematic review was to survey methods to assess the functional capacity of the musculoskeletal system within the context of work, daily activities, and sport. The following key words and synonyms were used: functional physical assessment, healthy/disabled subjects, and instruments. After applying the inclusion criteria on 697 potential studies and a methodological quality appraisal, 34 studies were included. A level of reliability > 0.80 and of > 0.60 resp 0.75 and 0.90, dependent of type of validity, was considered high. Four questionnaires (the Oswestry Disability Index, the Pain Disability Index, the Roland-Morris Disability Questionnaire, and the Upper Extremity Functional Scale) have high levels on both validity and reliability. None of the functional tests had a high level of both reliability and validity. A combination of a questionnaire and a functional test would seem to be the best instrument to assess functional capacity of the musculoskeletal system, but need further examined.