A comparison of logistic regression analysis and an artificial neural network using the BI-RADS lexicon for ultrasonography in conjunction with introbserver variability

Machine learning for screening and predicting the risk of anti-MDA5 antibody in juvenile dermatomyositis children

Objective

The anti-MDA5 (anti-melanoma differentiation associated gene 5) antibody is often associated with a poor prognosis in juvenile dermatomyositis (JDM) patients. In many developing countries, there is limited ability to access myositis- specific antibodies due to financial and technological issues, especially in remote regions. This study was performed to develop a prediction model for screening anti-MDA5 antibodies in JDM patients with commonly available clinical findings.

Methods

A cross-sectional study was undertaken with 152 patients enrolled from the inpatient wards of Beijing Children's Hospital between June 2018 and September 2021. Stepwise logistic regression, least absolute shrinkage and selection operator (LASSO) regression, and the random forest (RF) method were used to fit the model. Model discrimination, calibration, and decision curve analysis were performed for validation.

Results

The final prediction model included eight clinical variables (gender, fever, alopecia, periungual telangiectasia, digital ulcer, interstitial lung disease, arthritis/arthralgia, and Gottron sign) and four auxiliary results (WBC, CK, CKMB, and ALB). An anti-MDA5 antibody risk probability-predictive nomogram was established with an AUC of 0.975 predicted by the random forest algorithm. The model was internally validated by Harrell's concordance index (0.904), the Brier score (0.052), and a 500 bootstrapped satisfactory calibration curve. According to the net benefit and predicted probability thresholds of decision curve analysis, the established model showed a significantly higher net benefit than the traditional logistic regression model.

Conclusion

We developed a prediction model using routine clinical assessments to screen for JDM patients likely to be anti-MDA5 positive. This new tool may effectively predict the detection of anti-MDA5 in these patients using a non-invasive and efficient way.

Supervised Classification of Healthcare Text Data Based on Context-Defined Categories

Achieving a good success rate in supervised classification analysis of a text dataset, where the relationship between the text and its label can be extracted from the context, but not from isolated words in the text, is still an important challenge facing the fields of statistics and machine learning. For this purpose, we present a novel mathematical framework. We then conduct a comparative study between established classification methods for the case where the relationship between the text and the corresponding label is clearly depicted by specific words in the text. In particular, we use logistic LASSO, artificial neural networks, support vector machines, and decision-tree-like procedures. This methodology is applied to a real case study involving mapping Consolidated Framework for Implementation and Research (CFIR) constructs to health-related text data and achieves a prediction success rate of over 80% when just the first 55% of the text, or more, is used for training and the remaining for testing. The results indicate that the methodology can be useful to accelerate the CFIR coding process.

The importance of multi-modal imaging and clinical information for humans and AI-based algorithms to classify breast masses (INSPiRED 003): an international, multicenter analysis

OBJECTIVES

AI-based algorithms for medical image analysis showed comparable performance to human image readers. However, in practice, diagnoses are made using multiple imaging modalities alongside other data sources. We determined the importance of this multi-modal information and compared the diagnostic performance of routine breast cancer diagnosis to breast ultrasound interpretations by humans or AI-based algorithms.

METHODS

Patients were recruited as part of a multicenter trial (NCT02638935). The trial enrolled 1288 women undergoing routine breast cancer diagnosis (multi-modal imaging, demographic, and clinical information). Three physicians specialized in ultrasound diagnosis performed a second read of all ultrasound images. We used data from 11 of 12 study sites to develop two machine learning (ML) algorithms using unimodal information (ultrasound features generated by the ultrasound experts) to classify breast masses which were validated on the remaining study site. The same ML algorithms were subsequently developed and validated on multi-modal information (clinical and demographic information plus ultrasound features). We assessed performance using area under the curve (AUC).

RESULTS

Of 1288 breast masses, 368 (28.6%) were histopathologically malignant. In the external validation set (n = 373), the performance of the two unimodal ultrasound ML algorithms (AUC 0.83 and 0.82) was commensurate with performance of the human ultrasound experts (AUC 0.82 to 0.84; p for all comparisons > 0.05). The multi-modal ultrasound ML algorithms performed significantly better (AUC 0.90 and 0.89) but were statistically inferior to routine breast cancer diagnosis (AUC 0.95, p for all comparisons ≤ 0.05).

CONCLUSIONS

The performance of humans and AI-based algorithms improves with multi-modal information.

KEY POINTS

• The performance of humans and AI-based algorithms improves with multi-modal information. • Multimodal AI-based algorithms do not necessarily outperform expert humans. • Unimodal AI-based algorithms do not represent optimal performance to classify breast masses.

Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis

Deep learning (DL) has the potential to transform medical diagnostics. However, the diagnostic accuracy of DL is uncertain. Our aim was to evaluate the diagnostic accuracy of DL algorithms to identify pathology in medical imaging. Searches were conducted in Medline and EMBASE up to January 2020. We identified 11,921 studies, of which 503 were included in the systematic review. Eighty-two studies in ophthalmology, 82 in breast disease and 115 in respiratory disease were included for meta-analysis. Two hundred twenty-four studies in other specialities were included for qualitative review. Peer-reviewed studies that reported on the diagnostic accuracy of DL algorithms to identify pathology using medical imaging were included. Primary outcomes were measures of diagnostic accuracy, study design and reporting standards in the literature. Estimates were pooled using random-effects meta-analysis. In ophthalmology, AUC's ranged between 0.933 and 1 for diagnosing diabetic retinopathy, age-related macular degeneration and glaucoma on retinal fundus photographs and optical coherence tomography. In respiratory imaging, AUC's ranged between 0.864 and 0.937 for diagnosing lung nodules or lung cancer on chest X-ray or CT scan. For breast imaging, AUC's ranged between 0.868 and 0.909 for diagnosing breast cancer on mammogram, ultrasound, MRI and digital breast tomosynthesis. Heterogeneity was high between studies and extensive variation in methodology, terminology and outcome measures was noted. This can lead to an overestimation of the diagnostic accuracy of DL algorithms on medical imaging. There is an immediate need for the development of artificial intelligence-specific EQUATOR guidelines, particularly STARD, in order to provide guidance around key issues in this field.

Preliminary study of the technical limitations of automated breast ultrasound: from procedure to diagnosis

Objective

To evaluate the main technical limitations of automated breast ultrasound and to determine the proportion of examinations excluded.

Materials and Methods

We evaluated 440 automated breast ultrasound examinations performed, over a 12-month period, by technicians using an established protocol.

Results

In five cases (1.1%), the examination was deemed unacceptable for diagnostic purposes, those examinations therefore being excluded.

Conclusion

Automated breast ultrasound is expected to overcome some of the major limitations of conventional ultrasound in breast cancer screening. In Brazil, this new method can be accepted for inclusion in routine clinical practice only after its advantages have been validated in the national context.

A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis

BACKGROUND

Deep learning offers considerable promise for medical diagnostics. We aimed to evaluate the diagnostic accuracy of deep learning algorithms versus health-care professionals in classifying diseases using medical imaging.

METHODS

In this systematic review and meta-analysis, we searched Ovid-MEDLINE, Embase, Science Citation Index, and Conference Proceedings Citation Index for studies published from Jan 1, 2012, to June 6, 2019. Studies comparing the diagnostic performance of deep learning models and health-care professionals based on medical imaging, for any disease, were included. We excluded studies that used medical waveform data graphics material or investigated the accuracy of image segmentation rather than disease classification. We extracted binary diagnostic accuracy data and constructed contingency tables to derive the outcomes of interest: sensitivity and specificity. Studies undertaking an out-of-sample external validation were included in a meta-analysis, using a unified hierarchical model. This study is registered with PROSPERO, CRD42018091176.

FINDINGS

Our search identified 31 587 studies, of which 82 (describing 147 patient cohorts) were included. 69 studies provided enough data to construct contingency tables, enabling calculation of test accuracy, with sensitivity ranging from 9·7% to 100·0% (mean 79·1%, SD 0·2) and specificity ranging from 38·9% to 100·0% (mean 88·3%, SD 0·1). An out-of-sample external validation was done in 25 studies, of which 14 made the comparison between deep learning models and health-care professionals in the same sample. Comparison of the performance between health-care professionals in these 14 studies, when restricting the analysis to the contingency table for each study reporting the highest accuracy, found a pooled sensitivity of 87·0% (95% CI 83·0-90·2) for deep learning models and 86·4% (79·9-91·0) for health-care professionals, and a pooled specificity of 92·5% (95% CI 85·1-96·4) for deep learning models and 90·5% (80·6-95·7) for health-care professionals.

INTERPRETATION

Our review found the diagnostic performance of deep learning models to be equivalent to that of health-care professionals. However, a major finding of the review is that few studies presented externally validated results or compared the performance of deep learning models and health-care professionals using the same sample. Additionally, poor reporting is prevalent in deep learning studies, which limits reliable interpretation of the reported diagnostic accuracy. New reporting standards that address specific challenges of deep learning could improve future studies, enabling greater confidence in the results of future evaluations of this promising technology.

FUNDING

None.

Development of a computer-aided tool for the pattern recognition of facial features in diagnosing Turner syndrome: comparison of diagnostic accuracy with clinical workers

Technologies applied for the recognition of facial features in diagnosing certain disorders seem to be promising in reducing the medical burden and improve the efficiency. This pilot study aimed to develop a computer-assisted tool for the pattern recognition of facial features for diagnosing Turner syndrome (TS). Photographs of 54 patients with TS and 158 female controls were collected from July 2016 to May 2017. Finally, photographs of 32 patients with TS and 96 age-matched controls were included in the study that were further divided equally into training and testing groups. The process of automatic classification consisted of image preprocessing, facial feature extraction, feature reduction and fusion, automatic classification, and result presentation. A total of 27 physicians and 21 medical students completed a web-based test including the same photographs used in computer testing. After training, the automatic facial classification system for diagnosing TS achieved a 68.8% sensitivity and 87.5% specificity (and a 67.6% average sensitivity and 87.9% average specificity after resampling), which was significantly higher than the average sensitivity (57.4%, P < 0.001) and specificity (75.4%, P < 0.001) of 48 participants, respectively. The accuracy of this system was satisfactory and better than the diagnosis by clinicians. However, the system necessitates further improvement for achieving a high diagnostic accuracy in clinical practice.

Comparison of three data mining models for prediction of advanced schistosomiasis prognosis in the Hubei province

BACKGROUND

In order to better assist medical professionals, this study aimed to develop and compare the performance of three models-a multivariate logistic regression (LR) model, an artificial neural network (ANN) model, and a decision tree (DT) model-to predict the prognosis of patients with advanced schistosomiasis residing in the Hubei province.

METHODOLOGY/PRINCIPAL FINDINGS

Schistosomiasis surveillance data were collected from a previous study based on a Hubei population sample including 4136 advanced schistosomiasis cases. The predictive models use LR, ANN, and DT methods. From each of the three groups, 70% of the cases (2896 cases) were used as training data for the predictive models. The remaining 30% of the cases (1240 cases) were used as validation groups for performance comparisons between the three models. Prediction performance was evaluated using area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and accuracy. Univariate analysis indicated that 16 risk factors were significantly associated with a patient's outcome of prognosis. In the training group, the mean AUC was 0.8276 for LR, 0.9267 for ANN, and 0.8229 for DT. In the validation group, the mean AUC was 0.8349 for LR, 0.8318 for ANN, and 0.8148 for DT. The three models yielded similar results in terms of accuracy, sensitivity, and specificity.

CONCLUSIONS/SIGNIFICANCE

Predictive models for advanced schistosomiasis prognosis, respectively using LR, ANN and DT models were proved to be effective approaches based on our dataset. The ANN model outperformed the LR and DT models in terms of AUC.

Classification of breast cancer in ultrasound imaging using a generic deep learning analysis software: a pilot study

OBJECTIVE

To train a generic deep learning software (DLS) to classify breast cancer on ultrasound images and to compare its performance to human readers with variable breast imaging experience.

METHODS

In this retrospective study, all breast ultrasound examinations from January 1, 2014 to December 31, 2014 at our institution were reviewed. Patients with post-surgical scars, initially indeterminate, or malignant lesions with histological diagnoses or 2-year follow-up were included. The DLS was trained with 70% of the images, and the remaining 30% were used to validate the performance. Three readers with variable expertise also evaluated the validation set (radiologist, resident, medical student). Diagnostic accuracy was assessed with a receiver operating characteristic analysis.

RESULTS

82 patients with malignant and 550 with benign lesions were included. Time needed for training was 7 min (DLS). Evaluation time for the test data set were 3.7 s (DLS) and 28, 22 and 25 min for human readers (decreasing experience). Receiver operating characteristic analysis revealed non-significant differences (p-values 0.45-0.47) in the area under the curve of 0.84 (DLS), 0.88 (experienced and intermediate readers) and 0.79 (inexperienced reader).

CONCLUSION

DLS may aid diagnosing cancer on breast ultrasound images with an accuracy comparable to radiologists, and learns better and faster than a human reader with no prior experience. Further clinical trials with dedicated algorithms are warranted. Advances in knowledge: DLS can be trained classify cancer on breast ultrasound images high accuracy even with comparably few training cases. The fast evaluation speed makes real-time image analysis feasible.

Logistic LASSO regression for the diagnosis of breast cancer using clinical demographic data and the BI-RADS lexicon for ultrasonography

Purpose

The aim of this study was to compare the performance of image analysis for predicting breast cancer using two distinct regression models and to evaluate the usefulness of incorporating clinical and demographic data (CDD) into the image analysis in order to improve the diagnosis of breast cancer.

Methods

This study included 139 solid masses from 139 patients who underwent a ultrasonography-guided core biopsy and had available CDD between June 2009 and April 2010. Three breast radiologists retrospectively reviewed 139 breast masses and described each lesion using the Breast Imaging Reporting and Data System (BI-RADS) lexicon. We applied and compared two regression methods-stepwise logistic (SL) regression and logistic least absolute shrinkage and selection operator (LASSO) regression-in which the BI-RADS descriptors and CDD were used as covariates. We investigated the performances of these regression methods and the agreement of radiologists in terms of test misclassification error and the area under the curve (AUC) of the tests.

Results

Logistic LASSO regression was superior (P<0.05) to SL regression, regardless of whether CDD was included in the covariates, in terms of test misclassification errors (0.234 vs. 0.253, without CDD; 0.196 vs. 0.258, with CDD) and AUC (0.785 vs. 0.759, without CDD; 0.873 vs. 0.735, with CDD). However, it was inferior (P<0.05) to the agreement of three radiologists in terms of test misclassification errors (0.234 vs. 0.168, without CDD; 0.196 vs. 0.088, with CDD) and the AUC without CDD (0.785 vs. 0.844, P<0.001), but was comparable to the AUC with CDD (0.873 vs. 0.880, P=0.141).

Conclusion

Logistic LASSO regression based on BI-RADS descriptors and CDD showed better performance than SL in predicting the presence of breast cancer. The use of CDD as a supplement to the BI-RADS descriptors significantly improved the prediction of breast cancer using logistic LASSO regression.

Comparison of the performance of log-logistic regression and artificial neural networks for predicting breast cancer relapse

BACKGROUND

Breast cancer is the most common cancers in female populations. The exact cause is not known, but is most likely to be a combination of genetic and environmental factors. Log-logistic model (LLM) is applied as a statistical method for predicting survival and it influencing factors. In recent decades, artificial neural network (ANN) models have been increasingly applied to predict survival data. The present research was conducted to compare log-logistic regression and artificial neural network models in prediction of breast cancer (BC) survival.

MATERIALS AND METHODS

A historical cohort study was established with 104 patients suffering from BC from 1997 to 2005. To compare the ANN and LLM in our setting, we used the estimated areas under the receiver-operating characteristic (ROC) curve (AUC) and integrated AUC (iAUC). The data were analyzed using R statistical software.

RESULTS

The AUC for the first, second and third years after diagnosis are 0.918, 0.780 and 0.800 in ANN, and 0.834, 0.733 and 0.616 in LLM, respectively. The mean AUC for ANN was statistically higher than that of the LLM (0.845 vs. 0.744). Hence, this study showed a significant difference between the performance in terms of prediction by ANN and LLM.

CONCLUSIONS

This study demonstrated that the ability of prediction with ANN was higher than with the LLM model. Thus, the use of ANN method for prediction of survival in field of breast cancer is suggested.

Comparison of Breast Cancer Screening Results in Korean Middle-Aged Women: A Hospital-based Prospective Cohort Study

Objectives

The aim of this hospital-based prospective study was to evaluate the diagnostic ability of breast cancer screening in Korean middle-aged women using age, ultrasonography, mammography, and magnification mammography, which are commonly used in most hospitals.

Methods

A total of 21 patents were examined using ultrasonography, mammography, and magnification mammography, and their data were prospectively analyzed from August 2011 to March 2013. All patients were divided into benign and malignant groups and the screening results were classified using the American College of Radiology Breast Imaging Reporting and Data System (BI-RADS). The final pathology report was used as the reference standard and the sensitivity and specificity of ultrasonography, mammography, and magnification mammography were evaluated using receiver-operating characteristics (ROC) analysis.

Results

The analysis included 21 patients who underwent biopsy. Among them, three (14.3%) were positive and 18 (85.7%) negative for breast cancer. The average age was 50.5 years (range = 38–61 years). The sensitivity was the same for ultrasonography and magnification mammography and the specificity of magnification mammography was higher than that of ultrasonography. The highest area under the ROC curve (AUC) was observed in the combination of age and magnification mammography (1.000) and the decreasing order of AUC in others was magnification mammography (0.833), ultrasonography (0.787), mammography (0.667), and age (0.648).

Conclusions

In Korean women, the diagnostic accuracy of magnification mammography was better than that of ultrasonography and mammography. The combination of age and magnification mammography increased the sensitivity and diagnostic accuracy.