Indexes for three-class classification performance assessment--an empirical comparison

Comparing multi-class classifier performance by multi-class ROC analysis: A nonparametric approach

The area under the Receiver Operating Characteristic (ROC) curve (AUC) is a standard metric for quantifying and comparing binary classifiers. Real world applications often require classification into multiple (more than two) classes. For multi-class classifiers that produce class membership scores, a popular multi-class AUC (MAUC) variant is to average the pairwise AUC values [1]. Due to the complicated correlation patterns, the variance of MAUC is often estimated numerically using resampling techniques. This work is a generalization of DeLong's nonparameteric approach for binary AUC analysis [2] to MAUC. We first derive the closed-form expression of the covariance matrix of the pairwise AUCs within a single MAUC. Then by dropping higher order terms, we obtain an approximate covariance matrix with a compact, matrix factorization form, which then serves as the basis for variance estimation of a single MAUC. We further extend this approach to estimate the covariance of correlated MAUCs that arise from multiple competing classifiers. For the special case of binary correlated AUCs, our results coincide with that of DeLong. Our numerical studies confirm the accuracy of the variance and covariance estimates. We provide the source code of the proposed covariance estimation of correlated MAUCs on GitHub (https://tinyurl.com/euj6wvsz) for its easy adoption by machine learning and statistical analysis packages to quantify and compare multi-class classifiers.

Carotid artery ultrasound image analysis: A review of the literature

Stroke is one of the prominent causes of death in the recent days. The existence of susceptible plaque in the carotid artery can be used in ascertaining the possibilities of cardiovascular diseases and long-term disabilities. The imaging modality used for early screening of the disease is B-mode ultrasound image of the person in the artery area. The objective of this article is to give a widespread review of the imaging modes and methods used for studying the carotid artery for identifying stroke, atherosclerosis and related cardiovascular diseases. We encompass the review in methods used for artery wall tracking, intima-media, and lumen segmentation which will help in finding the extent of the disease. Due to the characteristics of the imaging modality used, the images have speckle noise which worsens the image quality. Adaptive homomorphic filtering with wavelet and contourlet transforms, Levy Shrink, gamma distribution were used for image denoising. Learning-based neural network approaches for denoising give better edge preservation. Domain knowledge-based segmentation approaches have proved to provide more accurate intima-media thickness measurements. There is a requirement of useful fully automatic segmentation approaches, 3D, 4D systems, and plaque motion analysis. Taking into consideration the image priors like geometry, imaging physics, intensity and temporal data, image analysis has to be performed. Encouragingly more research has focused on content-specific segmentation and classification techniques. With the evaluation of machine learning algorithms, classifying the image as with or without a fat deposit has gained better accuracy and sensitivity. Machine learning-based approaches like self-organizing map, k-nearest neighborhood and support vector machine achieve promising accuracy and sensitivity in classification. The literature reveals that there is more scope in identifying a patient-specific model in a fully automatic manner.

A comprehensive and comparative review of optimal cut-points selection methods for diseases with multiple ordinal stages

Cut-points selection is a key topic in the field of diagnostic studies. For binary classification, there exist several well-developed methods, some of which have been extended to three-class settings and beyond. This paper focuses on optimal cut-points selection methods for diseases with multiple ordinal stages. The purpose of this paper is two-fold: 1) to propose three new cut-points selection methods; and 2) to present a comprehensive simulation study to assess and compare the performance of all the available methods. Two real data sets, one from ovarian cancer and the other from pancreatic cancer, are analyzed.

Weighted Polynomial Approximation for Automated Detection of Inspiratory Flow Limitation

Inspiratory flow limitation (IFL) is a critical symptom of sleep breathing disorders. A characteristic flattened flow-time curve indicates the presence of highest resistance flow limitation. This study involved investigating a real-time algorithm for detecting IFL during sleep. Three categories of inspiratory flow shape were collected from previous studies for use as a development set. Of these, 16 cases were labeled as non-IFL and 78 as IFL which were further categorized into minor level (20 cases) and severe level (58 cases) of obstruction. In this study, algorithms using polynomial functions were proposed for extracting the features of IFL. Methods using first- to third-order polynomial approximations were applied to calculate the fitting curve to obtain the mean absolute error. The proposed algorithm is described by the weighted third-order (w.3rd-order) polynomial function. For validation, a total of 1,093 inspiratory breaths were acquired as a test set. The accuracy levels of the classifications produced by the presented feature detection methods were analyzed, and the performance levels were compared using a misclassification cobweb. According to the results, the algorithm using the w.3rd-order polynomial approximation achieved an accuracy of 94.14% for IFL classification. We concluded that this algorithm achieved effective automatic IFL detection during sleep.

Diagnostic thresholds with three ordinal groups

In practice, there exist many disease processes with three ordinal disease classes; for example, in the detection of Alzheimer's disease (AD) a patient can be classified as healthy (disease-free stage), mild cognitive impairment (early disease stage), or AD (full disease stage). The treatment interventions and effectiveness of such disease processes will depend on the disease stage. Therefore, it is important to develop diagnostic tests with the ability to discriminate between the three disease stages. Measuring the overall ability of diagnostic tests to discriminate between the three classes has been discussed extensively in the literature. However, there has been little proposed on how to select clinically meaningful thresholds for such diagnostic tests, except for a method based on the generalized Youden index by Nakas et al. (2010). In this article, we propose two new criteria for selecting diagnostic thresholds in the three-class setting. The numerical study demonstrated that the proposed methods may provide thresholds with less variability and more balance among the correct classification rates for the three stages. The proposed methods are applied to two real examples: the clinical diagnosis of AD from the Washington University Alzheimer's Disease Research Center and the detection of liver cancer (LC) using protein segments.

Cognition Network Technology prototype of a CAD system for mammography to assist radiologists by finding similar cases in a reference database

PURPOSE

We present a new approach for computer-aided detection and diagnosis in mammography based on Cognition Network Technology (CNT). Originally designed for image processing, CNT has been extended to also perform context- and knowledge-driven analysis of tabular data. For the first time using this technology, an application was created and evaluated for fully automatic searching of patient cases from a reference database of verified findings. The application aims to support radiologists in providing cases of similarity and relevance to a given query case. It adopts an extensible and knowledge-driven concept as a similarity measure.

METHODS

As a preprocessing step, all input images from more than 400 patients were fully automatically segmented and the resulting objects classified--this includes the complete breast shape, the position of the mammilla, the pectoral muscle, and various potential candidate objects for suspicious mass lesions. For the similarity search, collections of object properties and metadata from many patients were combined into a single table analysis project. Extended CNT allows for a convenient implementation of knowledge-based structures, for example, by meaningfully linking detected objects in different breast views that might represent identical lesions. Objects from alternative segmentation methods are also be considered, so as to collectively become a sufficient set of base-objects for identifying suspicious mass lesions.

RESULTS

For 80% of 112 patient cases with suspicious lesions, the system correctly identified at least one corresponding mass lesion as an object of interest. In this database, consisting of 1,024 images from a total of 303 patients, an average of 0.66 false-positive objects per image were detected. An additional testing database contained 480 images from 120 patients, 15 of whom were annotated with suspicious mass lesions. Here, 47% (7 out of 15) of these were detected automatically with 1.13 false-positive objects per image. A diagnosis is predicted for each patient case by applying a majority vote from the reference findings of the ten most similar cases. Two separate evaluation scenarios suggest a fraction of correct predictions of respectively 79 and 76%.

CONCLUSION

Cognition Network Technology was extended to process table data, making it possible to access and relate records from different images and non-image sources, such as demographic patient data or parameters from clinical examinations. A prototypal application enables efficient searching of a patient and image database for similar patient cases. Using concepts of knowledge-driven configuration and flexible extension, the application illustrates a path to a new generation of future CAD systems.