Receiver operating characteristic (ROC) curves: equivalences, beta model, and minimum distance estimation

Restricting datasets to classifiable samples augments discovery of immune disease biomarkers

Immunological diseases are typically heterogeneous in clinical presentation, severity and response to therapy. Biomarkers of immune diseases often reflect this variability, especially compared to their regulated behaviour in health. This leads to a common difficulty that frustrates biomarker discovery and interpretation - namely, unequal dispersion of immune disease biomarker expression between patient classes necessarily limits a biomarker's informative range. To solve this problem, we introduce dataset restriction, a procedure that splits datasets into classifiable and unclassifiable samples. Applied to synthetic flow cytometry data, restriction identifies biomarkers that are otherwise disregarded. In advanced melanoma, restriction finds biomarkers of immune-related adverse event risk after immunotherapy and enables us to build multivariate models that accurately predict immunotherapy-related hepatitis. Hence, dataset restriction augments discovery of immune disease biomarkers, increases predictive certainty for classifiable samples and improves multivariate models incorporating biomarkers with a limited informative range. This principle can be directly extended to any classification task.

Competitive performance as a discriminator of doping status in elite athletes

As the aim of any doping regime is to improve sporting performance, it has been suggested that analysis of athlete competitive results might be informative in identifying those at greater risk of doping. This research study aimed to investigate the utility of a statistical performance model to discriminate between athletes who have a previous anti-doping rule violation (ADRV) and those who do not. We analysed performances of male and female 100 and 800 m runners obtained from the World Athletics database using a Bayesian spline model. Measures of unusual improvement in performance were quantified by comparing the yearly change in athlete's performance (delta excess performance) to quantiles of performance in their age-matched peers from the database population. The discriminative ability of these measures was investigated using the area under the ROC curve (AUC) with the 55%, 75% and 90% quantiles of the population performance. The highest AUC values across age were identified for the model with a 75% quantile (AUC = 0.78-0.80). The results of this study demonstrate that delta excess performance was able to discriminate between athletes with and without ADRVs and therefore could be used to assist in the risk stratification of athletes for anti-doping purposes.

ROC Curves, Loss Functions, and Distorted Probabilities in Binary Classification

The main purpose of this work is to study how loss functions in machine learning influence the “binary machines”, i.e., probabilistic AI models for predicting binary classification problems. In particular, we show the following results: (i) Different measures of accuracy such as area under the curve (AUC) of the ROC curve, the maximal balanced accuracy, and the maximally weighted accuracy are topologically equivalent, with natural inequalities relating them; (ii) the so-called real probability machines with respect to given information spaces are the optimal machines, i.e., they have the highest precision among all possible machines, and moreover, their ROC curves are automatically convex; (iii) the cross-entropy and the square loss are the most natural loss functions in the sense that the real probability machine is their minimizer; (iv) an arbitrary strictly convex loss function will also have as its minimizer an optimal machine, which is related to the real probability machine by just a reparametrization of sigmoid values; however, if the loss function is not convex, then its minimizer is not an optimal machine, and strange phenomena may happen.

Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)

Throughout science and technology, receiver operating characteristic (ROC) curves and associated area under the curve ($$\mathrm{AUC}$$ AUC ) measures constitute powerful tools for assessing the predictive abilities of features, markers and tests in binary classification problems. Despite its immense popularity, ROC analysis has been subject to a fundamental restriction, in that it applies to dichotomous (yes or no) outcomes only. Here we introduce ROC movies and universal ROC (UROC) curves that apply to just any linearly ordered outcome, along with an associated coefficient of predictive ability ($${\mathrm{CPA}}$$ CPA ) measure. $${\mathrm{CPA}}$$ CPA equals the area under the UROC curve, and admits appealing interpretations in terms of probabilities and rank based covariances. For binary outcomes $${\mathrm{CPA}}$$ CPA equals $$\mathrm{AUC}$$ AUC , and for pairwise distinct outcomes $${\mathrm{CPA}}$$ CPA relates linearly to Spearman’s coefficient, in the same way that the C index relates linearly to Kendall’s coefficient. ROC movies, UROC curves, and $${\mathrm{CPA}}$$ CPA nest and generalize the tools of classical ROC analysis, and are bound to supersede them in a wealth of applications. Their usage is illustrated in data examples from biomedicine and meteorology, where rank based measures yield new insights in the WeatherBench comparison of the predictive performance of convolutional neural networks and physical-numerical models for weather prediction.