Comparing Performance of Multiclass Classification Systems with ROC Manifolds: When Volume and Correct Classification Fails

A nonparametric fiducial interval for the Youden index in multi-state diagnostic settings

The Youden index is a commonly employed metric to characterize the performance of a diagnostic test at its optimal point. For tests with three or more outcome classes, the Youden index has been extended; however, there are limited methods to compute a confidence interval (CI) about its value. Often, outcome classes are assumed to be normally distributed, which facilitates computational formulas for the CI bounds; however, many scenarios exist for which these assumptions cannot be made. In addition, many of these existing CI methods do not work well for small sample sizes. We propose a method to compute a nonparametric interval about the Youden index utilizing the fiducial argument. This fiducial interval ensures that CI coverage is met regardless of sample size, underlying distributional assumptions, or use of a complex classifier for diagnosis. Two alternate fiducial intervals are also considered. A simulation was conducted, which demonstrates the coverage and interval length for the proposed methods. Comparisons were made using no distributional assumptions on the outcome classes and for when outcomes were assumed to be normally distributed. In general, coverage probability was consistently met, and interval length was reasonable. The proposed fiducial method was also demonstrated in data examining biomarkers in subjects to predict diagnostic stages ranging from normal kidney function to chronic allograph nephropathy. Published 2015. This article is a U.S. Government work and is in the public domain in the USA.