Coverage-adjusted Confidence Intervals for a Binomial Proportion

A fiducial-based confidence interval for the linear combination of multinomial probabilities

Across a broad set of applications, system outcomes may be summarized as probabilities in confusion or contingency tables. In settings with more than two outcomes (e.g., stages of cancer), these outcomes represent multinomial experiments. Measures to summarize system performance have been presented as linear combinations of the resulting multinomial probabilities. Statistical inference on the linear combination of multinomial probabilities has been focused on large-sample and parametric settings and not small-sample settings. Such inference is valuable, however, especially in settings such as those resulting from pilot or low-cost studies. To address this gap, we leverage the fiducial approach to derive confidence intervals around the linear combination of multinomial parameters with desirable frequentist properties. One of the original arguments against the fiducial approach was its inability to extend to multiparameter settings. Therefore, the great novelty of this work is both the derived interval and the logical framework for applying the fiducial approach in multiparameter settings. Through simulation, we demonstrate that the proposed method maintains a minimum coverage of1 - α $1 - \alpha$ , unlike the bootstrap and large-sample methods, at comparable interval lengths. Finally, we illustrate its use in a medical problem of selecting classifiers for diagnosing chronic allograph nephropathy in postkidney transplant patients.

A nonparametric fiducial interval for the Youden index in multi-state diagnostic settings

The Youden index is a commonly employed metric to characterize the performance of a diagnostic test at its optimal point. For tests with three or more outcome classes, the Youden index has been extended; however, there are limited methods to compute a confidence interval (CI) about its value. Often, outcome classes are assumed to be normally distributed, which facilitates computational formulas for the CI bounds; however, many scenarios exist for which these assumptions cannot be made. In addition, many of these existing CI methods do not work well for small sample sizes. We propose a method to compute a nonparametric interval about the Youden index utilizing the fiducial argument. This fiducial interval ensures that CI coverage is met regardless of sample size, underlying distributional assumptions, or use of a complex classifier for diagnosis. Two alternate fiducial intervals are also considered. A simulation was conducted, which demonstrates the coverage and interval length for the proposed methods. Comparisons were made using no distributional assumptions on the outcome classes and for when outcomes were assumed to be normally distributed. In general, coverage probability was consistently met, and interval length was reasonable. The proposed fiducial method was also demonstrated in data examining biomarkers in subjects to predict diagnostic stages ranging from normal kidney function to chronic allograph nephropathy. Published 2015. This article is a U.S. Government work and is in the public domain in the USA.

Randomization inference for treatment effects on a binary outcome

Two methods are developed for constructing randomization-based confidence sets for the average effect of a treatment on a binary outcome. The methods are nonparametric and require no assumptions about random sampling from a larger population. Both of the resulting 1 - α confidence sets are exact in the sense that the probability of containing the true treatment effect is at least 1 - α. Both types of confidence sets are also guaranteed to have width no greater than one. In contrast, a previously proposed asymptotic confidence interval is not exact and may have width greater than 1. The first approach combines Bonferroni-adjusted prediction sets for the attributable effects in the treated and untreated. The second method entails inverting a permutation test. Simulations are presented comparing the two randomization-based confidence sets with the asymptotic interval as well as the standard Wald confidence interval and a commonly used exact interval for the difference in binomial proportions. Results show for small to moderate sample sizes that the permutation confidence set attains the narrowest width on average among the methods that maintain nominal coverage. Extensions that allow for stratifying on categorical baseline covariates are also discussed.