Construction of joint confidence regions for the optimal true class fractions of Receiver Operating Characteristic (ROC) surfaces and manifolds

Estimation and inference on the partial volume under the receiver operating characteristic surface

measures of biomarker accuracy that employ the receiver operating characteristic surface have been proposed for biomarkers that classify patients into one of three groups: healthy, benign, or aggressive disease. The volume under the receiver operating characteristic surface summarizes the overall discriminatory ability of a biomarker in such configurations, but includes cutoffs associated with clinically irrelevant true classification rates. Due to the lethal nature of pancreatic cancer, cutoffs associated with a low true classification rate for identifying patients with pancreatic cancer may be undesirable and not appropriate for use in a clinical setting. In this project, we study the properties of a more focused criterion, the partial volume under the receiver operating characteristic surface, that summarizes the diagnostic accuracy of a marker in the three-class setting for regions restricted to only those of clinical interest. We propose methods for estimation and inference on the partial volume under the receiver operating characteristic surface under parametric and non-parametric frameworks and apply these methods to the evaluation of potential biomarkers for the diagnosis of pancreatic cancer.

Combining multiple biomarkers linearly to minimize the Euclidean distance of the closest point on the receiver operating characteristic surface to the perfection corner in trichotomous settings

The performance of individual biomarkers in discriminating between two groups, typically the healthy and the diseased, may be limited. Thus, there is interest in developing statistical methodologies for biomarker combinations with the aim of improving upon the individual discriminatory performance. There is extensive literature referring to biomarker combinations under the two-class setting. However, the corresponding literature under a three-class setting is limited. In our study, we provide parametric and nonparametric methods that allow investigators to optimally combine biomarkers that seek to discriminate between three classes by minimizing the Euclidean distance from the receiver operating characteristic surface to the perfection corner. Using this Euclidean distance as the objective function allows for estimation of the optimal combination coefficients along with the optimal cutoff values for the combined score. An advantage of the proposed methods is that they can accommodate biomarker data from all three groups simultaneously, as opposed to a pairwise analysis such as the one implied by the three-class Youden index. We illustrate that the derived true classification rates exhibit narrower confidence intervals than those derived from the Youden-based approach under a parametric, flexible parametric, and nonparametric kernel-based framework. We evaluate our approaches through extensive simulations and apply them to real data sets that refer to liver cancer patients.

Evaluating joint confidence region of hypervolume under ROC manifold and generalized Youden index

In biomarker evaluation/diagnostic studies, the hypervolume under the receiver operating characteristic manifold (HUM K $$ {\mathrm{HUM}}_K $$ ) and the generalized Youden index (J K $$ {J}_K $$ ) are the most popular measures for assessing classification accuracy under multiple classes. WhileHUM K $$ {\mathrm{HUM}}_K $$ is frequently used to evaluate the overall accuracy,J K $$ {J}_K $$ provides direct measure of accuracy at the optimal cut-points. Simultaneous evaluation ofHUM K $$ {\mathrm{HUM}}_K $$ andJ K $$ {J}_K $$ provides a comprehensive picture about the classification accuracy of the biomarker/diagnostic test under consideration. This article studies both parametric and non-parametric approaches for estimating the confidence region ofHUM K $$ {\mathrm{HUM}}_K $$ andJ K $$ {J}_K $$ for a single biomarker. The performances of the proposed methods are investigated by an extensive simulation study and are applied to a real data set from the Alzheimer's Disease Neuroimaging Initiative.

A new accuracy metric under three classes when subclasses are involved and its confidence interval estimation

"Compound multi-class classification" refers to the setting where three or more main classes are involved and at least one of the main classes have multiple subclasses. A common practice in evaluating biomarker performance under "compound multi-class classification" is "subclasses pooling." In this article, we first explore the downsides of accuracy metrics based on pooled data. Then we propose a new accuracy measure proper for "compound multi-class classification" with three ordinal main classes, namely "volume under compoundR O C $$ ROC $$ surface (V U S C $$ VU{S}_C $$ )." The proposedV U S C $$ VU{S}_C $$ evaluates the accuracy of a biomarker appropriately by identifying main classes without requiring specification of an ordering for marker values of subclasses within each main class. For confidence interval estimation ofV U S C $$ VU{S}_C $$ , both parametric and nonparametric methods are studied, and simulation studies are carried out to assess coverage probabilities. A subset of Alzheimer's Disease Neuroimaging Initiative study dataset is analyzed.

Construction of joint confidence spaces for the optimal true class fraction triplet in the ROC space using alternative biomarker cutoffs

Hepatocellular carcinoma (HCC) is the most common primary cancer of the liver. Finding new biomarkers for its early detection is of high clinical importance. As with many other diseases, cancer has a progressive nature. In cancer biomarker studies, it is often the case that the true disease status of the recruited individuals exhibits more than two classes. The receiver operating characteristic (ROC) surface is a well-known statistical tool for assessing the biomarkers' discriminatory ability in trichotomous settings. The volume under the ROC surface (VUS) is an overall measure of the discriminatory ability of a marker. In practice, clinicians are often in need of cutoffs for decision-making purposes. A popular approach for computing such cutoffs is the Youden index and its recent three-class generalization. A drawback of such a method is that it treats the data in a pairwise fashion rather than consider all the data simultaneously. The use of the minimized Euclidean distance from the perfection corner to the ROC surface (also known as closest to perfection method) is an alternative to the Youden index that may be preferable in some settings. When such a method is employed, there is a need for inferences around the resulting true class rates/fractions that correspond to the optimal operating point. In this paper, we provide an inferential framework for the derivation of marginal confidence intervals (CIs) and joint confidence spaces (CSs) around the corresponding true class fractions, when dealing with trichotomous settings. We explore parametric and nonparametric approaches for the construction of such CIs and CSs. We evaluate our approaches through extensive simulations and apply them to a real data set that refers to HCC patients.

Rheostat functional outcomes occur when substitutions are introduced at nonconserved positions that diverge with speciation

When amino acids vary during evolution, the outcome can be functionally neutral or biologically-important. We previously found that substituting a subset of nonconserved positions, "rheostat" positions, can have surprising effects on protein function. Since changes at rheostat positions can facilitate functional evolution or cause disease, more examples are needed to understand their unique biophysical characteristics. Here, we explored whether "phylogenetic" patterns of change in multiple sequence alignments (such as positions with subfamily specific conservation) predict the locations of functional rheostat positions. To that end, we experimentally tested eight phylogenetic positions in human liver pyruvate kinase (hLPYK), using 10-15 substitutions per position and biochemical assays that yielded five functional parameters. Five positions were strongly rheostatic and three were non-neutral. To test the corollary that positions with low phylogenetic scores were not rheostat positions, we combined these phylogenetic positions with previously-identified hLPYK rheostat, "toggle" (most substitution abolished function), and "neutral" (all substitutions were like wild-type) positions. Despite representing 428 variants, this set of 33 positions was poorly statistically powered. Thus, we turned to the in vivo phenotypic dataset for E. coli lactose repressor protein (LacI), which comprised 12-13 substitutions at 329 positions and could be used to identify rheostat, toggle, and neutral positions. Combined hLPYK and LacI results show that positions with strong phylogenetic patterns of change are more likely to exhibit rheostat substitution outcomes than neutral or toggle outcomes. Furthermore, phylogenetic patterns were more successful at identifying rheostat positions than were co-evolutionary or eigenvector centrality measures of evolutionary change.

Estimation and construction of confidence intervals for biomarker cutoff-points under the shortest Euclidean distance from the ROC surface to the perfection corner

Pancreatic ductal adenocarcinoma (PDAC) is an aggressive type of cancer with a 5-year survival rate of less than 5%. As in many other diseases, its diagnosis might involve progressive stages. It is common that in biomarker studies referring to PDAC, recruitment involves three groups: healthy individuals, patients that suffer from chronic pancreatitis, and PDAC patients. Early detection and accurate classification of the state of the disease are crucial for patients' successful treatment. ROC analysis is the most popular way to evaluate the performance of a biomarker and the Youden index is commonly employed for cutoff derivation. The so-called generalized Youden index has a drawback in the three-class case of not accommodating the full data set when estimating the optimal cutoffs. In this article, we explore the use of the Euclidean distance of the ROC to the perfection corner for the derivation of cutoffs in trichotomous settings. We construct an inferential framework that involves both parametric and nonparametric techniques. Our methods can accommodate the full information of a given data set and thus provide more accurate estimates in terms of the decision-making cutoffs compared with a Youden-based strategy. We evaluate our approaches through extensive simulations and illustrate them on a PDAC biomarker study.

Estimating diagnostic accuracy for clustered ordinal diagnostic groups in the three-class case-Application to the early diagnosis of Alzheimer disease

Many medical diagnostic studies involve three ordinal diagnostic populations in which the diagnostic accuracy can be summarized by the volume or partial volume under the receiver operating characteristic surface for a diagnostic marker. When the diagnostic populations are clustered, e.g. by families, we propose to model the diagnostic marker by a general linear mixed model that takes into account of the correlation on the diagnostic marker from members of the same clusters. This model then facilitates the maximum likelihood estimation and statistical inferences of the diagnostic accuracy for the diagnostic marker. This approach naturally allows the incorporation of covariates as well as missing data when some clusters do not have subjects on all diagnostic groups in the estimation of, and the subsequent inferences on the diagnostic accuracy. We further study the performance of the proposed methods in a large simulation study with clustered data. Finally, we apply the proposed methodology to the data of several biomarkers collected by the Dominantly Inherited Alzheimer Network, an international family-clustered registry to study autosomal dominant Alzheimer disease which is a rare form of Alzheimer disease caused by mutations in any of the three genes including the amyloid precursor protein, presenilin 1 and presenilin 2. We estimate the accuracy of several cerebrospinal fluid and neuroimaging biomarkers in differentiating three diagnostic and genetic populations: normal non-mutation carriers, asymptomatic mutation carriers, and symptomatic mutation carriers.

Bayesian nonparametric inference for the three-class Youden index and its associated optimal cutoff points

The three-class Youden index serves both as a measure of medical test accuracy and a criterion to choose the optimal pair of cutoff values for classifying subjects into three ordinal disease categories (e.g. no disease, mild disease, advanced disease). We present a Bayesian nonparametric approach for estimating the three-class Youden index and its corresponding optimal cutoff values based on Dirichlet process mixtures, which are robust models that can handle intricate features of distributions for complex data. Results from a simulation study are presented and an application to data from the Trail Making Test to assess cognitive impairment in Parkinson's disease patients is detailed.

Confidence intervals for differences between volumes under receiver operating characteristic surfaces (VUS) and generalized Youden indices (GYIs)

This article explores both existing and new methods for the construction of confidence intervals for differences of indices of diagnostic accuracy of competing pairs of biomarkers in three-class classification problems and fills the methodological gaps for both parametric and non-parametric approaches in the receiver operating characteristic surface framework. The most widely used such indices are the volume under the receiver operating characteristic surface and the generalized Youden index. We describe implementation of all methods and offer insight regarding the appropriateness of their use through a large simulation study with different distributional and sample size scenarios. Methods are illustrated using data from the Alzheimer's Disease Neuroimaging Initiative study, where assessment of cognitive function naturally results in a three-class classification setting.