A comprehensive and comparative review of optimal cut-points selection methods for diseases with multiple ordinal stages

Predictors of membrane oxygenator failure in pediatric extracorporeal membrane oxygenation

BACKGROUND

Veno-arterial extracorporeal membrane oxygenation (V-A ECMO) is increasingly utilized in pediatric patients. Failure to recognize membrane oxygenator failure can lead to critical complications due to rapid deterioration of membrane oxygenator function. Therefore, identifying the predictors for membrane oxygenator exchange is crucial. However, risk factors for membrane oxygenator exchange in pediatric V-A ECMO remain unclear; therefore, this study aimed to evaluate these risk factors.

METHODS

This retrospective cohort study enrolled all pediatric patients aged <18 years who received V-A ECMO between August 2018 and July 2023 at a tertiary-care pediatric hospital in Japan. The Cox proportional hazards model was used to evaluate the predictors of membrane oxygenator failure within 72 h after initiation.

RESULTS

During the study period, membrane oxygenator failure occurred in 18/55 (32.7%) children within 72 h; membrane oxygenator failure within 72 h occurred in 4/29 (13.8%) and 14/26 (53.8%) in the groups with ratio of blood flow divided by the blood flow limit of the membrane oxygenator (B/L) of <0.5 and ≥0.5, respectively (adjusted hazards ratio, 4.97 [95% confidence interval, 1.33-18.5]; p = 0.017). After adjusting for delta pressure of the oxygenator, an increase in body weight and aspartate aminotransferase levels were associated with an increase in early membrane oxygenator failure.

CONCLUSIONS

This retrospective study demonstrated that a B/L ratio >0.5, an increase in body weight, and elevated aspartate aminotransferase were independent risk factors for early membrane oxygenator failure in pediatric V-A ECMO. However, a prospective multicenter study with an appropriate sample size is warranted to mitigate potential bias, and enhance generalizability for further investigation of the association between a B/L ratio and early membrane oxygenator failure.

Optimal Cut-Point Selection Methods Under Binary Classification When Subclasses Are Involved

In practice, we often encounter binary classification problems where both main classes consist of multiple subclasses. For example, in an ovarian cancer study where biomarkers were evaluated for their accuracy of distinguishing noncancer cases from cancer cases, the noncancer class consists of healthy subjects and benign cases, while the cancer class consists of subjects at both early and late stages. This article aims to provide a large number of optimal cut-point selection methods for such setting. Furthermore, we also study confidence interval estimation of the optimal cut-points. Simulation studies are carried out to explore the performance of the proposed cut-point selection methods as well as confidence interval estimation methods. A real ovarian cancer data set is analyzed using the proposed methods.

Combining multiple biomarkers linearly to minimize the Euclidean distance of the closest point on the receiver operating characteristic surface to the perfection corner in trichotomous settings

The performance of individual biomarkers in discriminating between two groups, typically the healthy and the diseased, may be limited. Thus, there is interest in developing statistical methodologies for biomarker combinations with the aim of improving upon the individual discriminatory performance. There is extensive literature referring to biomarker combinations under the two-class setting. However, the corresponding literature under a three-class setting is limited. In our study, we provide parametric and nonparametric methods that allow investigators to optimally combine biomarkers that seek to discriminate between three classes by minimizing the Euclidean distance from the receiver operating characteristic surface to the perfection corner. Using this Euclidean distance as the objective function allows for estimation of the optimal combination coefficients along with the optimal cutoff values for the combined score. An advantage of the proposed methods is that they can accommodate biomarker data from all three groups simultaneously, as opposed to a pairwise analysis such as the one implied by the three-class Youden index. We illustrate that the derived true classification rates exhibit narrower confidence intervals than those derived from the Youden-based approach under a parametric, flexible parametric, and nonparametric kernel-based framework. We evaluate our approaches through extensive simulations and apply them to real data sets that refer to liver cancer patients.

Evaluating joint confidence region of hypervolume under ROC manifold and generalized Youden index

In biomarker evaluation/diagnostic studies, the hypervolume under the receiver operating characteristic manifold (HUM K $$ {\mathrm{HUM}}_K $$ ) and the generalized Youden index (J K $$ {J}_K $$ ) are the most popular measures for assessing classification accuracy under multiple classes. WhileHUM K $$ {\mathrm{HUM}}_K $$ is frequently used to evaluate the overall accuracy,J K $$ {J}_K $$ provides direct measure of accuracy at the optimal cut-points. Simultaneous evaluation ofHUM K $$ {\mathrm{HUM}}_K $$ andJ K $$ {J}_K $$ provides a comprehensive picture about the classification accuracy of the biomarker/diagnostic test under consideration. This article studies both parametric and non-parametric approaches for estimating the confidence region ofHUM K $$ {\mathrm{HUM}}_K $$ andJ K $$ {J}_K $$ for a single biomarker. The performances of the proposed methods are investigated by an extensive simulation study and are applied to a real data set from the Alzheimer's Disease Neuroimaging Initiative.

On optimal biomarker cutoffs accounting for misclassification costs in diagnostic trilemmas with applications to pancreatic cancer

Pancreatic ductal adenocarcinoma (PDAC) is the most deadly cancer and currently there is strong clinical interest in novel biomarkers that contribute to its early detection. Assessing appropriately the accuracy of such biomarkers is a crucial issue and often one needs to take into account that many assays include biospecimens of individuals coming from three groups: healthy, chronic pancreatitis, and PDAC. The ROC surface is an appropriate tool for assessing the overall accuracy of a marker employed under such trichotomous settings. A decision/classification rule is often based on the so-called Youden index and its three-dimensional generalization. However, both the clinical and the statistical literature have not paid the necessary attention to the underlying false classification (FC) rates that are of equal or even greater importance. In this article we provide a framework to make inferences around all classification rates as well as comparisons. We explore the trinormal model, flexible models based on power transformations, and robust non-parametric alternatives. We provide a full framework for the construction of confidence intervals, regions, and spaces for joint inferences or for clinically meaningful points of interest. We further discuss the implications of costs related to different FCs. We evaluate our approaches through extensive simulations and illustrate them using data from a recent PDAC study conducted at the MD Anderson Cancer Center.

Construction of joint confidence spaces for the optimal true class fraction triplet in the ROC space using alternative biomarker cutoffs

Hepatocellular carcinoma (HCC) is the most common primary cancer of the liver. Finding new biomarkers for its early detection is of high clinical importance. As with many other diseases, cancer has a progressive nature. In cancer biomarker studies, it is often the case that the true disease status of the recruited individuals exhibits more than two classes. The receiver operating characteristic (ROC) surface is a well-known statistical tool for assessing the biomarkers' discriminatory ability in trichotomous settings. The volume under the ROC surface (VUS) is an overall measure of the discriminatory ability of a marker. In practice, clinicians are often in need of cutoffs for decision-making purposes. A popular approach for computing such cutoffs is the Youden index and its recent three-class generalization. A drawback of such a method is that it treats the data in a pairwise fashion rather than consider all the data simultaneously. The use of the minimized Euclidean distance from the perfection corner to the ROC surface (also known as closest to perfection method) is an alternative to the Youden index that may be preferable in some settings. When such a method is employed, there is a need for inferences around the resulting true class rates/fractions that correspond to the optimal operating point. In this paper, we provide an inferential framework for the derivation of marginal confidence intervals (CIs) and joint confidence spaces (CSs) around the corresponding true class fractions, when dealing with trichotomous settings. We explore parametric and nonparametric approaches for the construction of such CIs and CSs. We evaluate our approaches through extensive simulations and apply them to a real data set that refers to HCC patients.

Receiver operating characteristic estimation and threshold selection criteria in three-class classification problems for clustered data

Statistical evaluation of diagnostic tests, and, more generally, of biomarkers, is a constantly developing field, in which complexity of the assessment increases with the complexity of the design under which data are collected. One particularly prevalent type of data is clustered data, where individual units are naturally nested into clusters. In these cases, Bias can arise from omission, in the evaluation process, of cluster-level effects and/or individual covariates. Focusing on the three-class case and for continuous-valued diagnostic tests, we investigate how to exploit the clustered structure of data within a linear-mixed model approach, both when the assumption of normality holds and when it does not. We provide a method for the estimation of covariate-specific receiver operating characteristic surfaces and discuss methods for the choice of optimal thresholds, proposing three possible estimators. A proof of consistency and asymptotic normality of the proposed threshold estimators is given. All considered methods are evaluated by extensive simulation experiments. As an application, we study the use of the Lysosomal Associated Membrane Protein Family Member 5 gene expression as a biomarker to distinguish among three types of glutamatergic neurons.

Estimation and construction of confidence intervals for biomarker cutoff-points under the shortest Euclidean distance from the ROC surface to the perfection corner

Pancreatic ductal adenocarcinoma (PDAC) is an aggressive type of cancer with a 5-year survival rate of less than 5%. As in many other diseases, its diagnosis might involve progressive stages. It is common that in biomarker studies referring to PDAC, recruitment involves three groups: healthy individuals, patients that suffer from chronic pancreatitis, and PDAC patients. Early detection and accurate classification of the state of the disease are crucial for patients' successful treatment. ROC analysis is the most popular way to evaluate the performance of a biomarker and the Youden index is commonly employed for cutoff derivation. The so-called generalized Youden index has a drawback in the three-class case of not accommodating the full data set when estimating the optimal cutoffs. In this article, we explore the use of the Euclidean distance of the ROC to the perfection corner for the derivation of cutoffs in trichotomous settings. We construct an inferential framework that involves both parametric and nonparametric techniques. Our methods can accommodate the full information of a given data set and thus provide more accurate estimates in terms of the decision-making cutoffs compared with a Youden-based strategy. We evaluate our approaches through extensive simulations and illustrate them on a PDAC biomarker study.

Joint confidence region estimation on predictive values

For evaluating diagnostic accuracy of inherently continuous diagnostic tests/biomarkers, sensitivity and specificity are well-known measures both of which depend on a diagnostic cut-off, which is usually estimated. Sensitivity (specificity) is the conditional probability of testing positive (negative) given the true disease status. However, a more relevant question is "what is the probability of having (not having) a disease if a test is positive (negative)?". Such post-test probabilities are denoted as positive predictive value (PPV) and negative predictive value (NPV). The PPV and NPV at the same estimated cut-off are correlated, hence it is desirable to make the joint inference on PPV and NPV to account for such correlation. Existing inference methods for PPV and NPV focus on the individual confidence intervals and they were developed under binomial distribution assuming binary instead of continuous test results. Several approaches are proposed to estimate the joint confidence region as well as the individual confidence intervals of PPV and NPV. Simulation results indicate the proposed approaches perform well with satisfactory coverage probabilities for normal and non-normal data and, additionally, outperform existing methods with improved coverage as well as narrower confidence intervals for PPV and NPV. The Alzheimer's Disease Neuroimaging Initiative (ADNI) data set is used to illustrate the proposed approaches and compare them with the existing methods.

Combining multiple biomarkers to linearly maximize the diagnostic accuracy under ordered multi-class setting

Either in clinical study or biomedical research, it is a common practice to combine multiple biomarkers to improve the overall diagnostic performance. Despite the fact there exist a large number of statistical methods for biomarker combination under binary classification, research on this topic under multi-class setting is sparse. The overall diagnostic accuracy, i.e. the sum of correct classification rates, directly measures the classification accuracy of the combined biomarkers. Hence the overall accuracy can serve as an important objective function for biomarker combination, especially when the combined biomarkers are used for the purpose of making medical diagnosis. In this paper, we address the problem of combining multiple biomarkers to directly maximize the overall diagnostic accuracy by presenting several grid search methods and derivation-based methods. A comprehensive simulation study was conducted to compare the performances of these methods. An ovarian cancer data set is analyzed in the end.

Mutual Information as a Performance Measure for Binary Predictors Characterized by Both ROC Curve and PROC Curve Analysis

The predictive receiver operating characteristic (PROC) curve differs from the more well-known receiver operating characteristic (ROC) curve in that it provides a basis for the evaluation of binary diagnostic tests using metrics defined conditionally on the outcome of the test rather than metrics defined conditionally on the actual disease status. Application of PROC curve analysis may be hindered by the complex graphical patterns that are sometimes generated. Here we present an information theoretic analysis that allows concurrent evaluation of PROC curves and ROC curves together in a simple graphical format. The analysis is based on the observation that mutual information may be viewed both as a function of ROC curve summary statistics (sensitivity and specificity) and prevalence, and as a function of predictive values and prevalence. Mutual information calculated from a 2 × 2 prediction-realization table for a specified risk score threshold on an ROC curve is the same as the mutual information calculated at the same risk score threshold on a corresponding PROC curve. Thus, for a given value of prevalence, the risk score threshold that maximizes mutual information is the same on both the ROC curve and the corresponding PROC curve. Phytopathologists and clinicians who have previously relied solely on ROC curve summary statistics when formulating risk thresholds for application in practical agricultural or clinical decision-making contexts are thus presented with a methodology that brings predictive values within the scope of that formulation.