New Nonparametric Confidence Intervals for the Youden Index

Interval estimation for the Youden index of a continuous diagnostic test with verification biased data

In medical diagnostic studies, the Youden index plays a crucial role as a comprehensive measurement of the diagnostic test effectiveness, aiding in determining the optimal threshold values by maximizing the sum of sensitivity and specificity. However, in clinical practice, verification of true disease status might be partially missing and estimators based on partially validated subjects are usually biased. While verification bias-corrected estimation methods for the receiver operating characteristic curve have been widely studied, no such results have been specifically developed for the Youden index. In this paper, we propose bias-corrected interval estimation methods for the Youden index of a continuous test under the missing-at-random assumption. Based on four estimators (full imputation (FI), mean score imputation, inverse probability weighting, and the semiparametric efficient (SPE)) introduced by Alonzo and Pepe for handling verification bias, we develop multiple confidence intervals for the Youden index by applying bootstrap resampling and the method of variance estimates recovery (MOVER). Extensive simulation and real data studies show that when the disease model is correctly specified, MOVER-FI intervals yield better coverage probability. We also observe a tradeoff between methods when the verification proportion is low: Bootstrap approaches achieve higher accuracy, while MOVER approaches deliver greater precision. Remarkably, bootstrap-SPE interval exhibit appealing doubly robustness to model misspecification and perform adequately across almost all scenarios considered. Based on our findings, we recommend using the bootstrap-SPE intervals when the true disease model is unknown, and the MOVERws-FI interval if the true disease model can be well approximated.

Interval Estimation for the Youden Index and Optimal Cut-Off Point in AUC-Based Optimal Combinations of Multivariate Normal Biomarkers With Covariates

In this article, we present interval estimation methods for the Youden index and the optimal cut-off point in the context of AUC-based optimal combinations of multivariate normally distributed biomarkers, considering the presence of covariates. We propose a generalized pivotal confidence interval, a Bayesian credible interval, and several bootstrap confidence intervals for both the Youden index and its corresponding cut-off point. To evaluate the performance of these confidence and credible intervals, we conducted a Monte Carlo simulation study. Finally, we illustrate the proposed methods using a diabetic dataset.

Confidence intervals and sample size planning for optimal cutpoints

Various methods are available to determine optimal cutpoints for diagnostic measures. Unfortunately, many authors fail to report the precision at which these optimal cutpoints are being estimated and use sample sizes that are not suitable to achieve an adequate precision. The aim of the present study is to evaluate methods to estimate the variance of cutpoint estimations based on published descriptive statistics ('post-hoc') and to discuss sample size planning for estimating cutpoints. We performed a simulation study using widely-used methods to optimize the Youden index (empirical, normal, and transformed normal method) and three methods to determine confidence intervals (the delta method, the parametric bootstrap, and the nonparametric bootstrap). We found that both the delta method and the parametric bootstrap are suitable for post-hoc calculation of confidence intervals, depending on the sample size, the distribution of marker values, and the correctness of model assumptions. On average, the parametric bootstrap in combination with normal-theory-based cutpoint estimation has the best coverage. The delta method performs very well for normally distributed data, except in small samples, and is computationally more efficient. Obviously, not every combination of distributions, cutpoint optimization methods, and optimized metrics can be simulated and a lot of the literature is concerned specifically with cutpoints and confidence intervals for the Youden index. This complicates sample size planning for studies that estimate optimal cutpoints. As a practical tool, we introduce a web-application that allows for running simulations of width and coverage of confidence intervals using the percentile bootstrap with various distributions and cutpoint optimization methods.

Application of ROC Curve Analysis for Predicting Students’ Passing Grade in a Course Based on Prerequisite Grades

Determining prerequisite requirements is vital for successful curriculum development and student on-schedule completion of the course of study. This study adapts the Receiver Operating Characteristic (ROC) curve analysis to determine a threshold grade in a prerequisite course necessary for passing the next course in a sequence. This method was tested on a dataset of Calculus 1 and Calculus 2 grades of 164 undergraduate students majoring in mathematics at a private university in Kazakhstan. The results showed that while the currently used practice of setting prerequisite grade requirements is accurately identifying successful completions of Calculus 2, the ROC method is more accurate in identifying possible failures in Calculus 2. The findings also indicate that prior completion of Calculus 1 is positively associated with success in a Calculus 2 course. Thus, this study contributes to the field of mathematics education by providing a new data-driven methodology for determining the optimal threshold grade for mathematics prerequisite courses.

Prediction of gestational diabetes mellitus by different obesity indices

Background

The incidence rates of obesity and gestational diabetes mellitus (GDM) are increasing in parallel. This study aimed to evaluate the relationship between different obesity indices, including prepregnancy body mass index (preBMI), the first-trimester abdominal circumference (AC), and first-trimester abdominal circumference/height ratio (ACHtR), and GDM, and the efficacy of these three indices in predicting GDM was assessed.

Methods

A total of 15,472 pregnant women gave birth to a singleton at the Obstetrics and Gynecology Hospital of Fudan University, Shanghai, China. Prepregnancy weight was self-reported by study participants, body height and AC were measured by nurses at the first prenatal visit during weeks 11 to 13⁺⁶ of pregnancy. GDM was diagnosed through a 75-g oral glucose tolerance test at 24–28 gestational weeks. Using receiver operator characteristic (ROC) curve analysis, we evaluated the association between obesity indices and GDM.

Results

A total of 1912 women (12.4%) were diagnosed with GDM. Logistic regression analysis showed that AC, ACHtR, and preBMI (P < 0.001) were all independent risk factors for the development of GDM. In the normal BMI population, the higher the AC or ACHtR was, the more likely the pregnant woman was to develop GDM. The area under the ROC curve (AUC) was 0.63 (95% CI: 0.62–0.64) for the AC, 0.64 (95% CI: 0.62–0.65) for the ACHtR and 0.63 (95% CI: 0.62–0.64) for the preBMI. An AC ≥ 80.3 cm (sensitivity: 61.6%; specificity: 57.9%), an ACHtR of ≥ 0.49 (sensitivity: 67.3%; specificity: 54.0%), and a preBMI ≥ 22.7 (sensitivity: 48.4%; specificity: 71.8%) were determined to be the best cut-off levels for identifying subjects with GDM.

Conclusions

An increase in ACHtR may be an independent risk factor for GDM in the first trimester of pregnancy. Even in the normal BMI population, the higher the AC and ACHtR are, the more likely a pregnant woman is to develop GDM. AC, ACHtR in the first trimester and preBMI might be anthropometric indices for predicting GDM, but a single obesity index had limited predictive value for GDM.

Robust RNA-seq data analysis using an integrated method of ROC curve and Kolmogorov-Smirnov test

It is a common approach to dichotomize a continuous biomarker in clinical setting for the convenience of application. Analytically, results from using a dichotomized biomarker are often more reliable and resistant to outliers, bi-modal and other unknown distributions. There are two commonly used methods for selecting the best cut-off value for dichotomization of a continuous biomarker, using either maximally selected chi-square statistic or a ROC curve, specifically the Youden Index. In this paper, we explained that in many situations, it is inappropriate to use the former. By using the Maximum Absolute Youden Index (MAYI), we demonstrated that the integration of a MAYI and the Kolmogorov-Smirnov test is not only a robust non-parametric method, but also provides more meaningful p value for selecting the cut-off value than using a Mann-Whitney test. In addition, our method can be applied directly in clinical settings.

Statistical inference for the difference between two maximized Youden indices obtained from correlated biomarkers

Currently, there is global interest in deriving new promising cancer biomarkers that could complement or substitute the conventional ones. Clinical decisions can often be based on the cutoff that corresponds to the maximized Youden index when maximum accuracy drives decisions. When more than one classification criteria are measured within the same individuals, correlated measurements arise. In this work, we propose hypothesis tests and confidence intervals for the comparison of two correlated receiver operating characteristic (ROC) curves in terms of their corresponding maximized Youden indices. We explore delta-based techniques under parametric assumptions, or power transformations. Nonparametric kernel-based methods are also examined. We evaluate our approaches through simulations and illustrate them using data from a metabolomic study referring to the detection of pancreatic cancer.

Using AMANHI-ACT cohorts for external validation of Iowa new-born metabolic profiles based models for postnatal gestational age estimation

BACKGROUND

Globally, 15 million infants are born preterm and another 23.2 million infants are born small for gestational age (SGA). Determining burden of preterm and SGA births, is essential for effective planning, modification of health policies and targeting interventions for reducing these outcomes for which accurate estimation of gestational age (GA) is crucial. Early pregnancy ultrasound measurements, last menstrual period and post-natal neonatal examinations have proven to be not feasible or inaccurate. Proposed algorithms for GA estimation in western populations, based on routine new-born screening, though promising, lack validation in developing country settings. We evaluated the hypothesis that models developed in USA, also predicted GA in cohorts of South Asia (575) and Sub-Saharan Africa (736) with same precision.

METHODS

Dried heel prick blood spots collected 24-72 hours after birth from 1311 new-borns, were analysed for standard metabolic screen. Regression algorithm based, GA estimates were computed from metabolic data and compared to first trimester ultrasound validated, GA estimates (gold standard).

RESULTS

Overall Algorithm (metabolites + birthweight) estimated GA to within an average deviation of 1.5 weeks. The estimated GA was within the gold standard estimate by 1 and 2 weeks for 70.5% and 90.1% new-borns respectively. Inclusion of birthweight in the metabolites model improved discriminatory ability of this method, and showed promise in identifying preterm births. Receiver operating characteristic (ROC) curve analysis estimated an area under curve of 0.86 (conservative bootstrap 95% confidence interval (CI) = 0.83 to 0.89); P < 0.001) and Youden Index of 0.58 (95% CI = 0.51 to 0.64) with a corresponding sensitivity of 80.7% and specificity of 77.6%.

CONCLUSION

Metabolic gestational age dating offers a novel means for accurate population-level gestational age estimates in LMIC settings and help preterm birth surveillance initiatives. Further research should focus on use of machine learning and newer analytic methods broader than conventional metabolic screen analytes, enabling incorporation of region-specific analytes and cord blood metabolic profiles models predicting gestational age accurately.

The Youden Index in the Generalized Receiver Operating Characteristic Curve Context

The receiver operating characteristic (ROC) curve and their associated summary indices, such as the Youden index, are statistical tools commonly used to analyze the discrimination ability of a (bio)marker to distinguish between two populations. This paper presents the concept of Youden index in the context of the generalized ROC (gROC) curve for non-monotone relationships. The interval estimation of the Youden index and the associated cutoff points in a parametric (binormal) and a non-parametric setting is considered. Monte Carlo simulations and a real-world application illustrate the proposed methodology.

A Smooth Nonparametric Approach to Determining Cut-Points of A Continuous Scale

The problem of determining cut-points of a continuous scale according to an establish categorical scale is often encountered in practice for the purposes such as making diagnosis or treatment recommendation, determining study eligibility, or facilitating interpretations. A general analytic framework was recently proposed for assessing optimal cut-points defined based on some pre-specified criteria. However, the implementation of the existing nonparametric estimators under this framework and the associated inferences can be computationally intensive when more than a few cut-points need to be determined. To address this important issue, a smoothing-based modification of the current method is proposed and is found to substantially improve the computational speed as well as the asymptotic convergence rate. Moreover, a plug-in type variance estimation procedure is developed to further facilitate the computation. Extensive simulation studies confirm the theoretical results and demonstrate the computational benefits of the proposed method. The practical utility of the new approach is illustrated by an application to a mental health study.

Partial Youden index and its inferences

In medical diagnostic research, medical tests with continuous values are widely employed to distinguish between diseased and non-diseased subjects. The diagnostic accuracy of a medical test can be assessed by using the receiver operating characteristic (ROC) curve of the test. To summarize the ROC curve and determine an optimal cut-off point for test results, the Youden index is commonly used. In particular, the Youden index is optimized over the entire range of values for sensitivity and specificity, which determine the ROC space. However in clinical practice, one may only be interested in the regions of the ROC curve that correspond to low false-positive rates or/and high sensitivities. In this paper, a new summary index for the ROC curve, called the "partial Youden index", is defined on regions of the ROC space in which sensitivity/specificity values are of practical interest. The traditional Youden index is a special case of the partial Youden index. Various parametric and non-parametric interval estimation methods are proposed for the partial Youden index. Extensive simulation studies are conducted to evaluate the finite sample performances of the proposed methods. A real example is used to illustrate the application of the new methods.

The utility of HbA1c combined with haematocrit for early screening of gestational diabetes mellitus

AIMS

To evaluate the utility of glycated haemoglobin A1c (HbA1c) alone and in combination with haematocrit (HCT) for screening gestational diabetes mellitus (GDM) between 12-16 gestational weeks.

METHODS

This prospective study was carried out in the Obstetrics and Gynaecology Hospital of Fudan University from November 2014 to February 2015. In total, 690 pregnant women between 20 and 35 years old were included in this study. All subjects received a routine blood examination for HbA1c and HCT at 12-16 gestational weeks (gw) and a 75-g oral glucose tolerance test at 24-28 gw. Threshold values for the diagnosis of GDM were a plasma glucose concentration of 5.1 mmol/L after fasting, 10.0 mmol/L at 60 min, and 8.5 mmol/L at 120 min. Receiver operating characteristic curves were used to evaluate the diagnostic performance of HbA1c with or without HCT.

RESULTS

One hundred seven women were diagnosis with GDM at 24-28 gw. An HbA1c cutoff value < 4.55% at 12-16 gw showed adequate sensitivity to exclude GDM (85.0%) but low specificity (17.3%), while an HbA1c cutoff value ≥ 5.25% presented adequate specificity (96.6%) but low sensitivity (13.3%) in diagnosing GDM. The area under the receiver operating characteristic curve for HbA1c (12-16 gw) detection of GDM was 0.563 (95% confidence interval [CI], 0.50-0.625). When combined HbA1c with HCT ( > 38.8%) for the screening of GDM, the area under the receiver operating characteristic curve was 0.604 (95% [CI] 0.509, 0.701).

CONCLUSIONS

Whether the adoption of HbA1c as a screening test for GDM would benefit pregnant women remains to be determined. However, combining HbA1c with HCT for the screening of GDM may be a useful tool to predict GDM.

Radiologic-Pathologic Correlation of Solid Portions on Thin-section CT Images in Lung Adenocarcinoma: A Multicenter Study

BACKGROUND

Measuring the size of invasiveness on computed tomography (CT) for the T descriptor size was deemed important in the 8th edition of the TNM lung cancer classification. We aimed to correlate the maximal dimensions of the solid portions using both lung and mediastinal window settings on CT imaging with the pathologic invasiveness (> 0.5 cm) in lung adenocarcinoma patients.

MATERIALS AND METHODS

The study population consisted of 378 patients with a histologic diagnosis of adenocarcinoma in situ (AIS), minimally invasive adenocarcinoma (MIA), invasive adenocarcinoma (IVA)-lepidic, IVA-acinar and/or IVA-papillary, and IVA-micropapillary and/or solid adenocarcinoma. A panel of 15 radiologists was divided into 2 groups (group A, 9 radiologists; and group B, 6 radiologists). The 2 groups independently measured the maximal and perpendicular dimensions of the solid components and entire tumors on the lung and mediastinal window settings. The solid proportion of nodule was calculated by dividing the solid portion size (lung and mediastinal window settings) by the nodule size (lung window setting). The maximal dimensions of the invasive focus were measured on the corresponding pathologic specimens by 2 pathologists.

RESULTS

The solid proportion was larger in the following descending order: IVA-micropapillary and/or solid, IVA-acinar and/or papillary, IVA-lepidic, MIA, and AIS. For both groups A and B, a solid portion > 0.8 cm in the lung window setting or > 0.6 cm in the mediastinal window setting on CT was a significant indicator of pathologic invasiveness > 0.5 cm (P < .001; receiver operating characteristic analysis using Youden's index).

CONCLUSION

A solid portion > 0.8 cm on the lung window setting or solid portion > 0.6 cm on the mediastinal window setting on CT predicts for histopathologic invasiveness to differentiate IVA from MIA and AIS.

Developing Autism Screening Criteria for the Brief Infant Toddler Social Emotional Assessment (BITSEA)

There is a critical need for evidence-based, broadband behavioral, and ASD screening measures for use in pediatric and early educational settings to ensure that young children at risk for developing social-emotional disorders and/or ASD are provided with early intervention services to optimize long-term outcomes. The BITSEA is a 42-item screener designed to identify social-emotional/behavioral problems and delays/deficits in social-emotional competence among 11-48-month-olds; 19 items describe behaviors consistent with ASD. Secondary data analysis was employed to develop cut-scores for ASD subscales using Receiver Operating Curves, discriminating children with (n = 223) and without (n = 289) ASD. Cut-scores demonstrated moderate-to-high discriminative power, sensitivity, specificity, and PPV. Findings highlight feasibility of using a broadband social-emotional competence and behavior problem screener to improve early detection of ASD.

A nonparametric fiducial interval for the Youden index in multi-state diagnostic settings

The Youden index is a commonly employed metric to characterize the performance of a diagnostic test at its optimal point. For tests with three or more outcome classes, the Youden index has been extended; however, there are limited methods to compute a confidence interval (CI) about its value. Often, outcome classes are assumed to be normally distributed, which facilitates computational formulas for the CI bounds; however, many scenarios exist for which these assumptions cannot be made. In addition, many of these existing CI methods do not work well for small sample sizes. We propose a method to compute a nonparametric interval about the Youden index utilizing the fiducial argument. This fiducial interval ensures that CI coverage is met regardless of sample size, underlying distributional assumptions, or use of a complex classifier for diagnosis. Two alternate fiducial intervals are also considered. A simulation was conducted, which demonstrates the coverage and interval length for the proposed methods. Comparisons were made using no distributional assumptions on the outcome classes and for when outcomes were assumed to be normally distributed. In general, coverage probability was consistently met, and interval length was reasonable. The proposed fiducial method was also demonstrated in data examining biomarkers in subjects to predict diagnostic stages ranging from normal kidney function to chronic allograph nephropathy. Published 2015. This article is a U.S. Government work and is in the public domain in the USA.

Improved Confidence Intervals for the Youden Index

The Youden Index is a summary measurement of the receiver operating characteristic (ROC) curve for the accuracy of a diagnostic test with ordinal or continuous endpoints. The bootstrap confidence interval based on the adjusted proportion estimate was shown to have satisfactory performance among the existing confidence intervals, including the parametric interval via the delta method. In this article, we propose two confidence intervals using the square-and-add limits based on the Wilson score method. We compare the two proposed intervals with the existing interval with extensive simulation studies. The new interval based on the empirical proportion estimate generally has better performance than that based on the adjusted proportion estimate. A real example from a clinical trial of prostate cancer is illustrated for the application of the new intervals.