Powerful nonparametric statistics to compare k independent ROC curves

Examining factors related to low performance of predicting remission in participants with major depressive disorder using neuroimaging data and other clinical features

Major depressive disorder (MDD), a prevalent mental health issue, affects more than 8% of the US population, and almost 17% in the young group of 18-25 years old. Since Covid-19, its prevalence has become even more significant. However, the remission (being free of depression) rates of first-line antidepressant treatments on MDD are only about 30%. To improve treatment outcomes, researchers have built various predictive models for treatment responses and yet none of them have been adopted in clinical use. One reason is that most predictive models are based on data from subjective questionnaires, which are less reliable. Neuroimaging data are promising objective prognostic factors, but they are expensive to obtain and hence predictive models using neuroimaging data are limited and such studies were usually in small scale (N<100). In this paper, we proposed an advanced machine learning (ML) pipeline for small training dataset with large number of features. We implemented multiple imputation for missing data and repeated K-fold cross validation (CV) to robustly estimate predictive performances. Different feature selection methods and stacking methods using 6 general ML models including random forest, gradient boosting decision tree, XGBoost, penalized logistic regression, support vector machine (SVM), and neural network were examined to evaluate the model performances. All predictive models were compared using model performance metrics such as accuracy, balanced accuracy, area under ROC curve (AUC), sensitivity and specificity. Our proposed ML pipeline was applied to a training dataset and obtained an accuracy and AUC above 0.80. But such high performance failed while applying our ML pipeline using an external validation dataset from the EMBARC study which is a multi-center study. We further examined the possible reasons especially the site heterogeneity issue.

Sex determination using color fundus parameters in older adults of Kumejima population study

PURPOSE

Deep learning artificial intelligence can determine the sex using only fundus photographs. However, the factors used by deep learning to determine the sex are not visible. Therefore, the purpose of the study was to determine whether the sex of an older individual can be determined by regression analysis of their color fundus photographs (CFPs).

METHODS

Forty-two parameters were analyzed by regression analysis using 1653 CFPs of normal subjects in the Kumajima study. The parameters included the mean values of red, green, and blue intensities; the tessellation fundus index; the optic disc ovality ratio; the papillomacular angle; and the retinal vessel angles. Finally, the L2 regularized binomial logistic regression was used to predict the sex using all the parameters, and the diagnostic ability was assessed through the leave-one-cross-validation.

RESULTS

The mean age of the 838 men and 815 women were 52.8 and 54.0 years, respectively. The ovality ratio and retinal artery angles in women were significantly smaller than that in men. The green intensity at all locations for the women were significantly higher than that of men (P < 0.001). The discrimination accuracy rate assessed by the area-under-the-curve was 80.4%.

CONCLUSIONS

Our methods can determine the sex from the CFPs of the adult with an accuracy of 80.4%. The ovality ratio, retinal vessel angles, tessellation, and the green intensities of the fundus are important factors to identify the sex in individuals over 40 years old.

Confidence intervals for the length of the receiver-operating characteristic curve based on a smooth estimator

A good diagnostic test should show different behavior on both the positive and the negative populations. However, this is not enough for having a good classification system. The binary classification problem is a complex task, which implies to define decision criteria. The knowledge of the level of dissimilarity between the two involved distributions is not enough. We also have to know how to define those decision criteria. The length of the receiver-operating characteristic curve has been proposed as an index of the optimal discriminatory capacity of a biomarker. It is related not with the actual but with the optimal classification capacity of the considered diagnostic test. One particularity of this index is that its estimation should be based on parametric or smoothed models. We explore here the behavior of a kernel density estimator-based approximation for estimating the length of the receiver-operating characteristic curve. We prove the asymptotic distribution of the resulting statistic, propose a parametric bootstrap algorithm for confidence intervals construction, discuss the role that the bandwidth parameter plays in the quality of the provided estimations and, via Monte Carlo simulations, study its finite-sample behavior considering four different criteria for the bandwidth selection. The practical use of the length of the receiver-operating characteristic curve is illustrated through two real-world examples.

The area under the generalized receiver-operating characteristic curve

The receiver operating-characteristic (ROC) curve is a well-known graphical tool routinely used for evaluating the discriminatory ability of continuous markers, referring to a binary characteristic. The area under the curve (AUC) has been proposed as a summarized accuracy index. Higher values of the marker are usually associated with higher probabilities of having the characteristic under study. However, there are other situations where both, higher and lower marker scores, are associated with a positive result. The generalized ROC (gROC) curve has been proposed as a proper extension of the ROC curve to fit these situations. Of course, the corresponding area under the gROC curve, gAUC, has also been introduced as a global measure of the classification capacity. In this paper, we study in deep the gAUC properties. The weak convergence of its empirical estimator is provided while deriving an explicit and useful expression for the asymptotic variance. We also obtain the expression for the asymptotic covariance of related gAUCs and propose a non-parametric procedure to compare them. The finite-samples behavior is studied through Monte Carlo simulations under different scenarios, presenting a real-world problem in order to illustrate its practical application. The R code functions implementing the procedures are provided as Supplementary Material.

Prognostic Value of Combined Detection of Preoperative Albumin-to-Fibrinogen Ratio and Neutrophil-to-Lymphocyte Ratio in Operable Esophageal Squamous Cell Carcinoma Patients without Neoadjuvant Therapy

Background

We retrospectively analyzed the prognostic value of the albumin-to-fibrinogen ratio (AFR)–neutrophil-to-lymphocyte ratio (NLR) score, comprising preoperative AFR and NLR, in esophageal squamous cell carcinoma (ESCC) patients after radical resection.

Patients and Methods

Overall, 215 patients were included. The optimal cutoff value was determined using the receiver operating characteristic (ROC) curve. Based on a low AFR (<12.06) and high NLR (≥1.78), the AFR–NLR score was classified as 2 (both hematological abnormalities present), 1 (one abnormality present), or 0 (both abnormalities absent). Kaplan–Meier curves, Cox regression, and predicted nomogram were used to evaluate the prognostic value of the score.

Results

The prognostic value of the AFR–NLR score was better than that of AFR or NLR alone (P <0.05). Multivariate analysis showed that a high AFR–NLR score was an independent predictor of poor prognosis for overall survival (P <0.001). Additionally, in the nomogram including the AFR–NLR score, the net reclassification improvement index increased by 35.5% (P <0.001), and the integrated discrimination improvement index increased by 9.0% (P <0.001). The predictive accuracy of the established nomogram model was proved using Harrell’s concordance index (0.811, 95% confidence interval: 0.765–0.856) and calibration curve. Notably, the decision analysis curve showed that the nomogram had a higher net benefit within most of the threshold probability range, indicating better clinical applicability.

Conclusion

The AFR–NLR score is a useful predictor of the prognosis of ESCC patients after radical resection, and the nomogram established on the basis of this score has a good prognostic value.

Fully non-parametric receiver operating characteristic curve estimation for random-effects meta-analysis

Meta-analyses, broadly defined as the quantitative review and synthesis of the results of related but independent comparable studies, allow to know the state of the art of one considered topic. Since the amount of available bibliography has enhanced in almost all fields and, specifically, in biomedical research, its popularity has drastically increased during the last decades. In particular, different methodologies have been developed in order to perform meta-analytic studies of diagnostic tests for both fixed- and random-effects models. From a parametric point of view, these techniques often compute a bivariate estimation for the sensitivity and the specificity by using only one threshold per included study. Frequently, an overall receiver operating characteristic curve based on a bivariate normal distribution is also provided. In this work, the author deals with the problem of estimating an overall receiver operating characteristic curve from a fully non-parametric approach when the data come from a meta-analysis study i.e. only certain information about the diagnostic capacity is available. Both fixed- and random-effects models are considered. In addition, the proposed methodology lets to use the information of all cut-off points available (not only one of them) in the selected original studies. The performance of the method is explored through Monte Carlo simulations. The observed results suggest that the proposed estimator is better than the reference one when the reported information is related to a threshold based on the Youden index and when information for two or more points are provided. Real data illustrations are included.