Area under the ROC curve comparison in the presence of missing data

Missing values and inconclusive results in diagnostic studies - A scoping review of methods

Most diagnostic studies exclude missing values and inconclusive results from the analysis or apply simple methods resulting in biased accuracy estimates. This may be due to the lack of availability or awareness of appropriate methods. This scoping review aimed to provide an overview of strategies to handle missing values and inconclusive results in the reference standard or index test in diagnostic accuracy studies. Conducting a systematic literature search in MEDLINE, Cochrane Library, and Web of Science, we could identify many articles proposing methods for addressing missing values in the reference standard. There are also several articles describing methods regarding missing values or inconclusive results in the index test. The latter encompass imputation, frequentist and Bayesian likelihood, model-based, and latent class methods. While methods for missing values in the reference standard are regularly applied in practice, this is not true for methods addressing missing values and inconclusive results in the index test. Our comprehensive overview and description of available methods may raise further awareness of these methods and will enhance their application. Future research is needed to compare the performance of these methods under different conditions to give valid and robust recommendations for their usage in various diagnostic accuracy research scenarios.

The area under the generalized receiver-operating characteristic curve

The receiver operating-characteristic (ROC) curve is a well-known graphical tool routinely used for evaluating the discriminatory ability of continuous markers, referring to a binary characteristic. The area under the curve (AUC) has been proposed as a summarized accuracy index. Higher values of the marker are usually associated with higher probabilities of having the characteristic under study. However, there are other situations where both, higher and lower marker scores, are associated with a positive result. The generalized ROC (gROC) curve has been proposed as a proper extension of the ROC curve to fit these situations. Of course, the corresponding area under the gROC curve, gAUC, has also been introduced as a global measure of the classification capacity. In this paper, we study in deep the gAUC properties. The weak convergence of its empirical estimator is provided while deriving an explicit and useful expression for the asymptotic variance. We also obtain the expression for the asymptotic covariance of related gAUCs and propose a non-parametric procedure to compare them. The finite-samples behavior is studied through Monte Carlo simulations under different scenarios, presenting a real-world problem in order to illustrate its practical application. The R code functions implementing the procedures are provided as Supplementary Material.

The Predictive Properties of Violence Risk Instruments May Increase by Adding Items Assessing Sleep

Background: The psychometric instruments developed for short-term prediction of violence in psychiatric inpatients do not include variables assessing sleep. Disturbances in sleep may precede aggression in this setting. We investigated whether adding information on sleep improved the predictive properties of the Brøset Violence Checklist (BVC). Methods: The study population consists of all patients admitted to a psychiatric intensive care unit (PICU) over a 6-month period who were hospitalized for at least one night (n = 50). Sleep observed by staff (521 nights), behavior assessed with the BVC (433 days), and aggressive incidents recorded by the Staff Observation Scale-Revised (n = 14) were included in the analysis. Results: The ability of the BVC to predict aggressive incidents improved from AUC_ROC 0.757 to AUC_ROC 0.873 when a combined sleep variable including both sleep duration and night-to-night variations of sleep duration was added to the BVC recordings. The combined sleep variable did not significantly predict aggressive incidents (AUC_ROC 0.653, p = 0.051). Conclusions: A sleep disturbance variable improves the predictive properties of the BVC in PICUs. Further studies of sleep duration, night-to-night variations in duration of sleep, and aggression are needed.

Identifying Fraction Measures as Screeners of Mathematics Risk Status

This study investigated the accuracy of three fraction measures (i.e., fraction number line estimation accuracy, general fraction concepts, and fraction arithmetic) for screening fourth graders who might be at risk for mathematics difficulties. Receiver operating characteristic (ROC) curve analyses assessed diagnostic accuracy of the fraction measures for predicting which students would not meet state standards on the state mathematics test in fourth grade (n = 411), fifth grade (n = 362), and sixth grade (n = 304). A combined measure consisting primarily of fraction number line estimation items and general fraction concept items was the most accurate screener of risk status in fourth, fifth, and sixth grades (area under the curve [AUC] = .84, .81, and .85, respectively). To maximize efficiency for classroom use, the length of the combined screener was reduced using best subset automatic linear modeling. The study highlights the importance of fraction knowledge for predicting mathematics achievement more generally and validates an effective and practical screening tool for the intermediate grades.

Fully non-parametric receiver operating characteristic curve estimation for random-effects meta-analysis

Meta-analyses, broadly defined as the quantitative review and synthesis of the results of related but independent comparable studies, allow to know the state of the art of one considered topic. Since the amount of available bibliography has enhanced in almost all fields and, specifically, in biomedical research, its popularity has drastically increased during the last decades. In particular, different methodologies have been developed in order to perform meta-analytic studies of diagnostic tests for both fixed- and random-effects models. From a parametric point of view, these techniques often compute a bivariate estimation for the sensitivity and the specificity by using only one threshold per included study. Frequently, an overall receiver operating characteristic curve based on a bivariate normal distribution is also provided. In this work, the author deals with the problem of estimating an overall receiver operating characteristic curve from a fully non-parametric approach when the data come from a meta-analysis study i.e. only certain information about the diagnostic capacity is available. Both fixed- and random-effects models are considered. In addition, the proposed methodology lets to use the information of all cut-off points available (not only one of them) in the selected original studies. The performance of the method is explored through Monte Carlo simulations. The observed results suggest that the proposed estimator is better than the reference one when the reported information is related to a threshold based on the Youden index and when information for two or more points are provided. Real data illustrations are included.

Receiver operating characteristic curve generalization for non-monotone relationships

The receiver operating characteristic curve is a popular graphical method frequently used in order to study the diagnostic capacity of continuous markers. It represents in a plot true-positive rates against the false-positive ones. Both the practical and theoretical aspects of the receiver operating characteristic curve have been extensively studied. Conventionally, it is assumed that the considered marker has a monotone relationship with the studied characteristic; i.e., the upper (lower) values of the (bio)marker are associated with a higher probability of a positive result. However, there exist real situations where both the lower and the upper values of the marker are associated with higher probability of a positive result. We propose a receiver operating characteristic curve generalization, [Formula: see text], useful in this context. All pairs of possible cut-off points, one for the lower and another one for the upper marker values, are taken into account and the best of them are selected. The natural empirical estimator for the [Formula: see text] curve is considered and its uniform consistency and asymptotic distribution are derived. Finally, two real-world applications are studied.