Comparing three-class diagnostic tests by three-way ROC analysis

Performance of Nuclear Magnetic Resonance-Based Estimated Glomerular Filtration Rate in a Real-World Setting

An accurate estimate of glomerular filtration rate (eGFR) is essential for proper clinical management, especially in patients with kidney dysfunction. This prospective observational study evaluated the real-world performance of the nuclear magnetic resonance (NMR)-based GFR_NMR equation, which combines creatinine, cystatin C, valine, and myo-inositol with age and sex. We compared GFR_NMR performance to that of the 2021 CKD-EPI creatinine and creatinine-cystatin C equations (CKD-EPI_2021Cr and CKD-EPI_2021CrCys), using 115 fresh routine samples of patients scheduled for urinary iothalamate clearance measurement (mGFR). Median bias to mGFR of the three eGFR equations was comparably low, ranging from 0.4 to 2.0 mL/min/1.73 m². GFR_NMR outperformed the 2021 CKD-EPI equations in terms of precision (interquartile range to mGFR of 10.5 vs. 17.9 mL/min/1.73 m² for GFR_NMR vs. CKD-EPI_2021CrCys; p = 0.01) and accuracy (P15, P20, and P30 of 66.1% vs. 48.7% [p = 0.007], 80.0% vs. 60.0% [p < 0.001] and 95.7% vs. 86.1% [p = 0.006], respectively, for GFR_NMR vs. CKD-EPI_2021CrCys). Clinical parameters such as etiology, comorbidities, or medications did not significantly alter the performance of the three eGFR equations. Altogether, this study confirmed the utility of GFR_NMR for accurate GFR estimation, and its potential value in routine clinical practice for improved medical care.

The polytomous discrimination index for prediction involving multistate processes under intermittent observation

With the increasing importance of predictive modeling in health research comes the need for methods to rigorously assess predictive accuracy. We consider the problem of evaluating the accuracy of predictive models for nominal outcomes when outcome data are coarsened at random. We first consider the problem in the context of a multinomial response modeled by polytomous logistic regression. Attention is then directed to the motivating setting in which class membership corresponds to the state occupied in a multistate disease process at a time horizon of interest. Here, class (state) membership may be unknown at the time horizon since disease processes are under intermittent observation. We propose a novel extension to the polytomous discrimination index to address this and evaluate the predictive accuracy of an intensity-based model in the context of a study involving patients with arthritis from a registry at the University of Toronto Centre for Prognosis Studies in Rheumatic Diseases.

On optimal biomarker cutoffs accounting for misclassification costs in diagnostic trilemmas with applications to pancreatic cancer

Pancreatic ductal adenocarcinoma (PDAC) is the most deadly cancer and currently there is strong clinical interest in novel biomarkers that contribute to its early detection. Assessing appropriately the accuracy of such biomarkers is a crucial issue and often one needs to take into account that many assays include biospecimens of individuals coming from three groups: healthy, chronic pancreatitis, and PDAC. The ROC surface is an appropriate tool for assessing the overall accuracy of a marker employed under such trichotomous settings. A decision/classification rule is often based on the so-called Youden index and its three-dimensional generalization. However, both the clinical and the statistical literature have not paid the necessary attention to the underlying false classification (FC) rates that are of equal or even greater importance. In this article we provide a framework to make inferences around all classification rates as well as comparisons. We explore the trinormal model, flexible models based on power transformations, and robust non-parametric alternatives. We provide a full framework for the construction of confidence intervals, regions, and spaces for joint inferences or for clinically meaningful points of interest. We further discuss the implications of costs related to different FCs. We evaluate our approaches through extensive simulations and illustrate them using data from a recent PDAC study conducted at the MD Anderson Cancer Center.

Evaluation of Ring Amplitude and Factors Affecting Ring Amplitude in Multifocal Electroretinography in Diabetic Eyes

PURPOSE

The aim of this paper was to evaluate the ring amplitudes in diabetic patients and to evaluate the effect of the risk factors for diabetic retinopathy on the ring amplitudes. We also aimed to investigate the success of ring amplitudes in classifying diabetic retinopathy.

METHODS

The study included 32 eyes of 32 diabetic patients without retinopathy (DM), 34 eyes of 34 patients with mild non-proliferative diabetic retinopathy (NPDR) without macular edema, and 62 eyes of 62 age- and sex-matched controls (CG). All subjects were evaluated using mfERG. The relationship between age, diabetes duration, HbA1c and ring amplitudes and the effect of diabetes and hypertension on ring amplitudes were evaluated. Three-way ROC analysis was performed to evaluate the discrimination power of the ring amplitudes.

RESULTS

In the comparison of the ring amplitudes, the amplitudes of the DM and NPDR groups were statistically significantly decreased compared to the CG (p < .05). A moderate to strong correlation was found between the duration of diabetes, HbA1c and ring amplitudes (p < .05). The effect of diabetes decreased towards the peripheral rings and hypertension did not affect ring amplitudes. Volume under the ROC surface of R1 = 0.65 had p < .05 and 95% CI [0.50-0.72], and the best cut-off point pair to differentiate the three classes was found to be c₁ = 217.3, c₂ = 151.2 in three-way ROC analysis.

CONCLUSION

In conclusion, the effects of diabetes are unevenly distributed on the retina topographically. Diabetes affects the central rings more than peripheral rings in multifocal ERG. Both ring densities and ring ratios are effective ways to identify early changes in retinal function.

Accuracy of one-step automated orthodontic diagnosis model using a convolutional neural network and lateral cephalogram images with different qualities obtained from nationwide multi-hospitals

Objective

The purpose of this study was to investigate the accuracy of one-step automated orthodontic diagnosis of skeletodental discrepancies using a convolutional neural network (CNN) and lateral cephalogram images with different qualities from nationwide multi-hospitals.

Methods

Among 2,174 lateral cephalograms, 1,993 cephalograms from two hospitals were used for training and internal test sets and 181 cephalograms from eight other hospitals were used for an external test set. They were divided into three classification groups according to anteroposterior skeletal discrepancies (Class I, II, and III), vertical skeletal discrepancies (normodivergent, hypodivergent, and hyperdivergent patterns), and vertical dental discrepancies (normal overbite, deep bite, and open bite) as a gold standard. Pre-trained DenseNet-169 was used as a CNN classifier model. Diagnostic performance was evaluated by receiver operating characteristic (ROC) analysis, t-stochastic neighbor embedding (t-SNE), and gradient-weighted class activation mapping (Grad-CAM).

Results

In the ROC analysis, the mean area under the curve and the mean accuracy of all classifications were high with both internal and external test sets (all, > 0.89 and > 0.80). In the t-SNE analysis, our model succeeded in creating good separation between three classification groups. Grad-CAM figures showed differences in the location and size of the focus areas between three classification groups in each diagnosis.

Conclusions

Since the accuracy of our model was validated with both internal and external test sets, it shows the possible usefulness of a one-step automated orthodontic diagnosis tool using a CNN model. However, it still needs technical improvement in terms of classifying vertical dental discrepancies.

A family of estimators to diagnostic accuracy when candidate tests are subject to detection limits-Application to diagnosing early stage Alzheimer disease

In disease diagnosis, individuals are usually assumed to be one of the two basic types, healthy or diseased, as typically based on an established gold standard. Candidate markers for diagnosing a disease often are much cheaper and less invasive than the gold standard but must be evaluated against the gold standard for their sensitivity and specificity to accurately diagnose the disease. When candidate diagnostic markers are fully measured, receiver operating characteristic curves have been the standard approaches for assessing diagnostic accuracy. However, full measurements of diagnostic markers may not be available above or below certain limits due to various practical and technical limitations. For example, in the diagnosis of Alzheimer disease using cerebrospinal fluid biomarkers, the Roche Elecsys® immunoassays have a measuring range for multiple cerebrospinal fluid molecular concentrations. Many cognitive tests used in diagnosing dementia due to Alzheimer disease are also subject to detection limits, often referred to as the floor and ceiling effects in the neuropsychological literature. We propose a new statistical methodology for estimating the diagnostic accuracy when a diagnostic marker is subject to detection limits by dividing the entire study sample into two sub-samples by a threshold of the diagnostic marker. We then propose a family of estimators to the area under the receiver operating characteristic curve by combining a conditional nonparametric estimator and another conditional semi-parametric estimator derived from Cox's proportional hazards model. We derive the variance to the proposed estimators, and further, assess the performance of the proposed estimators as a function of possible thresholds through an extensive simulation study, and recommend the optimum thresholds. Finally, we apply the proposed methodology to assess the ability of several cerebrospinal fluid biomarkers and cognitive tests in diagnosing early stage Alzheimer disease dementia.

Prognostic accuracy for predicting ordinal competing risk outcomes using ROC surfaces

Many medical conditions are marked by a sequence of events in association with continuous changes in biomarkers. Few works have evaluated the overall accuracy of a biomarker in predicting disease progression. We thus extend the concept of receiver operating characteristic (ROC) surface and the volume under the surface (VUS) from multi-category outcomes to ordinal competing-risk outcomes that are also subject to noninformative censoring. Two VUS estimators are considered. One is based on the definition of the ROC surface and obtained by integrating the estimated ROC surface. The other is an inverse probability weighted U estimator that is built upon the equivalence of the VUS to the concordance probability between the marker and sequential outcomes. Both estimators have nice asymptotic results that can be derived using counting process techniques and U-statistics theory. We illustrate their good practical performances through simulations and applications to two studies of cognition and a transplant dataset.

Empirical likelihood confidence interval for sensitivity to the early disease stage

Disease status can naturally be classified into three or more ordinal stages rather than just being binary stages. Many works have been done for the estimation and inference procedure regarding three ordinal disease stages, which are non-disease, early disease, and full disease stages. The early disease stage can be very important for therapeutic intervention and prevention potentiality. As a diagnostic measure, sensitivity to the early disease stage is often used. In this article, we propose confidence intervals for the sensitivity to early disease stage based on given target specificity for non-disease stage and target sensitivity to full disease stage using both empirical likelihood (EL) and adjusted EL procedures. We compare the performance of the proposed EL confidence intervals with other procedures in our simulation study. The proposed procedures are further applied to Alzheimer's Disease Neuroimaging Initiative data set.

Estimating Glomerular Filtration Rate from Serum Myo-Inositol, Valine, Creatinine and Cystatin C

Assessment of renal function relies on the estimation of the glomerular filtration rate (eGFR). Existing eGFR equations, usually based on serum levels of creatinine and/or cystatin C, are not uniformly accurate across patient populations. In the present study, we expanded a recent proof-of-concept approach to optimize an eGFR equation targeting the adult population with and without chronic kidney disease (CKD), based on a nuclear magnetic resonance spectroscopy (NMR) derived 'metabolite constellation' (GFRNMR). A total of 1855 serum samples were partitioned into development, internal validation and external validation datasets. The new GFRNMR equation used serum myo-inositol, valine, creatinine and cystatin C plus age and sex. GFRNMR had a lower bias to tracer measured GFR (mGFR) than existing eGFR equations, with a median bias (95% confidence interval [CI]) of 0.0 (-1.0; 1.0) mL/min/1.73 m2 for GFRNMR vs. -6.0 (-7.0; -5.0) mL/min/1.73 m2 for the Chronic Kidney Disease Epidemiology Collaboration equation that combines creatinine and cystatin C (CKD-EPI2012) (p < 0.0001). Accuracy (95% CI) within 15% of mGFR (1-P15) was 38.8% (34.3; 42.5) for GFRNMR vs. 47.3% (43.2; 51.5) for CKD-EPI2012 (p < 0.010). Thus, GFRNMR holds promise as an alternative way to assess eGFR with superior accuracy in adult patients with and without CKD.

Flexible diagnostic measures and new cut-point selection methods under multiple ordered classes

Medical diagnosis is essentially a classification problem and usually it is done with multiple ordered classes. For example, cancer diagnosis might be "non-malignant," "early stage," or "late stage." Therefore, appropriate measures are needed to assess the accuracy of diagnostic markers under multiple ordered classes. However, all existing measures fail to differentiate among some distinctly different biomarkers. This paper presents a multi-step procedure for evaluating biomarker accuracy under multiple ordered classes. This procedure leads to two new flexible overall measures as well as three new cut-point selection methods with great computational ease. The performance of proposed measures and cut-point selection methods are numerically explored via a simulation study. In the end, an ovarian cancer dataset from the Prostate, Lung, Colorectal, and Ovarian cancer study is analyzed. The proposed accuracy measures were estimated for markers CA125 and HE4, and cut-points were estimated for the risk of ovarian malignancy algorithm score.

Statistical Methodology for Multiclass Classifications: Applications to Dementia

Alzheimer's disease (AD) is a common, devastating disease which carries a heavy economic burden. Accelerated efforts to identify presymptomatic stages of AD and biomarkers to classify the disease are urgent needs. Currently, no biomarkers can perfectly discriminate individuals into multiple disease categories of AD (no cognitive impairment, mild cognitive impairment, and dementia). Although many biomarkers for diagnosis and their various features are being studied, we lack advanced statistical methods which can fully utilize biomarkers to classify AD accurately, thereby facilitating evaluation of putative markers both alone and in combination. In this paper, we propose two approaches: 1) a forward addition procedure in which we adapt an additive logistic regression model to the setting for disease with ordered multiple categories. Using this approach, we select and combine multiple cross-sectional biomarkers to improve diagnostic accuracy, and 2) a method by extending the Neyman-Pearson Lemma to the ordered three disease categories to construct optimal cutoff points to distinguish multiple disease categories. We evaluate the robustness of the proposed model using a simulation study. Then we apply these two methods to data from the Religious Orders Study to examine the feasibility of combining biomarkers, and compare the diagnostic accuracy between the proposed methods and existing methods including model-based methods (ordinal logistic regression and quadratic discriminant analysis), a tree-based method CART, and the Youden index method. The two proposed methods facilitate evaluations of biomarkers for conditions with graded, rather than binary, classifications. The evaluation of the performance of different approaches provides guidance of how to choose approaches to address research questions.

Cumulative ROC curves for discriminating three or more ordinal outcomes with cutpoints on a shared continuous measurement scale

Cumulative receiver operator characteristic (ROC) curve analysis extends classic ROC curve analysis to discriminate three or more ordinal outcome levels on a shared continuous scale. The procedure combines cumulative logit regression with a cumulative extension to the ROC curve and performs as expected with ternary (three-level) ordinal outcomes under a variety of simulated conditions (unbalanced data, proportional and non-proportional odds, areas under the ROC curve [AUCs] from 0.70 to 0.95). Simulations also compared several criteria for selecting cutpoints to discriminate outcome levels: the Youden Index, Matthews Correlation Coefficient, Total Accuracy, and Markedness. Total Accuracy demonstrated the least absolute percent-bias. Cutpoints computed from maximum likelihood regression parameters demonstrated bias that was often negligible. The procedure was also applied to publicly available data related to computer imaging and biomarker exposure science, yielding good to excellent AUCs, as well as cutpoints with sensitivities and specificities of commensurate quality. Implementation of cumulative ROC curve analysis and extension to more than three outcome levels are straightforward. The author's programs for ternary ordinal outcomes are publicly available.

Estimating diagnostic accuracy for clustered ordinal diagnostic groups in the three-class case-Application to the early diagnosis of Alzheimer disease

Many medical diagnostic studies involve three ordinal diagnostic populations in which the diagnostic accuracy can be summarized by the volume or partial volume under the receiver operating characteristic surface for a diagnostic marker. When the diagnostic populations are clustered, e.g. by families, we propose to model the diagnostic marker by a general linear mixed model that takes into account of the correlation on the diagnostic marker from members of the same clusters. This model then facilitates the maximum likelihood estimation and statistical inferences of the diagnostic accuracy for the diagnostic marker. This approach naturally allows the incorporation of covariates as well as missing data when some clusters do not have subjects on all diagnostic groups in the estimation of, and the subsequent inferences on the diagnostic accuracy. We further study the performance of the proposed methods in a large simulation study with clustered data. Finally, we apply the proposed methodology to the data of several biomarkers collected by the Dominantly Inherited Alzheimer Network, an international family-clustered registry to study autosomal dominant Alzheimer disease which is a rare form of Alzheimer disease caused by mutations in any of the three genes including the amyloid precursor protein, presenilin 1 and presenilin 2. We estimate the accuracy of several cerebrospinal fluid and neuroimaging biomarkers in differentiating three diagnostic and genetic populations: normal non-mutation carriers, asymptomatic mutation carriers, and symptomatic mutation carriers.

Measuring diagnostic accuracy for biomarkers under tree-ordering

In the field of diagnostic studies for tree or umbrella ordering, under which the marker measurement for one class is lower or higher than those for the rest unordered classes, there exist a few diagnostic measures such as the naive AUC ( NAUC), the umbrella volume ( UV), and the recently proposed TAUC, i.e. area under a ROC curve for tree or umbrella ordering (TROC). However, an important characteristic about tree or umbrella ordering has been neglected. This paper mainly focuses on promoting the use of the integrated false negative rate under tree ordering ( ITFNR) as an additional diagnostic measure besides TAUC, and proposing the idea of using ( TAUC, ITFNR) instead of TAUC to evaluate the diagnostic accuracy of a biomarker under tree or umbrella ordering. Parametric and non-parametric approaches for constructing joint confidence region of ( TAUC, ITFNR) are proposed. Simulation studies under a variety of settings are carried out to assess and compare the performance of these methods. In the end, a published microarray data set is analyzed.

Three-way ROC validation of rs-fMRI visual information propagation transfer functions used to differentiate between RRMS and CIS optic neuritis patients

Resting-state fMRI (rs-fMRI) measures the temporal synchrony between different brain regions while the subject is at rest. We present an investigation using visual information propagation transfer functions as potential optic neuritis (ON) markers for the pathways between the lateral geniculate nuclei, the primary visual cortex, the lateral occipital cortex and the superior parietal cortex. We investigate marker reliability in differentiating between healthy controls and ON patients with clinically isolated syndrome (CIS), and relapsing-remitting multiple sclerosis (RRMS) using a three-way receiver operating characteristics analysis. We identify useful and reliable three-way ON related metrics in the rs-fMRI low-frequency band 0.0 Hz to 0.1 Hz, with potential markers associated with the higher frequency harmonics of these signals in the 0.1 Hz to 0.2 Hz and 0.2 Hz to 0.3 Hz bands.

Bronchiolitis Score of Sant Joan de Déu: BROSJOD Score, validation and usefulness

OBJECTIVE

To validate the bronchiolitis score of Sant Joan de Déu (BROSJOD) and to examine the previously defined scoring cutoff.

PATIENTS AND METHODS

Prospective, observational study. BROSJOD scoring was done by two independent physicians (at admission, 24 and 48 hr). Internal consistency of the score was assessed using Cronbach's α. To determine inter-rater reliability, the concordance correlation coefficient estimated as an intraclass correlation coefficient (CCC) and limits of agreement estimated as the 90% total deviation index (TDI) were estimated. An expert opinion was used to classify patients according to clinical severity. A validity analysis was conducted comparing the 3-level classification score to that expert opinion. Volume under the surface (VUS), predictive values, and probability of correct classification (PCC) were measured to assess discriminant validity.

RESULTS

About 112 patients were recruited, 62 of them (55.4%) males. Median age: 52.5 days (IQR: 32.75-115.25). The admission Cronbach's α was 0.77 (CI95%: 0.71; 0.82) and at 24 hr it was 0.65 (CI95%: 0.48; 0.7). The inter-rater reliability analysis was: CCC at admission 0.96 (95%CI 0.94-0.97), at 24 h 0.77 (95%CI 0.65-0.86), and at 48 hr 0.94 (95%CI 0.94-0.97); TDI 90%: 1.6, 2.9, and 1.57, respectively. The discriminant validity at admission: VUS of 0.8 (95%CI 0.70-0.90), at 24 h 0.92 (95%CI 0.85-0.99), and at 48 hr 0.93 (95%CI 0.87-0.99). The predictive values and PCC values were within 38-100% depending on the level of clinical severity.

CONCLUSION

There is a high inter-rater reliability, showing the BROSJOD score to be reliable and valid, even when different observers apply it. Pediatr Pulmonol. 2017;52:533-539. © 2016 Wiley Periodicals, Inc.

Nonparametric estimation and inference for polytomous discrimination index

Polytomous discrimination index is a novel and important diagnostic accuracy measure for multi-category classification. After reconstructing its probabilistic definition, we propose a nonparametric approach to the estimation of polytomous discrimination index based on an empirical sample of biomarker values. In this paper, we provide the finite-sample and asymptotic properties of the proposed estimators and such analytic results may facilitate the statistical inference. Simulation studies are performed to examine the performance of the nonparametric estimators. Two real data examples are analysed to illustrate our methodology.

Order Statistics Concordance Coefficient With Applications to Multichannel Biosignal Analysis

In this paper, we propose a novel concordance coefficient, called order statistics concordance coefficient (OSCOC), to quantify the association among multichannel biosignals. To uncover its properties, we compare OSCOC with three other similar indexes, i.e., average Pearson's product moment correlation coefficient (APPMCC), Kendall's concordance coefficients (KCC), and average Kendall's tau (AKT), under a multivariate normal model (MNM), linear model (LM), and nonlinear model. To further demonstrate its usefulness, we present an example on atrial arrhythmia analysis based on real-world multichannel cardiac signals. Theoretical derivations as well as numerical results suggest that 1) under MNM and LM, OSCOC performs equally well with APPMCC, and outperforms the other two methods, 2) in nonlinear case, OSCOC even has better performance than KCC and AKT, which are well known to be robust under increasing nonlinear transformations, and 3) OSCOC performs the best in the case study of arrhythmia analysis in terms of the volume under the surface.

Inverse probability weighting estimation of the volume under the ROC surface in the presence of verification bias

In diagnostic medicine, the volume under the receiver operating characteristic (ROC) surface (VUS) is a commonly used index to quantify the ability of a continuous diagnostic test to discriminate between three disease states. In practice, verification of the true disease status may be performed only for a subset of subjects under study since the verification procedure is invasive, risky, or expensive. The selection for disease examination might depend on the results of the diagnostic test and other clinical characteristics of the patients, which in turn can cause bias in estimates of the VUS. This bias is referred to as verification bias. Existing verification bias correction in three-way ROC analysis focuses on ordinal tests. We propose verification bias-correction methods to construct ROC surface and estimate the VUS for a continuous diagnostic test, based on inverse probability weighting. By applying U-statistics theory, we develop asymptotic properties for the estimator. A Jackknife estimator of variance is also derived. Extensive simulation studies are performed to evaluate the performance of the new estimators in terms of bias correction and variance. The proposed methods are used to assess the ability of a biomarker to accurately identify stages of Alzheimer's disease.

A new diagnostic accuracy measure and cut-point selection criterion

Most diagnostic accuracy measures and criteria for selecting optimal cut-points are only applicable to diseases with binary or three stages. Currently, there exist two diagnostic measures for diseases with general k stages: the hypervolume under the manifold and the generalized Youden index. While hypervolume under the manifold cannot be used for cut-points selection, generalized Youden index is only defined upon correct classification rates. This paper proposes a new measure named maximum absolute determinant for diseases with k stages ([Formula: see text]). This comprehensive new measure utilizes all the available classification information and serves as a cut-points selection criterion as well. Both the geometric and probabilistic interpretations for the new measure are examined. Power and simulation studies are carried out to investigate its performance as a measure of diagnostic accuracy as well as cut-points selection criterion. A real data set from Alzheimer's Disease Neuroimaging Initiative is analyzed using the proposed maximum absolute determinant.

Assessing the suitability of sets-based approaches: estimating the discriminative power of risk models for ordinal outcome treatments

BACKGROUND

In order to evaluate the discrimination performance of an ordinal model for improved disease screening, a new test was proposed where information was obtained across all samples simultaneously.

METHODS

The ordinal c-index builds upon the volume under the surface methodology without focusing on the accompanying receiver operating characteristic surfaces. However, it can be simplified to an average of pairwise c-indexes. In this paper, a set-based estimate (information was obtained across all samples simultaneously) was proposed by summing all correctly ordered groups. The asymptotic distribution of this proposed estimate was derived using U-statistics.

RESULTS

A predictive model was applied using the blood urea nitrogen/creatinine ratio to discriminate stroke in evolution in acute ischemic stroke patients, which could potentially be life-saving in emergency departments.

CONCLUSIONS

By conducting Monte Carlo simulations, it was concluded that the measure proposed herein is a better choice for clinical use because of the asymmetry of the predicted probabilities of groups.

Grid multi-category response logistic models

Background

Multi-category response models are very important complements to binary logistic models in medical decision-making. Decomposing model construction by aggregating computation developed at different sites is necessary when data cannot be moved outside institutions due to privacy or other concerns. Such decomposition makes it possible to conduct grid computing to protect the privacy of individual observations.

Methods

This paper proposes two grid multi-category response models for ordinal and multinomial logistic regressions. Grid computation to test model assumptions is also developed for these two types of models. In addition, we present grid methods for goodness-of-fit assessment and for classification performance evaluation.

Results

Simulation results show that the grid models produce the same results as those obtained from corresponding centralized models, demonstrating that it is possible to build models using multi-center data without losing accuracy or transmitting observation-level data. Two real data sets are used to evaluate the performance of our proposed grid models.

Conclusions

The grid fitting method offers a practical solution for resolving privacy and other issues caused by pooling all data in a central site. The proposed method is applicable for various likelihood estimation problems, including other generalized linear models.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-015-0133-y) contains supplementary material, which is available to authorized users.

Confidence Interval Estimation for Sensitivity to the Early Diseased Stage Based on Empirical Likelihood

Many disease processes can be divided into three stages: the non-diseased stage: the early diseased stage, and the fully diseased stage. To assess the accuracy of diagnostic tests for such diseases, various summary indexes have been proposed, such as volume under the surface (VUS), partial volume under the surface (PVUS), and the sensitivity to the early diseased stage given specificity and the sensitivity to the fully diseased stage (P2). This paper focuses on confidence interval estimation for P2 based on empirical likelihood. Simulation studies are carried out to assess the performance of the new methods compared to the existing parametric and nonparametric ones. A real dataset from Alzheimer's Disease Neuroimaging Initiative (ADNI) is analyzed.

Confidence interval estimation of the difference between two sensitivities to the early disease stage

Although most of the statistical methods for diagnostic studies focus on disease processes with binary disease status, many diseases can be naturally classified into three ordinal diagnostic categories, that is normal, early stage, and fully diseased. For such diseases, the volume under the ROC surface (VUS) is the most commonly used index of diagnostic accuracy. Because the early disease stage is most likely the optimal time window for therapeutic intervention, the sensitivity to the early diseased stage has been suggested as another diagnostic measure. For the purpose of comparing the diagnostic abilities on early disease detection between two markers, it is of interest to estimate the confidence interval of the difference between sensitivities to the early diseased stage. In this paper, we present both parametric and non-parametric methods for this purpose. An extensive simulation study is carried out for a variety of settings for the purpose of evaluating and comparing the performance of the proposed methods. A real example of Alzheimer's disease (AD) is analyzed using the proposed approaches.

Validation of Monte Carlo estimates of three-class ideal observer operating points for normal data

RATIONALE AND OBJECTIVES

Traditional two-class receiver operating characteristic (ROC) analysis is inadequate for the complete evaluation of observer performance in tasks with more than two classes.

MATERIALS AND METHODS

Here, a Monte Carlo estimation method for operating point coordinates on a three-class ROC surface is developed and compared with analytically calculated coordinates in two special cases: (1) univariate and (2) restricted bivariate trinormal underlying data.

RESULTS

In both cases, the statistical estimates were found to be good in the sense that the analytical values lay within the 95% confidence interval of the estimated values about 95% of the time.

CONCLUSIONS

The statistical estimation method should be key in the development of a pragmatic performance metric for evaluation of observers in classification tasks with three or more classes.

Assessing the discriminative ability of risk models for more than two outcome categories

The discriminative ability of risk models for dichotomous outcomes is often evaluated with the concordance index (c-index). However, many medical prediction problems are polytomous, meaning that more than two outcome categories need to be predicted. Unfortunately such problems are often dichotomized in prediction research. We present a perspective on the evaluation of discriminative ability of polytomous risk models, which may instigate researchers to consider polytomous prediction models more often. First, we suggest a "discrimination plot" as a tool to visualize the model's discriminative ability. Second, we discuss the use of one overall polytomous c-index versus a set of dichotomous measures to summarize the performance of the model. Third, we address several aspects to consider when constructing a polytomous c-index. These involve the assessment of concordance in pairs versus sets of patients, weighting by outcome prevalence, the value related to models with random performance, the reduction to the dichotomous c-index for dichotomous problems, and interpretation. We illustrate these issues on case studies dealing with ovarian cancer (four outcome categories) and testicular cancer (three categories). We recommend the use of a discrimination plot together with an overall c-index such as the Polytomous Discrimination Index. If the overall c-index suggests that the model has relevant discriminative ability, pairwise c-indexes for each pair of outcome categories are informative. For pairwise c-indexes we recommend the 'conditional-risk' method which is consistent with the analytical approach of the multinomial logistic regression used to develop polytomous risk models.