Measuring and estimating diagnostic accuracy when there are three ordinal diagnostic groups

Optimal Cut-Point Selection Methods Under Binary Classification When Subclasses Are Involved

In practice, we often encounter binary classification problems where both main classes consist of multiple subclasses. For example, in an ovarian cancer study where biomarkers were evaluated for their accuracy of distinguishing noncancer cases from cancer cases, the noncancer class consists of healthy subjects and benign cases, while the cancer class consists of subjects at both early and late stages. This article aims to provide a large number of optimal cut-point selection methods for such setting. Furthermore, we also study confidence interval estimation of the optimal cut-points. Simulation studies are carried out to explore the performance of the proposed cut-point selection methods as well as confidence interval estimation methods. A real ovarian cancer data set is analyzed using the proposed methods.

Direct estimation of volume under the ROC surface with verification bias

In practice, the receiver operating characteristic (ROC) curve of a diagnostic test is widely used to show the performance of the test for discriminating two-class events. The area under the ROC curve (AUC) is proposed as an index for the assessment of the diagnostic accuracy of the test under consideration. Due to ethical and cost considerations associated with application of gold standard (GS) tests, only a subset of the patients initially tested have verified disease status. Statistical evaluation of the test performance based only on test results from subjects with verified disease status are typically biased. Various AUC estimation methods for tests with verification biased data have been developed over the last few decades. In this article, we develop new direct estimation methods for the volume under the ROC surface (VUS) by extending the AUC estimation methods for two-class diagnostic tests to three-class diagnostic tests in the presence of verification bias. The proposed methods will provide a comprehensive guide to deal with the verification bias in three-class diagnostic test accuracy studies and lead to a better choice of diagnostic tests.

Interval estimation in three-class receiver operating characteristic analysis: A fairly general approach based on the empirical likelihood

The empirical likelihood is a powerful nonparametric tool, that emulates its parametric counterpart-the parametric likelihood-preserving many of its large-sample properties. This article tackles the problem of assessing the discriminatory power of three-class diagnostic tests from an empirical likelihood perspective. In particular, we concentrate on interval estimation in a three-class receiver operating characteristic analysis, where a variety of inferential tasks could be of interest. We present novel theoretical results and tailored techniques studied to efficiently solve some of such tasks. Extensive simulation experiments are provided in a supporting role, with our novel proposals compared to existing competitors, when possible. It emerges that our new proposals are extremely flexible, being able to compete with contestants and appearing suited to accommodating several distributions, such, for example, mixtures, for target populations. We illustrate the application of the novel proposals with a real data example. The article ends with a discussion and a presentation of some directions for future research.

Evaluating joint confidence region of hypervolume under ROC manifold and generalized Youden index

In biomarker evaluation/diagnostic studies, the hypervolume under the receiver operating characteristic manifold (HUM K $$ {\mathrm{HUM}}_K $$ ) and the generalized Youden index (J K $$ {J}_K $$ ) are the most popular measures for assessing classification accuracy under multiple classes. WhileHUM K $$ {\mathrm{HUM}}_K $$ is frequently used to evaluate the overall accuracy,J K $$ {J}_K $$ provides direct measure of accuracy at the optimal cut-points. Simultaneous evaluation ofHUM K $$ {\mathrm{HUM}}_K $$ andJ K $$ {J}_K $$ provides a comprehensive picture about the classification accuracy of the biomarker/diagnostic test under consideration. This article studies both parametric and non-parametric approaches for estimating the confidence region ofHUM K $$ {\mathrm{HUM}}_K $$ andJ K $$ {J}_K $$ for a single biomarker. The performances of the proposed methods are investigated by an extensive simulation study and are applied to a real data set from the Alzheimer's Disease Neuroimaging Initiative.

A new accuracy metric under three classes when subclasses are involved and its confidence interval estimation

"Compound multi-class classification" refers to the setting where three or more main classes are involved and at least one of the main classes have multiple subclasses. A common practice in evaluating biomarker performance under "compound multi-class classification" is "subclasses pooling." In this article, we first explore the downsides of accuracy metrics based on pooled data. Then we propose a new accuracy measure proper for "compound multi-class classification" with three ordinal main classes, namely "volume under compoundR O C $$ ROC $$ surface (V U S C $$ VU{S}_C $$ )." The proposedV U S C $$ VU{S}_C $$ evaluates the accuracy of a biomarker appropriately by identifying main classes without requiring specification of an ordering for marker values of subclasses within each main class. For confidence interval estimation ofV U S C $$ VU{S}_C $$ , both parametric and nonparametric methods are studied, and simulation studies are carried out to assess coverage probabilities. A subset of Alzheimer's Disease Neuroimaging Initiative study dataset is analyzed.

Bayesian and influence function-based empirical likelihoods for inference of sensitivity to the early diseased stage in diagnostic tests

In practice, a disease process might involve three ordinal diagnostic stages: the normal healthy stage, the early stage of the disease, and the stage of full development of the disease. Early detection is critical for some diseases since it often means an optimal time window for therapeutic treatments of the diseases. In this study, we propose a new influence function-based empirical likelihood method and Bayesian empirical likelihood methods to construct confidence/credible intervals for the sensitivity of a test to patients in the early diseased stage given a specificity and a sensitivity of the test to patients in the fully diseased stage. Numerical studies are performed to compare the finite sample performances of the proposed approaches with existing methods. The proposed methods are shown to outperform existing methods in terms of coverage probability. A real dataset from the Alzheimer's Disease Neuroimaging Initiative (ANDI) is used to illustrate the proposed methods.

A network approach to compute hypervolume under receiver operating characteristic manifold for multi-class biomarkers

Computation of hypervolume under ROC manifold (HUM) is necessary to evaluate biomarkers for their capability to discriminate among multiple disease types or diagnostic groups. However the original definition of HUM involves multiple integration and thus a medical investigation for multi-class receiver operating characteristic (ROC) analysis could suffer from huge computational cost when the formula is implemented naively. We introduce a novel graph-based approach to compute HUM efficiently in this article. The computational method avoids the time-consuming multiple summation when sample size or the number of categories is large. We conduct extensive simulation studies to demonstrate the improvement of our method over existing R packages. We apply our method to two real biomedical data sets to illustrate its application.

Construction of joint confidence spaces for the optimal true class fraction triplet in the ROC space using alternative biomarker cutoffs

Hepatocellular carcinoma (HCC) is the most common primary cancer of the liver. Finding new biomarkers for its early detection is of high clinical importance. As with many other diseases, cancer has a progressive nature. In cancer biomarker studies, it is often the case that the true disease status of the recruited individuals exhibits more than two classes. The receiver operating characteristic (ROC) surface is a well-known statistical tool for assessing the biomarkers' discriminatory ability in trichotomous settings. The volume under the ROC surface (VUS) is an overall measure of the discriminatory ability of a marker. In practice, clinicians are often in need of cutoffs for decision-making purposes. A popular approach for computing such cutoffs is the Youden index and its recent three-class generalization. A drawback of such a method is that it treats the data in a pairwise fashion rather than consider all the data simultaneously. The use of the minimized Euclidean distance from the perfection corner to the ROC surface (also known as closest to perfection method) is an alternative to the Youden index that may be preferable in some settings. When such a method is employed, there is a need for inferences around the resulting true class rates/fractions that correspond to the optimal operating point. In this paper, we provide an inferential framework for the derivation of marginal confidence intervals (CIs) and joint confidence spaces (CSs) around the corresponding true class fractions, when dealing with trichotomous settings. We explore parametric and nonparametric approaches for the construction of such CIs and CSs. We evaluate our approaches through extensive simulations and apply them to a real data set that refers to HCC patients.

A family of estimators to diagnostic accuracy when candidate tests are subject to detection limits-Application to diagnosing early stage Alzheimer disease

In disease diagnosis, individuals are usually assumed to be one of the two basic types, healthy or diseased, as typically based on an established gold standard. Candidate markers for diagnosing a disease often are much cheaper and less invasive than the gold standard but must be evaluated against the gold standard for their sensitivity and specificity to accurately diagnose the disease. When candidate diagnostic markers are fully measured, receiver operating characteristic curves have been the standard approaches for assessing diagnostic accuracy. However, full measurements of diagnostic markers may not be available above or below certain limits due to various practical and technical limitations. For example, in the diagnosis of Alzheimer disease using cerebrospinal fluid biomarkers, the Roche Elecsys® immunoassays have a measuring range for multiple cerebrospinal fluid molecular concentrations. Many cognitive tests used in diagnosing dementia due to Alzheimer disease are also subject to detection limits, often referred to as the floor and ceiling effects in the neuropsychological literature. We propose a new statistical methodology for estimating the diagnostic accuracy when a diagnostic marker is subject to detection limits by dividing the entire study sample into two sub-samples by a threshold of the diagnostic marker. We then propose a family of estimators to the area under the receiver operating characteristic curve by combining a conditional nonparametric estimator and another conditional semi-parametric estimator derived from Cox's proportional hazards model. We derive the variance to the proposed estimators, and further, assess the performance of the proposed estimators as a function of possible thresholds through an extensive simulation study, and recommend the optimum thresholds. Finally, we apply the proposed methodology to assess the ability of several cerebrospinal fluid biomarkers and cognitive tests in diagnosing early stage Alzheimer disease dementia.

A note on modeling placement values in the analysis of receiver operating characteristic curves

Recent advances in receiver operating characteristic (ROC) curve analyses advocate modeling of placement value (PV), a quantity that measures the position of diseased test scores relative to the healthy population. Compared to traditional approaches, this PV-based alternative works directly with ROC curves and is attractive when assessing covariate effects on, or incorporating a priori constraints of, ROC curves. Several distributions can be used to model the PV, yet little guidelines exist in the literature on which to use. Through extensive simulation studies, we investigate several parametric models for PV when data are generated from a variety of mechanisms. We discuss the pros and cons of each of these models and illustrate their applications with data from a study of prenatal ultrasound examinations and large-for-gestational age birth.

Flexible diagnostic measures and new cut-point selection methods under multiple ordered classes

Medical diagnosis is essentially a classification problem and usually it is done with multiple ordered classes. For example, cancer diagnosis might be "non-malignant," "early stage," or "late stage." Therefore, appropriate measures are needed to assess the accuracy of diagnostic markers under multiple ordered classes. However, all existing measures fail to differentiate among some distinctly different biomarkers. This paper presents a multi-step procedure for evaluating biomarker accuracy under multiple ordered classes. This procedure leads to two new flexible overall measures as well as three new cut-point selection methods with great computational ease. The performance of proposed measures and cut-point selection methods are numerically explored via a simulation study. In the end, an ovarian cancer dataset from the Prostate, Lung, Colorectal, and Ovarian cancer study is analyzed. The proposed accuracy measures were estimated for markers CA125 and HE4, and cut-points were estimated for the risk of ovarian malignancy algorithm score.

Reynolds Intellectual Screening Instrument 1st versus 2nd Edition in a Memory Disorder Sample

OBJECTIVE

The Reynolds Intellectual Screening Instrument (RIST) and its second edition (RIST-2) are brief intelligence screening instruments that potentially have value in older populations as their norms extend over age 90. This study examined performance on these two instruments in a sample of individuals presenting for evaluation in a memory disorder clinic.

METHOD

A sample of 1,145 subjects over the age of 50 was chosen from 1,761 consecutive referrals. Individuals who obtained a consensus diagnosis of Mild Cognitive Impairment (MCI; n = 536), possible dementia of the Alzheimer Type (DAT; n = 400), or those with subjective cognitive complaints (SCC; n = 209) and who completed a neuropsychological battery that included either the RIST (n = 747) or the RIST-2 (n = 398) were included in the sample. No clinically significant demographic or neuropsychological performance differences were found for those taking either version of the RIST.

RESULTS

Unlike the original version, RIST-2 Total and subtest scores were well below the mean for the DAT group and over 1 SD mean difference was seen for the DAT group when comparing the RIST and RIST-2 Totals. Diagnostic accuracy calculations suggested that the RIST-2 showed greater discrimination between the three groups although both versions achieved greater sensitivity than specificity.

CONCLUSIONS

Performance differences were evident when comparing the RIST and RIST-2, particularly for the DAT group. Although the RIST-2 evidenced greater diagnostic accuracy than its predecessor it should not be utilized in isolation for the clinical determination of DAT or MCI.

Combining multiple biomarkers to linearly maximize the diagnostic accuracy under ordered multi-class setting

Either in clinical study or biomedical research, it is a common practice to combine multiple biomarkers to improve the overall diagnostic performance. Despite the fact there exist a large number of statistical methods for biomarker combination under binary classification, research on this topic under multi-class setting is sparse. The overall diagnostic accuracy, i.e. the sum of correct classification rates, directly measures the classification accuracy of the combined biomarkers. Hence the overall accuracy can serve as an important objective function for biomarker combination, especially when the combined biomarkers are used for the purpose of making medical diagnosis. In this paper, we address the problem of combining multiple biomarkers to directly maximize the overall diagnostic accuracy by presenting several grid search methods and derivation-based methods. A comprehensive simulation study was conducted to compare the performances of these methods. An ovarian cancer data set is analyzed in the end.

Item response theory analysis of the Clinical Dementia Rating

INTRODUCTION

The Clinical Dementia Rating (CDR) is widely used in Alzheimer's disease research studies and has well established reliability and validity. To facilitate the development of an online, electronic CDR (eCDR) for more efficient clinical applications, this study aims to produce a shortened version of the CDR, and to develop the statistical model for automatic scoring.

METHODS

Item response theory (IRT) was used for item evaluation and model development. An automatic scoring algorithm was validated using existing CDR global and domain box scores as the reference standard.

RESULTS

Most CDR items discriminate well at mild and very mild levels of cognitive impairment. The bi-factor IRT model fits best and the shortened CDR still demonstrates very high classification accuracy (81%∼92%).

DISCUSSION

The shortened version of the CDR and the automatic scoring algorithm has established a good foundation for developing an eCDR and will ultimately improve the efficiency of cognitive assessment.

Accurate likelihood inference for the volume under the ROC surface

BACKGROUND

With three ordered diagnostic categories, the volume under the receiver operating characteristic (ROC) surface, which is the extension of the area under the ROC curve for binary diagnostic outcomes, is the most commonly used measure for the overall diagnostic accuracy. For a continuous-scale diagnostic test, classical likelihood-based inference about the area under the ROC curve can be inaccurate, in particular when the sample size is small, and higher order inferential procedures have been proposed.

AIM

The goal of this paper is to illustrate higher order likelihood procedures for parametric inference in small samples, which provide accurate point estimates and confidence intervals for the volume under the ROC surface.

METHODS

Simulation studies are performed in order to illustrate the accuracy of the proposed methodology, and two applications to real data are discussed.

RESULTS

We show that likelihood modern inference provide refinements to classical inferential results. Furthermore, the freely available R package likelihoodAsy makes now their use almost automatic.

CONCLUSION

Modern likelihood inference based on higher-order asymptotic methods for the area under the ROC surface provide refinements to classical inferential results. A possible limitation of higher-order asymptotic methods for practical use is that their software implementation can be awkward. Nevertheless, use of the freely available R package likelihoodAsy makes such implementation straightforward.

Usefulness of the 15-item geriatric depression scale (GDS-15) for classifying minor and major depressive disorders among community-dwelling elders

BACKGROUND

The 15-item geriatric depression scale (GDS-15) is a short form of GDS and is used to screen, diagnose, and evaluate depression in elderly individuals. Most previous studies evaluated the ability of GDS-15 to discriminate between depressive and non-depressive states. In this study, we investigated the multi-stage discriminating ability of GDS-15.

METHODS

A total of 774 participants, over 65 years of age were included (normal, n = 650; minor depressive disorder [MnDD], n = 94; major depressive disorder [MDD], n = 30). Multi-category receiver operating characteristic (ROC) surfaces were evaluated to identify three stages of geriatric depression. The optimal cutoff points were selected based on the volume under the ROC surface (VUS) and the Youden index.

RESULTS

In the results of multi-category classification analyses, VUS of the GDS-15 of 0.61 was obtained, and optimal cutoff points of the GDS-15 for multiple stages of depression of 4 (between normal and MnDD) and 11 (between MnDD and MDD) were derived. The Youden index for the GDS-15 was 0.49, and the derived optimal cutoff points were 5 and 10, for the multiple stages, respectively. The overall diagnostic accuracy based on the Youden index was superior to that based on the VUS in the GDS-15.

LIMITATIONS

The participants' cognitive function has potential to affect the GDS-15 score; nevertheless, the study included those with mild cognitive impairment.

CONCLUSIONS

GDS-15 was a useful tool to classify stages of geriatric depression into either minor or major depressive disorder.

A Bayesian semiparametric approach to correlated ROC surfaces with stochastic order constraints

In application of diagnostic accuracy, it is possible that a priori information may exist regarding the test score distributions, either between different disease populations for a single test or between multiple correlated tests. Few have considered constrained diagnostic accuracy analysis when the true disease status is binary; almost none when the disease status is ordinal. Motivated by a study on diagnosing endometriosis, we propose an approach to estimating diagnostic accuracy measures that can incorporate different stochastic order constraints on the test scores when an ordinal true disease status is in consideration. We show that the Dirichlet process mixture provides a convenient framework to both flexibly model the test score distributions and embed the a priori ordering constraints. We also utilize the Dirichlet process mixture to model the correlation between multiple tests. In taking a Bayesian perspective to inference, we develop an efficient Markov chain Monte Carlo algorithm to sample from the posterior distribution and provide posterior estimates of the receiver operating characteristic surfaces and the associated summary measures. The proposed approach is evaluated with extensive simulation studies, and is demonstrated with an application to the endometriosis study.

Screening for Normal Cognition, Mild Cognitive Impairment, and Dementia with the Korean Dementia Screening Questionnaire

OBJECTIVE

The Korean Dementia Screening Questionnaire (KDSQ) is an informant-based instrument used to screen for cognitive dysfunction. However, its ability to only dichotomously discriminate between dementia and normal cognition has been previously investigated. This study investigated the ability of the KDSQ to classify not only dichotomous but also multiple stages of cognitive dysfunction.

METHODS

We examined 582 participants. Receiver operating characteristic (ROC) curves were used to determine dichotomous classification parameters. Multi-category ROC surfaces were evaluated to classify the three stages of cognitive dysfunction.

RESULTS

Dichotomous classification using the ROC curve analyses showed that the area under the curve was 0.92 for dementia for subjects without dementia and 0.96 for dementia in controls. Simultaneous multi-category classification analyses showed that the volume under the ROC surface (VUS) was 0.57 and that the derived optimal cut-off points were 2 and 8 for controls, MCI, and dementia. The estimated Youden index for the KDSQ was 0.48, and the derived optimal cut-off points were 5 and 10. The overall classification accuracy of the VUS and Youden index was 61.2% and 58.6%, respectively.

CONCLUSION

The KDSQ is useful for classifying dichotomous and multi-category stages of cognitive dysfunction.

Estimating diagnostic accuracy for clustered ordinal diagnostic groups in the three-class case-Application to the early diagnosis of Alzheimer disease

Many medical diagnostic studies involve three ordinal diagnostic populations in which the diagnostic accuracy can be summarized by the volume or partial volume under the receiver operating characteristic surface for a diagnostic marker. When the diagnostic populations are clustered, e.g. by families, we propose to model the diagnostic marker by a general linear mixed model that takes into account of the correlation on the diagnostic marker from members of the same clusters. This model then facilitates the maximum likelihood estimation and statistical inferences of the diagnostic accuracy for the diagnostic marker. This approach naturally allows the incorporation of covariates as well as missing data when some clusters do not have subjects on all diagnostic groups in the estimation of, and the subsequent inferences on the diagnostic accuracy. We further study the performance of the proposed methods in a large simulation study with clustered data. Finally, we apply the proposed methodology to the data of several biomarkers collected by the Dominantly Inherited Alzheimer Network, an international family-clustered registry to study autosomal dominant Alzheimer disease which is a rare form of Alzheimer disease caused by mutations in any of the three genes including the amyloid precursor protein, presenilin 1 and presenilin 2. We estimate the accuracy of several cerebrospinal fluid and neuroimaging biomarkers in differentiating three diagnostic and genetic populations: normal non-mutation carriers, asymptomatic mutation carriers, and symptomatic mutation carriers.

Measuring diagnostic accuracy for biomarkers under tree-ordering

In the field of diagnostic studies for tree or umbrella ordering, under which the marker measurement for one class is lower or higher than those for the rest unordered classes, there exist a few diagnostic measures such as the naive AUC ( NAUC), the umbrella volume ( UV), and the recently proposed TAUC, i.e. area under a ROC curve for tree or umbrella ordering (TROC). However, an important characteristic about tree or umbrella ordering has been neglected. This paper mainly focuses on promoting the use of the integrated false negative rate under tree ordering ( ITFNR) as an additional diagnostic measure besides TAUC, and proposing the idea of using ( TAUC, ITFNR) instead of TAUC to evaluate the diagnostic accuracy of a biomarker under tree or umbrella ordering. Parametric and non-parametric approaches for constructing joint confidence region of ( TAUC, ITFNR) are proposed. Simulation studies under a variety of settings are carried out to assess and compare the performance of these methods. In the end, a published microarray data set is analyzed.

Bayesian nonparametric inference for the three-class Youden index and its associated optimal cutoff points

The three-class Youden index serves both as a measure of medical test accuracy and a criterion to choose the optimal pair of cutoff values for classifying subjects into three ordinal disease categories (e.g. no disease, mild disease, advanced disease). We present a Bayesian nonparametric approach for estimating the three-class Youden index and its corresponding optimal cutoff values based on Dirichlet process mixtures, which are robust models that can handle intricate features of distributions for complex data. Results from a simulation study are presented and an application to data from the Trail Making Test to assess cognitive impairment in Parkinson's disease patients is detailed.

Confidence intervals for differences between volumes under receiver operating characteristic surfaces (VUS) and generalized Youden indices (GYIs)

This article explores both existing and new methods for the construction of confidence intervals for differences of indices of diagnostic accuracy of competing pairs of biomarkers in three-class classification problems and fills the methodological gaps for both parametric and non-parametric approaches in the receiver operating characteristic surface framework. The most widely used such indices are the volume under the receiver operating characteristic surface and the generalized Youden index. We describe implementation of all methods and offer insight regarding the appropriateness of their use through a large simulation study with different distributional and sample size scenarios. Methods are illustrated using data from the Alzheimer's Disease Neuroimaging Initiative study, where assessment of cognitive function naturally results in a three-class classification setting.

Informant questionnaire on cognitive decline in the elderly (IQCODE) for classifying cognitive dysfunction as cognitively normal, mild cognitive impairment, and dementia

BACKGROUND

The Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE) is a reliable, validated informant-based instrument in screening for cognitive dysfunction. However, previous studies have evaluated only the ability to discriminate dichotomously, such as dementia from cognitively normal (CN) individuals or mild cognitive impairment (MCI) from CN. This study investigated the ability of the IQCODE to classify not only dichotomous but also multiple stages of cognitive dysfunction.

METHODS

We examined 228 consecutive participants (76 CN, 76 with MCI, and 76 with dementia). Receiver operating characteristic (ROC) curves determined dichotomous classification parameters. Multi-category ROC surfaces were evaluated to classify three stages of cognitive dysfunction.

RESULTS

Dichotomous classification using the ROC curve analyses showed that the area under the ROC curve was 0.91 for dementia from participants without dementia and 0.71 for MCI from CN. Simultaneous multi-category classification analyses showed that the volume under the ROC surface was 0.61 and the derived optimal cut-off points were 3.15 and 3.73 for CN, MCI, and dementia. The Youden index for the IQCODE was estimated as 0.51 and the derived optimal cut-off points were 3.33 and 3.70. The overall classification accuracy by the VUS was 58.3% and that by the Youden index 61.8%.

CONCLUSIONS

IQCODE is useful to classify the dichotomous and multi-category stages of cognitive dysfunction.

A new diagnostic accuracy measure and cut-point selection criterion

Most diagnostic accuracy measures and criteria for selecting optimal cut-points are only applicable to diseases with binary or three stages. Currently, there exist two diagnostic measures for diseases with general k stages: the hypervolume under the manifold and the generalized Youden index. While hypervolume under the manifold cannot be used for cut-points selection, generalized Youden index is only defined upon correct classification rates. This paper proposes a new measure named maximum absolute determinant for diseases with k stages ([Formula: see text]). This comprehensive new measure utilizes all the available classification information and serves as a cut-points selection criterion as well. Both the geometric and probabilistic interpretations for the new measure are examined. Power and simulation studies are carried out to investigate its performance as a measure of diagnostic accuracy as well as cut-points selection criterion. A real data set from Alzheimer's Disease Neuroimaging Initiative is analyzed using the proposed maximum absolute determinant.

Better see to better agree: phosphohistone H3 increases interobserver agreement in mitotic count for meningioma grading and imposes new specific thresholds

BACKGROUND

Mitotic count on hematoxylin and eosin (H&E)-stained slides is a crucial diagnostic criterion in meningioma grading. However, mitosis assessment on H&E slides can be impaired by technical factors and by pathologist's experience. Phosphohistone H3 (PHH3) serine-10 is a mitosis-specific antibody that has proven to facilitate mitotic count in various tumors.

METHODS

A series of 70 meningiomas (15 grade I, 40 grade II, 15 grade III) was used to validate PHH3 intra- and interobserver reproducibility and to identify PHH3-specific mitotic thresholds. Four pathologists with different experience in neuropathology counted mitoses on both H&E- and PHH3-stained slides.

RESULTS

H&E and PHH3 mitotic rates were highly correlated (Pearson's r = 0.92, P < .0001). PHH3 mitotic counts had both a good mean interobserver correlation (R(m) = 0.83) and a good intraclass correlation (0.78), higher than H&E mitotic indices (R(m) = 0.77, intraclass correlation = 0.71). After further stratification of meningiomas according to World Health Organization grade, PHH3 performed better in terms of interobserver concordance (Kendall's W = 0.761) compared with H&E (Kendall's W = 0.697). Referring to the same meningioma groups identified by World Health Organization grade as the gold standard, the volume under the receiver operator characteristic surface was 0.91, indicating a very good diagnostic ability of PHH3 scores in discriminating the 3 meningioma groups. The 2 optimal PHH3-specific cutoff values were 6.61 and 22.02.

CONCLUSION

PHH3 staining is a useful diagnostic complementary tool to standard H&E mitotic count, optimizing intra- and interobserver reproducibility. PHH3-specific mitotic thresholds should be adopted to avoid overgrading of meningioma when ancillary methods are employed.

The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

BACKGROUND

Severity of intellectual disability (ID) is associated with a range of outcomes for the individual and having an indication of severity can help inform support needs. Previous research has not evaluated whether screening tools can accurately ascertain severity category in addition to providing a red flag for the presence of ID.

METHODS

We used multi-category receiver operating characteristic (ROC) analysis to assess whether the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) could be used clinically to classify individuals (n = 191) aged between 12 and 18 according to British Psychological Society (BPS) categories of severity of impairment.

RESULTS

The volume under the surface statistic (VUS) was 0.59. The optimal cut-points estimated based on the ROC surface and Youden Index provided correct classification probabilities for the severe, significant and non-ID groups of 0.44, 0.63 and 0.86 and 0.79, 0.29 and 0.88 respectively.

CONCLUSIONS

While the CAIDS-Q can accurately discriminate between those with and without ID, and provides a heuristic for severity of ID, the results indicate that it does not reliably identify whether an individual falls into the severe or significant category of intellectual impairment.

Confidence Interval Estimation for Sensitivity to the Early Diseased Stage Based on Empirical Likelihood

Many disease processes can be divided into three stages: the non-diseased stage: the early diseased stage, and the fully diseased stage. To assess the accuracy of diagnostic tests for such diseases, various summary indexes have been proposed, such as volume under the surface (VUS), partial volume under the surface (PVUS), and the sensitivity to the early diseased stage given specificity and the sensitivity to the fully diseased stage (P2). This paper focuses on confidence interval estimation for P2 based on empirical likelihood. Simulation studies are carried out to assess the performance of the new methods compared to the existing parametric and nonparametric ones. A real dataset from Alzheimer's Disease Neuroimaging Initiative (ADNI) is analyzed.

Estimating confidence intervals for the difference in diagnostic accuracy with three ordinal diagnostic categories without a gold standard

With three ordinal diagnostic categories, the most commonly used measures for the overall diagnostic accuracy are the volume under the ROC surface (VUS) and partial volume under the ROC surface (PVUS), which are the extensions of the area under the ROC curve (AUC) and partial area under the ROC curve (PAUC), respectively. A gold standard (GS) test on the true disease status is required to estimate the VUS and PVUS. However, oftentimes it may be difficult, inappropriate, or impossible to have a GS because of misclassification error, risk to the subjects or ethical concerns. Therefore, in many medical research studies, the true disease status may remain unobservable. Under the normality assumption, a maximum likelihood (ML) based approach using the expectation-maximization (EM) algorithm for parameter estimation is proposed. Three methods using the concepts of generalized pivot and parametric/nonparametric bootstrap for confidence interval estimation of the difference in paired VUSs and PVUSs without a GS are compared. The coverage probabilities of the investigated approaches are numerically studied. The proposed approaches are then applied to a real data set of 118 subjects from a cohort study in early stage Alzheimer's disease (AD) from the Washington University Knight Alzheimer's Disease Research Center to compare the overall diagnostic accuracy of early stage AD between two different pairs of neuropsychological tests.

Diagnostic thresholds with three ordinal groups

In practice, there exist many disease processes with three ordinal disease classes; for example, in the detection of Alzheimer's disease (AD) a patient can be classified as healthy (disease-free stage), mild cognitive impairment (early disease stage), or AD (full disease stage). The treatment interventions and effectiveness of such disease processes will depend on the disease stage. Therefore, it is important to develop diagnostic tests with the ability to discriminate between the three disease stages. Measuring the overall ability of diagnostic tests to discriminate between the three classes has been discussed extensively in the literature. However, there has been little proposed on how to select clinically meaningful thresholds for such diagnostic tests, except for a method based on the generalized Youden index by Nakas et al. (2010). In this article, we propose two new criteria for selecting diagnostic thresholds in the three-class setting. The numerical study demonstrated that the proposed methods may provide thresholds with less variability and more balance among the correct classification rates for the three stages. The proposed methods are applied to two real examples: the clinical diagnosis of AD from the Washington University Alzheimer's Disease Research Center and the detection of liver cancer (LC) using protein segments.

Confidence interval estimation of the difference between two sensitivities to the early disease stage

Although most of the statistical methods for diagnostic studies focus on disease processes with binary disease status, many diseases can be naturally classified into three ordinal diagnostic categories, that is normal, early stage, and fully diseased. For such diseases, the volume under the ROC surface (VUS) is the most commonly used index of diagnostic accuracy. Because the early disease stage is most likely the optimal time window for therapeutic intervention, the sensitivity to the early diseased stage has been suggested as another diagnostic measure. For the purpose of comparing the diagnostic abilities on early disease detection between two markers, it is of interest to estimate the confidence interval of the difference between sensitivities to the early diseased stage. In this paper, we present both parametric and non-parametric methods for this purpose. An extensive simulation study is carried out for a variety of settings for the purpose of evaluating and comparing the performance of the proposed methods. A real example of Alzheimer's disease (AD) is analyzed using the proposed approaches.

Youden index and Associated Cut-points for Three Ordinal Diagnostic Groups

Directly relating to sensitivity and specificity and providing an optimal cut-point which maximizes overall classification effectiveness for diagnosis purpose, the Youden index has been frequently utilized in biomedical diagnosis practice. Current application of the Youden index is limited to two diagnostic groups. However, there usually exists a transitional intermediate stage in many disease processes. Early recognition of this intermediate stage is vital to open an optimal window for therapeutic intervention. In this paper, we extend the Youden index to assess diagnostic accuracy when there are three ordinal diagnostic groups. Parametric and nonparametric methods are presented to estimate the optimal Youden index, the underlying optimal cut-points and the associated confidence intervals. Extensive simulation studies covering representative distributional assumptions are reported to compare performance of the proposed methods. A real example illustrates the usefulness of the Youden index in evaluating discriminating ability of diagnostic tests.

DiagTest3Grp: An R Package for Analyzing Diagnostic Tests with Three Ordinal Groups

Medical researchers endeavor to identify potentially useful biomarkers to develop marker-based screening assays for disease diagnosis and prevention. Useful summary measures which properly evaluate the discriminative ability of diagnostic markers are critical for this purpose. Literature and existing software, for example, R packages nicely cover summary measures for diagnostic markers used for the binary case (e.g., healthy vs. diseased). An intermediate population at an early disease stage usually exists between the healthy and the fully diseased population in many disease processes. Supporting utilities for three-group diagnostic tests are highly desired and important for identifying patients at the early disease stage for timely treatments. However, application packages which provide summary measures for three ordinal groups are currently lacking. This paper focuses on two summary measures of diagnostic accuracy-volume under the receiver operating characteristic surface and the extended Youden index, with three diagnostic groups. We provide the R package DiagTest3Grp to estimate, under both parametric and nonparametric assumptions, the two summary measures and the associated variances, as well as the optimal cut-points for disease diagnosis. An omnibus test for multiple markers and a Wald test for two markers, on independent or paired samples, are incorporated to compare diagnostic accuracy across biomarkers. Sample size calculation under the normality assumption can be performed in the R package to design future diagnostic studies. A real world application evaluating the diagnostic accuracy of neuropsychological markers for Alzheimer's disease is used to guide readers through step-by-step implementation of DiagTest3Grp to demonstrate its utility.